You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2021/08/10 17:56:21 UTC
[GitHub] [hbase] wchevreuil opened a new pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
wchevreuil opened a new pull request #3574:
URL: https://github.com/apache/hbase/pull/3574
…d Merging
Change-Id: I5d0bbd99d27a6b3ecdd0a069dc8741239b32edfa
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
Apache9 commented on a change in pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#discussion_r687752070
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/SplitTableRegionProcedure.java
##########
@@ -648,9 +656,11 @@ public void createDaughterRegions(final MasterProcedureEnv env) throws IOExcepti
// there's files to split. It then fires up everything, waits for
// completion and finally checks for any exception
//
- // Note: splitStoreFiles creates daughter region dirs under the parent splits dir
- // Nothing to unroll here if failure -- re-run createSplitsDir will
- // clean this up.
+ // Note: From HBASE-26187, splitStoreFiles now creates daughter region dirs straight under the
Review comment:
The comment is incorrect now? We do not rely on CatalogJanitor to clean up the partial regions. We need to clean in rollback.
I think we should just recreate the whole region directly when we call this method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#issuecomment-896236928
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 32s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 19s | master passed |
| +1 :green_heart: | compile | 3m 30s | master passed |
| +1 :green_heart: | checkstyle | 1m 18s | master passed |
| +1 :green_heart: | spotbugs | 2m 47s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 51s | the patch passed |
| +1 :green_heart: | compile | 3m 27s | the patch passed |
| +1 :green_heart: | javac | 3m 27s | the patch passed |
| -0 :warning: | checkstyle | 1m 16s | hbase-server: The patch generated 9 new + 190 unchanged - 4 fixed = 199 total (was 194) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 24m 10s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 2m 29s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. |
| | | 60m 55s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3574 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 4f7589670228 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 5e8a269b1a |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt |
| Max. process+thread count | 85 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/1/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a change in pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
wchevreuil commented on a change in pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#discussion_r686828587
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
##########
@@ -638,47 +602,33 @@ void cleanupDaughterRegion(final RegionInfo regionInfo) throws IOException {
*/
public Path commitDaughterRegion(final RegionInfo regionInfo)
throws IOException {
- Path regionDir = new Path(this.tableDir, regionInfo.getEncodedName());
- Path daughterTmpDir = this.getSplitsDir(regionInfo);
-
- if (fs.exists(daughterTmpDir)) {
-
+ Path regionDir = this.getSplitsDir(regionInfo);
+ if (fs.exists(regionDir)) {
// Write HRI to a file in case we need to recover hbase:meta
- Path regionInfoFile = new Path(daughterTmpDir, REGION_INFO_FILE);
+ Path regionInfoFile = new Path(regionDir, REGION_INFO_FILE);
byte[] regionInfoContent = getRegionInfoFileContent(regionInfo);
writeRegionInfoFileContent(conf, fs, regionInfoFile, regionInfoContent);
-
- // Move the daughter temp dir to the table dir
- if (!rename(daughterTmpDir, regionDir)) {
- throw new IOException("Unable to rename " + daughterTmpDir + " to " + regionDir);
- }
}
return regionDir;
}
/**
- * Create the region splits directory.
+ * Creates region split daughter directories under the table dir.
*/
public void createSplitsDir(RegionInfo daughterA, RegionInfo daughterB) throws IOException {
Path splitdir = getSplitsDir();
- if (fs.exists(splitdir)) {
- LOG.info("The " + splitdir + " directory exists. Hence deleting it to recreate it");
- if (!deleteDir(splitdir)) {
- throw new IOException("Failed deletion of " + splitdir + " before creating them again.");
- }
- }
// splitDir doesn't exists now. No need to do an exists() call for it.
- if (!createDir(splitdir)) {
- throw new IOException("Failed create of " + splitdir);
+ if (!fs.exists(splitdir)) {
Review comment:
Removing the check, not needed now that we are creating the resulting split region on the table dir itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#issuecomment-897991762
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 2m 21s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 10s | master passed |
| +1 :green_heart: | compile | 3m 24s | master passed |
| +1 :green_heart: | checkstyle | 1m 12s | master passed |
| +1 :green_heart: | spotbugs | 2m 10s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 7s | the patch passed |
| +1 :green_heart: | compile | 3m 23s | the patch passed |
| +1 :green_heart: | javac | 3m 23s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 14s | hbase-server: The patch generated 0 new + 190 unchanged - 4 fixed = 190 total (was 194) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 20m 11s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 2m 21s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. |
| | | 54m 50s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/4/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3574 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux ee4ad4a7f6cd 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 11222fc4df |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Max. process+thread count | 85 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/4/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
Apache9 commented on pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#issuecomment-897480236
> One thought I had now is that we probably need to back off CatalogJanitor from running whilst a split/merge is going on. As we are creating the dirs under table directory, I guess there's a risk CatalogJanitor wipe those dirs off before the proc reaches the point to add the region in meta. Let me check on this further.
Ah, yes, I also noticed this when review this PR as you mentioned CatalogJanitor. But even we do not write directly to region dir, there could still be an interval between we move the region into place and add it in meta table? I guess we should have already found the way to deal with this, otherwise it could cause data loss...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
Apache9 commented on a change in pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#discussion_r689413485
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
##########
@@ -638,47 +598,42 @@ void cleanupDaughterRegion(final RegionInfo regionInfo) throws IOException {
*/
public Path commitDaughterRegion(final RegionInfo regionInfo)
throws IOException {
- Path regionDir = new Path(this.tableDir, regionInfo.getEncodedName());
- Path daughterTmpDir = this.getSplitsDir(regionInfo);
-
- if (fs.exists(daughterTmpDir)) {
-
+ Path regionDir = this.getSplitsDir(regionInfo);
+ if (fs.exists(regionDir)) {
// Write HRI to a file in case we need to recover hbase:meta
- Path regionInfoFile = new Path(daughterTmpDir, REGION_INFO_FILE);
+ Path regionInfoFile = new Path(regionDir, REGION_INFO_FILE);
byte[] regionInfoContent = getRegionInfoFileContent(regionInfo);
writeRegionInfoFileContent(conf, fs, regionInfoFile, regionInfoContent);
-
- // Move the daughter temp dir to the table dir
- if (!rename(daughterTmpDir, regionDir)) {
- throw new IOException("Unable to rename " + daughterTmpDir + " to " + regionDir);
- }
}
return regionDir;
}
/**
- * Create the region splits directory.
+ * Creates region split daughter directories under the table dir. If the daughter regions already
+ * exist, for example, in the case of a recovery from a previous failed split procedure, this
+ * method deletes the given region dir recursively, then recreates it again.
*/
public void createSplitsDir(RegionInfo daughterA, RegionInfo daughterB) throws IOException {
- Path splitdir = getSplitsDir();
- if (fs.exists(splitdir)) {
- LOG.info("The " + splitdir + " directory exists. Hence deleting it to recreate it");
- if (!deleteDir(splitdir)) {
- throw new IOException("Failed deletion of " + splitdir + " before creating them again.");
+ Path daughterADir = getSplitsDir(daughterA);
+ if (fs.exists(daughterADir)) {
Review comment:
if (fs.exists(daughtADir) && !deleteDir(daughterADir))
Can combine the two if in one line.
Anyway, not a big problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#issuecomment-897644924
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 4s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 4s | master passed |
| +1 :green_heart: | compile | 1m 2s | master passed |
| +1 :green_heart: | shadedjars | 8m 55s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 38s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 8s | the patch passed |
| +1 :green_heart: | compile | 1m 3s | the patch passed |
| +1 :green_heart: | javac | 1m 3s | the patch passed |
| +1 :green_heart: | shadedjars | 8m 59s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 38s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 234m 18s | hbase-server in the patch passed. |
| | | 266m 52s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3574 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 4a96cd9806c9 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 238c9b40bf |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/2/testReport/ |
| Max. process+thread count | 2646 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/2/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#issuecomment-897688686
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 6s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 13s | master passed |
| +1 :green_heart: | compile | 3m 27s | master passed |
| +1 :green_heart: | checkstyle | 1m 13s | master passed |
| +1 :green_heart: | spotbugs | 2m 15s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 4s | the patch passed |
| +1 :green_heart: | compile | 3m 20s | the patch passed |
| +1 :green_heart: | javac | 3m 20s | the patch passed |
| -0 :warning: | checkstyle | 1m 13s | hbase-server: The patch generated 10 new + 190 unchanged - 4 fixed = 200 total (was 194) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 20m 19s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 2m 41s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. |
| | | 52m 45s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/3/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3574 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 9ea0a3d87b79 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 238c9b40bf |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt |
| Max. process+thread count | 85 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/3/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#issuecomment-897507937
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 8s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 22s | master passed |
| +1 :green_heart: | compile | 3m 27s | master passed |
| +1 :green_heart: | checkstyle | 1m 13s | master passed |
| +1 :green_heart: | spotbugs | 2m 17s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 6s | the patch passed |
| +1 :green_heart: | compile | 3m 25s | the patch passed |
| +1 :green_heart: | javac | 3m 25s | the patch passed |
| -0 :warning: | checkstyle | 1m 12s | hbase-server: The patch generated 10 new + 190 unchanged - 4 fixed = 200 total (was 194) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 20m 41s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 2m 23s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. |
| | | 52m 53s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3574 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 78cc9fb96dfa 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 238c9b40bf |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt |
| Max. process+thread count | 86 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/2/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#issuecomment-896341324
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 2s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 22s | master passed |
| +1 :green_heart: | compile | 1m 3s | master passed |
| +1 :green_heart: | shadedjars | 9m 4s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 38s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 8s | the patch passed |
| +1 :green_heart: | compile | 1m 3s | the patch passed |
| +1 :green_heart: | javac | 1m 3s | the patch passed |
| +1 :green_heart: | shadedjars | 9m 12s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 36s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 210m 35s | hbase-server in the patch failed. |
| | | 243m 42s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3574 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 86e53fe1c597 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 5e8a269b1a |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/1/testReport/ |
| Max. process+thread count | 2916 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/1/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#issuecomment-897833703
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 58s | Docker mode activated. |
| -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 23s | master passed |
| +1 :green_heart: | compile | 1m 3s | master passed |
| +1 :green_heart: | shadedjars | 9m 9s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 39s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 6s | the patch passed |
| +1 :green_heart: | compile | 1m 5s | the patch passed |
| +1 :green_heart: | javac | 1m 5s | the patch passed |
| +1 :green_heart: | shadedjars | 9m 9s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 36s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 210m 41s | hbase-server in the patch failed. |
| | | 243m 51s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3574 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux abe9229dd2e4 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 238c9b40bf |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/3/testReport/ |
| Max. process+thread count | 2806 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/3/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#issuecomment-899562335
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 3s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 5s | master passed |
| +1 :green_heart: | compile | 3m 23s | master passed |
| +1 :green_heart: | checkstyle | 1m 12s | master passed |
| +1 :green_heart: | spotbugs | 2m 10s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 4s | the patch passed |
| +1 :green_heart: | compile | 3m 25s | the patch passed |
| +1 :green_heart: | javac | 3m 25s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 14s | hbase-server: The patch generated 0 new + 190 unchanged - 4 fixed = 190 total (was 194) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 20m 9s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 2m 38s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. |
| | | 52m 49s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/6/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3574 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux a126260ee7c9 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 44d5624908 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Max. process+thread count | 85 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/6/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] wchevreuil merged pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
wchevreuil merged pull request #3574:
URL: https://github.com/apache/hbase/pull/3574
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#issuecomment-899401655
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 3s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 6s | master passed |
| +1 :green_heart: | compile | 3m 21s | master passed |
| +1 :green_heart: | checkstyle | 1m 14s | master passed |
| +1 :green_heart: | spotbugs | 2m 10s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 6s | the patch passed |
| +1 :green_heart: | compile | 3m 19s | the patch passed |
| +1 :green_heart: | javac | 3m 19s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 13s | hbase-server: The patch generated 0 new + 190 unchanged - 4 fixed = 190 total (was 194) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 20m 7s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 2m 23s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. |
| | | 51m 43s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/5/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3574 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux f5744711e64e 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 44d5624908 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Max. process+thread count | 85 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/5/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
Apache9 commented on a change in pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#discussion_r687847994
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/SplitTableRegionProcedure.java
##########
@@ -648,9 +656,11 @@ public void createDaughterRegions(final MasterProcedureEnv env) throws IOExcepti
// there's files to split. It then fires up everything, waits for
// completion and finally checks for any exception
//
- // Note: splitStoreFiles creates daughter region dirs under the parent splits dir
- // Nothing to unroll here if failure -- re-run createSplitsDir will
- // clean this up.
+ // Note: From HBASE-26187, splitStoreFiles now creates daughter region dirs straight under the
Review comment:
I suppose so. Unless we could make sure that every time we will generate the reference files with the same name, then we could just create the reference file with overwrite = true. Otherwise I think we'd better clean the directory and recreate it...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#issuecomment-898071375
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 3s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 7s | master passed |
| +1 :green_heart: | compile | 1m 18s | master passed |
| +1 :green_heart: | shadedjars | 9m 8s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 43s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 48s | the patch passed |
| +1 :green_heart: | compile | 1m 19s | the patch passed |
| +1 :green_heart: | javac | 1m 19s | the patch passed |
| +1 :green_heart: | shadedjars | 8m 58s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 42s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 202m 2s | hbase-server in the patch passed. |
| | | 237m 6s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3574 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 6e0a68a86607 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 11222fc4df |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/4/testReport/ |
| Max. process+thread count | 2644 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/4/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#issuecomment-899696004
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 3s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 4s | master passed |
| +1 :green_heart: | compile | 1m 4s | master passed |
| +1 :green_heart: | shadedjars | 9m 1s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 37s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 7s | the patch passed |
| +1 :green_heart: | compile | 1m 4s | the patch passed |
| +1 :green_heart: | javac | 1m 4s | the patch passed |
| +1 :green_heart: | shadedjars | 9m 3s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 37s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 208m 14s | hbase-server in the patch passed. |
| | | 240m 53s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3574 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 9cc07f8e6b64 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 44d5624908 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/6/testReport/ |
| Max. process+thread count | 2512 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/6/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
Apache9 commented on a change in pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#discussion_r688434905
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
##########
@@ -638,47 +598,36 @@ void cleanupDaughterRegion(final RegionInfo regionInfo) throws IOException {
*/
public Path commitDaughterRegion(final RegionInfo regionInfo)
throws IOException {
- Path regionDir = new Path(this.tableDir, regionInfo.getEncodedName());
- Path daughterTmpDir = this.getSplitsDir(regionInfo);
-
- if (fs.exists(daughterTmpDir)) {
-
+ Path regionDir = this.getSplitsDir(regionInfo);
+ if (fs.exists(regionDir)) {
// Write HRI to a file in case we need to recover hbase:meta
- Path regionInfoFile = new Path(daughterTmpDir, REGION_INFO_FILE);
+ Path regionInfoFile = new Path(regionDir, REGION_INFO_FILE);
byte[] regionInfoContent = getRegionInfoFileContent(regionInfo);
writeRegionInfoFileContent(conf, fs, regionInfoFile, regionInfoContent);
-
- // Move the daughter temp dir to the table dir
- if (!rename(daughterTmpDir, regionDir)) {
- throw new IOException("Unable to rename " + daughterTmpDir + " to " + regionDir);
- }
}
return regionDir;
}
/**
- * Create the region splits directory.
+ * Creates region split daughter directories under the table dir. If the daughter regions already
+ * exist, for example, in the case of a recovery from a previous failed split procedure, this
+ * method deletes the given region dir recursively, then recreates it again.
*/
public void createSplitsDir(RegionInfo daughterA, RegionInfo daughterB) throws IOException {
- Path splitdir = getSplitsDir();
- if (fs.exists(splitdir)) {
- LOG.info("The " + splitdir + " directory exists. Hence deleting it to recreate it");
- if (!deleteDir(splitdir)) {
- throw new IOException("Failed deletion of " + splitdir + " before creating them again.");
- }
+ Path daughterADir = getSplitsDir(daughterA);
+ if (fs.exists(daughterADir)) {
+ fs.delete(daughterADir, true);
Review comment:
We do not use the deleteDir method in HRegionFileSystem? Seems it is better to keep the old way, test the return value of deleteDir and throw IOException?
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionFileSystem.java
##########
@@ -70,6 +70,7 @@
private static final Logger LOG = LoggerFactory.getLogger(TestHRegionFileSystem.class);
public static final byte[] FAMILY_NAME = Bytes.toBytes("info");
+
Review comment:
Do not need to modify this file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] wchevreuil commented on pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
wchevreuil commented on pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#issuecomment-897583443
> After reviewing the code for CatalogJanitor, I do not think it will clean the partial regions on FileSystem, as I haven't seen any related code about scanning the table directory on FileSystem, it just scans the meta table.
>
> So in general, I think we should change the rollback code in SplitTableRegionProcedure and MergeTableRegionsProcedure, to remove the paritial regions when rolling back.
>
> And I think we should also have a tool in HBCK2 to remove the partial regions.
>
> Thanks.
Yep, had just found the same. Had added the cleanup logic on the rollback. I will work on hbck2 on a separate issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
Apache9 commented on a change in pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#discussion_r686450336
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
##########
@@ -638,47 +602,33 @@ void cleanupDaughterRegion(final RegionInfo regionInfo) throws IOException {
*/
public Path commitDaughterRegion(final RegionInfo regionInfo)
throws IOException {
- Path regionDir = new Path(this.tableDir, regionInfo.getEncodedName());
- Path daughterTmpDir = this.getSplitsDir(regionInfo);
-
- if (fs.exists(daughterTmpDir)) {
-
+ Path regionDir = this.getSplitsDir(regionInfo);
+ if (fs.exists(regionDir)) {
// Write HRI to a file in case we need to recover hbase:meta
- Path regionInfoFile = new Path(daughterTmpDir, REGION_INFO_FILE);
+ Path regionInfoFile = new Path(regionDir, REGION_INFO_FILE);
byte[] regionInfoContent = getRegionInfoFileContent(regionInfo);
writeRegionInfoFileContent(conf, fs, regionInfoFile, regionInfoContent);
-
- // Move the daughter temp dir to the table dir
- if (!rename(daughterTmpDir, regionDir)) {
- throw new IOException("Unable to rename " + daughterTmpDir + " to " + regionDir);
- }
}
return regionDir;
}
/**
- * Create the region splits directory.
+ * Creates region split daughter directories under the table dir.
*/
public void createSplitsDir(RegionInfo daughterA, RegionInfo daughterB) throws IOException {
Path splitdir = getSplitsDir();
- if (fs.exists(splitdir)) {
- LOG.info("The " + splitdir + " directory exists. Hence deleting it to recreate it");
- if (!deleteDir(splitdir)) {
- throw new IOException("Failed deletion of " + splitdir + " before creating them again.");
- }
- }
// splitDir doesn't exists now. No need to do an exists() call for it.
- if (!createDir(splitdir)) {
- throw new IOException("Failed create of " + splitdir);
+ if (!fs.exists(splitdir)) {
Review comment:
So the split directory is now the table directory, why we still need to check existence here? And IMO here we should not use 'splitDir' and 'mergeDir' in code any more, as now there are only region directories...
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
##########
@@ -638,47 +602,33 @@ void cleanupDaughterRegion(final RegionInfo regionInfo) throws IOException {
*/
public Path commitDaughterRegion(final RegionInfo regionInfo)
throws IOException {
- Path regionDir = new Path(this.tableDir, regionInfo.getEncodedName());
- Path daughterTmpDir = this.getSplitsDir(regionInfo);
-
- if (fs.exists(daughterTmpDir)) {
-
+ Path regionDir = this.getSplitsDir(regionInfo);
+ if (fs.exists(regionDir)) {
// Write HRI to a file in case we need to recover hbase:meta
- Path regionInfoFile = new Path(daughterTmpDir, REGION_INFO_FILE);
+ Path regionInfoFile = new Path(regionDir, REGION_INFO_FILE);
byte[] regionInfoContent = getRegionInfoFileContent(regionInfo);
writeRegionInfoFileContent(conf, fs, regionInfoFile, regionInfoContent);
-
- // Move the daughter temp dir to the table dir
- if (!rename(daughterTmpDir, regionDir)) {
- throw new IOException("Unable to rename " + daughterTmpDir + " to " + regionDir);
- }
}
return regionDir;
}
/**
- * Create the region splits directory.
+ * Creates region split daughter directories under the table dir.
*/
public void createSplitsDir(RegionInfo daughterA, RegionInfo daughterB) throws IOException {
Path splitdir = getSplitsDir();
- if (fs.exists(splitdir)) {
- LOG.info("The " + splitdir + " directory exists. Hence deleting it to recreate it");
- if (!deleteDir(splitdir)) {
- throw new IOException("Failed deletion of " + splitdir + " before creating them again.");
- }
- }
// splitDir doesn't exists now. No need to do an exists() call for it.
Review comment:
This comment is incorrect now?
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
##########
@@ -572,51 +572,15 @@ public void removeStoreFiles(String familyName, Collection<HStoreFile> storeFile
// ===========================================================================
// Splits Helpers
// ===========================================================================
- /** @return {@link Path} to the temp directory used during split operations */
+ /** @return {@link Path} to the table directory for split daughters */
Path getSplitsDir() {
Review comment:
Do we still need this method?
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionFileSystem.java
##########
@@ -380,4 +382,190 @@ public void testTempAndCommit() throws IOException {
fs.delete(rootDir, true);
}
+
+ @Test
+ public void testSplitStoreDir() throws Exception {
Review comment:
Better place the below methods to a different test class? And is it possible to only start mini cluster once? Now every test method needs to start and then stop the mini cluster, which is time consuming...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#issuecomment-898083187
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 59s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 8s | master passed |
| +1 :green_heart: | compile | 1m 3s | master passed |
| +1 :green_heart: | shadedjars | 8m 59s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 37s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 8s | the patch passed |
| +1 :green_heart: | compile | 1m 4s | the patch passed |
| +1 :green_heart: | javac | 1m 4s | the patch passed |
| +1 :green_heart: | shadedjars | 9m 5s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 36s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 220m 11s | hbase-server in the patch passed. |
| | | 253m 27s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3574 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 27ccc1febfd9 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 11222fc4df |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/4/testReport/ |
| Max. process+thread count | 2555 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/4/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a change in pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
wchevreuil commented on a change in pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#discussion_r688079612
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/SplitTableRegionProcedure.java
##########
@@ -648,9 +656,11 @@ public void createDaughterRegions(final MasterProcedureEnv env) throws IOExcepti
// there's files to split. It then fires up everything, waits for
// completion and finally checks for any exception
//
- // Note: splitStoreFiles creates daughter region dirs under the parent splits dir
- // Nothing to unroll here if failure -- re-run createSplitsDir will
- // clean this up.
+ // Note: From HBASE-26187, splitStoreFiles now creates daughter region dirs straight under the
Review comment:
Ok, added extra check on HRegionFileSystem.createSplitsDir to delete the daughter regions if those already exist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#issuecomment-899519827
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 3s | Docker mode activated. |
| -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 9s | master passed |
| +1 :green_heart: | compile | 1m 4s | master passed |
| +1 :green_heart: | shadedjars | 9m 3s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 37s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 6s | the patch passed |
| +1 :green_heart: | compile | 1m 4s | the patch passed |
| +1 :green_heart: | javac | 1m 4s | the patch passed |
| +1 :green_heart: | shadedjars | 9m 1s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 38s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 210m 43s | hbase-server in the patch passed. |
| | | 243m 27s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3574 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux caf96e48cf4a 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 44d5624908 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/5/testReport/ |
| Max. process+thread count | 2570 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/5/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a change in pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
wchevreuil commented on a change in pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#discussion_r687844530
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/SplitTableRegionProcedure.java
##########
@@ -648,9 +656,11 @@ public void createDaughterRegions(final MasterProcedureEnv env) throws IOExcepti
// there's files to split. It then fires up everything, waits for
// completion and finally checks for any exception
//
- // Note: splitStoreFiles creates daughter region dirs under the parent splits dir
- // Nothing to unroll here if failure -- re-run createSplitsDir will
- // clean this up.
+ // Note: From HBASE-26187, splitStoreFiles now creates daughter region dirs straight under the
Review comment:
Yeah, let me remove the CatalogJanitor mention.
>I think we should just recreate the whole region directly when we call this method?
Like, first we check if the given daughter region dir already exists and if so, delete it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a change in pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
wchevreuil commented on a change in pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#discussion_r686873333
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionFileSystem.java
##########
@@ -380,4 +382,190 @@ public void testTempAndCommit() throws IOException {
fs.delete(rootDir, true);
}
+
+ @Test
+ public void testSplitStoreDir() throws Exception {
Review comment:
Ok, let me move these to a separate class and do the mini cluster init only once.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#issuecomment-897780559
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 28s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 15s | master passed |
| +1 :green_heart: | compile | 1m 13s | master passed |
| +1 :green_heart: | shadedjars | 8m 13s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 40s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 15s | the patch passed |
| +1 :green_heart: | compile | 1m 14s | the patch passed |
| +1 :green_heart: | javac | 1m 14s | the patch passed |
| +1 :green_heart: | shadedjars | 8m 4s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 41s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 134m 39s | hbase-server in the patch passed. |
| | | 166m 4s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3574 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 1c0c3ba70237 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 238c9b40bf |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/3/testReport/ |
| Max. process+thread count | 3837 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/3/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#issuecomment-897578019
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 27s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 33s | master passed |
| +1 :green_heart: | compile | 1m 11s | master passed |
| +1 :green_heart: | shadedjars | 8m 19s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 44s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 21s | the patch passed |
| +1 :green_heart: | compile | 1m 12s | the patch passed |
| +1 :green_heart: | javac | 1m 12s | the patch passed |
| +1 :green_heart: | shadedjars | 8m 16s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 40s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 134m 50s | hbase-server in the patch passed. |
| | | 167m 0s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3574 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux d99d1303660b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 238c9b40bf |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/2/testReport/ |
| Max. process+thread count | 3862 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/2/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#issuecomment-896339504
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 4s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 6s | master passed |
| +1 :green_heart: | compile | 1m 17s | master passed |
| +1 :green_heart: | shadedjars | 9m 9s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 44s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 48s | the patch passed |
| +1 :green_heart: | compile | 1m 16s | the patch passed |
| +1 :green_heart: | javac | 1m 16s | the patch passed |
| +1 :green_heart: | shadedjars | 9m 4s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 41s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 204m 15s | hbase-server in the patch failed. |
| | | 239m 24s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3574 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 19e0622aeb59 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 5e8a269b1a |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/1/testReport/ |
| Max. process+thread count | 2647 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3574/1/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] wchevreuil commented on pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
wchevreuil commented on pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#issuecomment-897477826
One thought I had now is that we probably need to back off CatalogJanitor from running whilst a split/merge is going on. As we are creating the dirs under table directory, I guess there's a risk CatalogJanitor wipe those dirs off before the proc reaches the point to add the region in meta. Let me check on this futher.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a change in pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
wchevreuil commented on a change in pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#discussion_r686200903
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
##########
@@ -572,51 +572,15 @@ public void removeStoreFiles(String familyName, Collection<HStoreFile> storeFile
// ===========================================================================
// Splits Helpers
// ===========================================================================
- /** @return {@link Path} to the temp directory used during split operations */
+ /** @return {@link Path} to the table directory for split daughters */
Path getSplitsDir() {
- return new Path(getRegionDir(), REGION_SPLITS_DIR);
+ return getTableDir();
}
public Path getSplitsDir(final RegionInfo hri) {
return new Path(getSplitsDir(), hri.getEncodedName());
}
- /**
- * Clean up any split detritus that may have been left around from previous split attempts.
- */
- void cleanupSplitsDir() throws IOException {
- deleteDir(getSplitsDir());
- }
-
- /**
- * Clean up any split detritus that may have been left around from previous
- * split attempts.
- * Call this method on initial region deploy.
- * @throws IOException
- */
- void cleanupAnySplitDetritus() throws IOException {
- Path splitdir = this.getSplitsDir();
- if (!fs.exists(splitdir)) return;
- // Look at the splitdir. It could have the encoded names of the daughter
- // regions we tried to make. See if the daughter regions actually got made
- // out under the tabledir. If here under splitdir still, then the split did
- // not complete. Try and do cleanup. This code WILL NOT catch the case
- // where we successfully created daughter a but regionserver crashed during
- // the creation of region b. In this case, there'll be an orphan daughter
- // dir in the filesystem. TOOD: Fix.
- FileStatus[] daughters = CommonFSUtils.listStatus(fs, splitdir, new FSUtils.DirFilter(fs));
- if (daughters != null) {
- for (FileStatus daughter: daughters) {
- Path daughterDir = new Path(getTableDir(), daughter.getPath().getName());
- if (fs.exists(daughterDir) && !deleteDir(daughterDir)) {
- throw new IOException("Failed delete of " + daughterDir);
- }
- }
- }
- cleanupSplitsDir();
- LOG.info("Cleaned up old failed split transaction detritus: " + splitdir);
- }
Review comment:
Since we won't be using temp dirs anymore, these methods would have no use. Any potential failed split/merge dir would be cleaned by CatalogJanitor, once it finds no related entry in meta for these regions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] wchevreuil edited a comment on pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
wchevreuil edited a comment on pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#issuecomment-897477826
One thought I had now is that we probably need to back off CatalogJanitor from running whilst a split/merge is going on. As we are creating the dirs under table directory, I guess there's a risk CatalogJanitor wipe those dirs off before the proc reaches the point to add the region in meta. Let me check on this further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
Apache9 commented on pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#issuecomment-897519279
After reviewing the code for CatalogJanitor, I do not think it will clean the partial regions on FileSystem, as I haven't seen any related code about scanning the table directory on FileSystem, it just scans the meta table.
So in general, I think we should change the rollback code in SplitTableRegionProcedure and MergeTableRegionsProcedure, to remove the paritial regions when rolling back.
And I think we should also have a tool in HBCK2 to remove the partial regions.
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a change in pull request #3574: HBASE-26187 Write straight into the store directory when Splitting an…
Posted by GitBox <gi...@apache.org>.
wchevreuil commented on a change in pull request #3574:
URL: https://github.com/apache/hbase/pull/3574#discussion_r686814909
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
##########
@@ -572,51 +572,15 @@ public void removeStoreFiles(String familyName, Collection<HStoreFile> storeFile
// ===========================================================================
// Splits Helpers
// ===========================================================================
- /** @return {@link Path} to the temp directory used during split operations */
+ /** @return {@link Path} to the table directory for split daughters */
Path getSplitsDir() {
Review comment:
Not really, let me remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org