You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2021/10/21 11:56:29 UTC
[GitHub] [hbase] comnetwork opened a new pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
comnetwork opened a new pull request #3777:
URL: https://github.com/apache/hbase/pull/3777
… CompactingMemStore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#issuecomment-949518758
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 4s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 16s | master passed |
| +1 :green_heart: | compile | 1m 5s | master passed |
| +1 :green_heart: | shadedjars | 9m 5s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 36s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 15s | the patch passed |
| +1 :green_heart: | compile | 1m 5s | the patch passed |
| +1 :green_heart: | javac | 1m 5s | the patch passed |
| +1 :green_heart: | shadedjars | 9m 4s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 36s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 216m 43s | hbase-server in the patch passed. |
| | | 249m 44s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3777 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 62554ae46186 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / e0813e5402 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/3/testReport/ |
| Max. process+thread count | 3203 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/3/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#issuecomment-952867291
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 26s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 48s | master passed |
| +1 :green_heart: | compile | 3m 8s | master passed |
| +1 :green_heart: | checkstyle | 1m 4s | master passed |
| +1 :green_heart: | spotbugs | 2m 0s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 47s | the patch passed |
| +1 :green_heart: | compile | 3m 9s | the patch passed |
| +1 :green_heart: | javac | 3m 9s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 4s | hbase-server: The patch generated 0 new + 34 unchanged - 1 fixed = 34 total (was 35) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 19m 4s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 2m 15s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. |
| | | 48m 13s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/6/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3777 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 9f40feff0bd4 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 9b7ba7f300 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Max. process+thread count | 95 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/6/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] comnetwork commented on a change in pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
comnetwork commented on a change in pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#discussion_r737341407
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionPipeline.java
##########
@@ -95,7 +103,7 @@ public VersionedSegmentsList getVersionedList() {
public VersionedSegmentsList getVersionedTail() {
synchronized (pipeline){
- List<ImmutableSegment> segmentList = new ArrayList<>();
+ LinkedList<ImmutableSegment> segmentList = new LinkedList<>();
Review comment:
Ok, thank you for review, I would to try to fix the List problem,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] comnetwork commented on pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
comnetwork commented on pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#issuecomment-952841190
@Apache9 , thank you for review, I modified the LinkedList problem and test comments problem following your suggestions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] comnetwork commented on a change in pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
comnetwork commented on a change in pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#discussion_r740113588
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHStore.java
##########
@@ -1921,6 +1922,225 @@ private void doWriteTestLargeCellAndSmallCellConcurrently(
}
}
+ /**
+ * <pre>
+ * This test is for HBASE-26384,
+ * test {@link CompactingMemStore#flattenOneSegment} and {@link CompactingMemStore#snapshot()}
+ * execute concurrently.
+ * The threads sequence before HBASE-26384 is(The bug only exists for branch-2,and I add UTs
+ * for both branch-2 and master):
+ * 1. The {@link CompactingMemStore} size exceeds
+ * {@link CompactingMemStore#getInmemoryFlushSize()},the write thread adds a new
+ * {@link ImmutableSegment} to the head of {@link CompactingMemStore#pipeline},and start a
+ * in memory compact thread to execute {@link CompactingMemStore#inMemoryCompaction}.
+ * 2. The in memory compact thread starts and then stopping before
+ * {@link CompactingMemStore#flattenOneSegment}.
+ * 3. The snapshot thread starts {@link CompactingMemStore#snapshot} concurrently,after the
+ * snapshot thread executing {@link CompactingMemStore#getImmutableSegments},the in memory
+ * compact thread continues.
+ * Assuming {@link VersionedSegmentsList#version} returned from
+ * {@link CompactingMemStore#getImmutableSegments} is v.
+ * 4. The snapshot thread stopping before {@link CompactingMemStore#swapPipelineWithNull}.
+ * 5. The in memory compact thread completes {@link CompactingMemStore#flattenOneSegment},
+ * {@link CompactionPipeline#version} is still v.
+ * 6. The snapshot thread continues {@link CompactingMemStore#swapPipelineWithNull}, and because
+ * {@link CompactionPipeline#version} is v, {@link CompactingMemStore#swapPipelineWithNull}
+ * thinks it is successful and continue flushing,but the {@link ImmutableSegment} in
+ * {@link CompactionPipeline} has changed because
+ * {@link CompactingMemStore#flattenOneSegment},so the {@link ImmutableSegment} is not
+ * removed in fact and still remaining in {@link CompactionPipeline}.
+ *
+ * After HBASE-26384, the 5-6 step is changed to following, which is expected behavior:
+ * 5. The in memory compact thread completes {@link CompactingMemStore#flattenOneSegment},
+ * {@link CompactingMemStore#flattenOneSegment} change {@link CompactionPipeline#version} to
+ * v+1.
Review comment:
Yes ，I explain this at the end of the JIRA description， because this `version `bug for `CompactingMemStore#flattenOneSegment` is only exists for branch-2， for master，`version` is correctly increased for `CompactingMemStore#flattenOneSegment` ，so I made separate PRs for master(#3777) and branch-2(#3779) , so the code for master and brach-2 could keep consistent and master could also has UTs for this problem.
and I included folowing comments at the head of this test method comments:
```
/*....
* The threads sequence before HBASE-26384 is(**The bug only exists for branch-2,and I add UTs
* for both branch-2 and master**):
* .../
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] comnetwork commented on a change in pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
comnetwork commented on a change in pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#discussion_r737341407
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionPipeline.java
##########
@@ -95,7 +103,7 @@ public VersionedSegmentsList getVersionedList() {
public VersionedSegmentsList getVersionedTail() {
synchronized (pipeline){
- List<ImmutableSegment> segmentList = new ArrayList<>();
+ LinkedList<ImmutableSegment> segmentList = new LinkedList<>();
Review comment:
Ok, tank you for review, I would to try to fix the List problem,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#issuecomment-948584849
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 27s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 16s | master passed |
| +1 :green_heart: | compile | 3m 18s | master passed |
| +1 :green_heart: | checkstyle | 1m 5s | master passed |
| +1 :green_heart: | spotbugs | 2m 9s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 54s | the patch passed |
| +1 :green_heart: | compile | 3m 18s | the patch passed |
| +1 :green_heart: | javac | 3m 18s | the patch passed |
| -0 :warning: | checkstyle | 1m 3s | hbase-server: The patch generated 1 new + 34 unchanged - 1 fixed = 35 total (was 35) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 19m 23s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 2m 17s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. |
| | | 49m 43s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3777 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 0af33e2483e4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 64eb237120 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt |
| Max. process+thread count | 96 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/1/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#issuecomment-948610669
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 3m 57s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 8m 44s | master passed |
| +1 :green_heart: | compile | 2m 11s | master passed |
| +1 :green_heart: | shadedjars | 14m 18s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 21s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 8m 33s | the patch passed |
| +1 :green_heart: | compile | 2m 37s | the patch passed |
| +1 :green_heart: | javac | 2m 37s | the patch passed |
| +1 :green_heart: | shadedjars | 11m 55s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 52s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 20m 22s | hbase-server in the patch failed. |
| | | 76m 54s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3777 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 2b70a04ce0bc 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 64eb237120 |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/1/testReport/ |
| Max. process+thread count | 660 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/1/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] comnetwork commented on a change in pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
comnetwork commented on a change in pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#discussion_r737075750
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionPipeline.java
##########
@@ -95,7 +103,7 @@ public VersionedSegmentsList getVersionedList() {
public VersionedSegmentsList getVersionedTail() {
synchronized (pipeline){
- List<ImmutableSegment> segmentList = new ArrayList<>();
+ LinkedList<ImmutableSegment> segmentList = new LinkedList<>();
Review comment:
@Apache9 ， thank you very much for review，Here I change to `LinkedList `because I expect `VersionedSegmentsList.storeSegments` is `LinkedList`， so in the following `CompactionPipline.matchAndRemoveSuffixFromPipeline` I could use `LinkedList.descendingIterator` to to simplify code.
I think here change to LinkedList is somwhat ok, because:
1. Here the `List `has only one element, `LinkedList `and `ArrayList `make no difference.
2. Furthermore, the` CompactionPipeline.pipeline` and `CompactionPipeline.readOnlyCopy` which used in` CompactionPipeline.getVersionedList` are all LinkedList, that is to say, `VersionedSegmentsList.storeSegments` is actually LinkedList in fact(`CompactionPipline.getVersionedTail` is only used for test), so here I change to LinkedList could also make the code looks consistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
Apache9 commented on a change in pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#discussion_r740127643
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHStore.java
##########
@@ -1921,6 +1922,225 @@ private void doWriteTestLargeCellAndSmallCellConcurrently(
}
}
+ /**
+ * <pre>
+ * This test is for HBASE-26384,
+ * test {@link CompactingMemStore#flattenOneSegment} and {@link CompactingMemStore#snapshot()}
+ * execute concurrently.
+ * The threads sequence before HBASE-26384 is(The bug only exists for branch-2,and I add UTs
+ * for both branch-2 and master):
+ * 1. The {@link CompactingMemStore} size exceeds
+ * {@link CompactingMemStore#getInmemoryFlushSize()},the write thread adds a new
+ * {@link ImmutableSegment} to the head of {@link CompactingMemStore#pipeline},and start a
+ * in memory compact thread to execute {@link CompactingMemStore#inMemoryCompaction}.
+ * 2. The in memory compact thread starts and then stopping before
+ * {@link CompactingMemStore#flattenOneSegment}.
+ * 3. The snapshot thread starts {@link CompactingMemStore#snapshot} concurrently,after the
+ * snapshot thread executing {@link CompactingMemStore#getImmutableSegments},the in memory
+ * compact thread continues.
+ * Assuming {@link VersionedSegmentsList#version} returned from
+ * {@link CompactingMemStore#getImmutableSegments} is v.
+ * 4. The snapshot thread stopping before {@link CompactingMemStore#swapPipelineWithNull}.
+ * 5. The in memory compact thread completes {@link CompactingMemStore#flattenOneSegment},
+ * {@link CompactionPipeline#version} is still v.
+ * 6. The snapshot thread continues {@link CompactingMemStore#swapPipelineWithNull}, and because
+ * {@link CompactionPipeline#version} is v, {@link CompactingMemStore#swapPipelineWithNull}
+ * thinks it is successful and continue flushing,but the {@link ImmutableSegment} in
+ * {@link CompactionPipeline} has changed because
+ * {@link CompactingMemStore#flattenOneSegment},so the {@link ImmutableSegment} is not
+ * removed in fact and still remaining in {@link CompactionPipeline}.
+ *
+ * After HBASE-26384, the 5-6 step is changed to following, which is expected behavior:
+ * 5. The in memory compact thread completes {@link CompactingMemStore#flattenOneSegment},
+ * {@link CompactingMemStore#flattenOneSegment} change {@link CompactionPipeline#version} to
+ * v+1.
Review comment:
OK, checked the commit history for a while, I think the problem is that, for master and branch-2, we have different commit order for HBASE-18375 and HBASE-18010, while for branch-2, we missed the version++ line...
Good catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#issuecomment-953253364
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 1s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 14s | master passed |
| +1 :green_heart: | compile | 1m 3s | master passed |
| +1 :green_heart: | shadedjars | 9m 3s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 37s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 18s | the patch passed |
| +1 :green_heart: | compile | 1m 4s | the patch passed |
| +1 :green_heart: | javac | 1m 4s | the patch passed |
| +1 :green_heart: | shadedjars | 9m 2s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 37s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 211m 38s | hbase-server in the patch passed. |
| | | 244m 36s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/7/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3777 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux b232c6085547 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 45f76a4104 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/7/testReport/ |
| Max. process+thread count | 2798 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/7/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] comnetwork commented on a change in pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
comnetwork commented on a change in pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#discussion_r740113588
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHStore.java
##########
@@ -1921,6 +1922,225 @@ private void doWriteTestLargeCellAndSmallCellConcurrently(
}
}
+ /**
+ * <pre>
+ * This test is for HBASE-26384,
+ * test {@link CompactingMemStore#flattenOneSegment} and {@link CompactingMemStore#snapshot()}
+ * execute concurrently.
+ * The threads sequence before HBASE-26384 is(The bug only exists for branch-2,and I add UTs
+ * for both branch-2 and master):
+ * 1. The {@link CompactingMemStore} size exceeds
+ * {@link CompactingMemStore#getInmemoryFlushSize()},the write thread adds a new
+ * {@link ImmutableSegment} to the head of {@link CompactingMemStore#pipeline},and start a
+ * in memory compact thread to execute {@link CompactingMemStore#inMemoryCompaction}.
+ * 2. The in memory compact thread starts and then stopping before
+ * {@link CompactingMemStore#flattenOneSegment}.
+ * 3. The snapshot thread starts {@link CompactingMemStore#snapshot} concurrently,after the
+ * snapshot thread executing {@link CompactingMemStore#getImmutableSegments},the in memory
+ * compact thread continues.
+ * Assuming {@link VersionedSegmentsList#version} returned from
+ * {@link CompactingMemStore#getImmutableSegments} is v.
+ * 4. The snapshot thread stopping before {@link CompactingMemStore#swapPipelineWithNull}.
+ * 5. The in memory compact thread completes {@link CompactingMemStore#flattenOneSegment},
+ * {@link CompactionPipeline#version} is still v.
+ * 6. The snapshot thread continues {@link CompactingMemStore#swapPipelineWithNull}, and because
+ * {@link CompactionPipeline#version} is v, {@link CompactingMemStore#swapPipelineWithNull}
+ * thinks it is successful and continue flushing,but the {@link ImmutableSegment} in
+ * {@link CompactionPipeline} has changed because
+ * {@link CompactingMemStore#flattenOneSegment},so the {@link ImmutableSegment} is not
+ * removed in fact and still remaining in {@link CompactionPipeline}.
+ *
+ * After HBASE-26384, the 5-6 step is changed to following, which is expected behavior:
+ * 5. The in memory compact thread completes {@link CompactingMemStore#flattenOneSegment},
+ * {@link CompactingMemStore#flattenOneSegment} change {@link CompactionPipeline#version} to
+ * v+1.
Review comment:
Yes ，I explain this at the end of the JIRA description， because this `version `bug for `CompactingMemStore#flattenOneSegment` is only exists for branch-2， for master，`version` is correctly increased for `CompactingMemStore#flattenOneSegment` ，so I made separate PRs for master(#3777) and branch-2(#3779) , so the code for master and brach-2 could keep consistent and master could also has UTs for this problem.
and I included folowing comments at the head of this test method comments:
/*....
* The threads sequence before HBASE-26384 is(**The bug only exists for branch-2,and I add UTs
* for both branch-2 and master**):
* .../
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#issuecomment-949333831
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 4s | Docker mode activated. |
| -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 39s | master passed |
| +1 :green_heart: | compile | 1m 4s | master passed |
| +1 :green_heart: | shadedjars | 9m 7s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 39s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 16s | the patch passed |
| +1 :green_heart: | compile | 1m 3s | the patch passed |
| +1 :green_heart: | javac | 1m 3s | the patch passed |
| +1 :green_heart: | shadedjars | 9m 7s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 37s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 214m 29s | hbase-server in the patch passed. |
| | | 248m 6s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3777 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 108747cc3ea1 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / e0813e5402 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/2/testReport/ |
| Max. process+thread count | 3885 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/2/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#issuecomment-949544003
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 36s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 56s | master passed |
| +1 :green_heart: | compile | 1m 35s | master passed |
| +1 :green_heart: | shadedjars | 10m 34s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 0s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 6m 31s | the patch passed |
| +1 :green_heart: | compile | 1m 45s | the patch passed |
| +1 :green_heart: | javac | 1m 45s | the patch passed |
| +1 :green_heart: | shadedjars | 10m 49s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 58s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 237m 48s | hbase-server in the patch failed. |
| | | 280m 32s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3777 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 97221f3874e6 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / e0813e5402 |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/3/testReport/ |
| Max. process+thread count | 3193 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/3/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] comnetwork commented on pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
comnetwork commented on pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#issuecomment-950044721
Seems error not related to this patch, rerun checking
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#issuecomment-950106319
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 1s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 13s | master passed |
| +1 :green_heart: | compile | 1m 2s | master passed |
| +1 :green_heart: | shadedjars | 9m 10s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 38s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 15s | the patch passed |
| +1 :green_heart: | compile | 1m 3s | the patch passed |
| +1 :green_heart: | javac | 1m 3s | the patch passed |
| +1 :green_heart: | shadedjars | 9m 12s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 36s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 212m 18s | hbase-server in the patch passed. |
| | | 245m 29s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3777 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 86850259ada4 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 9e47358ecb |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/4/testReport/ |
| Max. process+thread count | 3630 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/4/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#issuecomment-948778020
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 13s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 56s | master passed |
| +1 :green_heart: | compile | 1m 13s | master passed |
| +1 :green_heart: | shadedjars | 9m 10s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 38s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 31s | the patch passed |
| +1 :green_heart: | compile | 1m 9s | the patch passed |
| +1 :green_heart: | javac | 1m 9s | the patch passed |
| +1 :green_heart: | shadedjars | 9m 36s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 39s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 227m 41s | hbase-server in the patch failed. |
| | | 262m 47s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3777 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux a6e56ffcef01 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 64eb237120 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/1/testReport/ |
| Max. process+thread count | 3366 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/1/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#issuecomment-953259852
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 1s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 24s | master passed |
| +1 :green_heart: | compile | 1m 18s | master passed |
| +1 :green_heart: | shadedjars | 9m 8s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 46s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 15s | the patch passed |
| +1 :green_heart: | compile | 1m 24s | the patch passed |
| +1 :green_heart: | javac | 1m 24s | the patch passed |
| +1 :green_heart: | shadedjars | 9m 40s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 48s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 217m 52s | hbase-server in the patch passed. |
| | | 254m 48s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/7/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3777 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 38ab6abb03f5 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 45f76a4104 |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/7/testReport/ |
| Max. process+thread count | 2743 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/7/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#issuecomment-949257288
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 25s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 11s | master passed |
| +1 :green_heart: | compile | 3m 14s | master passed |
| +1 :green_heart: | checkstyle | 1m 4s | master passed |
| +1 :green_heart: | spotbugs | 2m 10s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 56s | the patch passed |
| +1 :green_heart: | compile | 3m 13s | the patch passed |
| +1 :green_heart: | javac | 3m 13s | the patch passed |
| -0 :warning: | checkstyle | 1m 3s | hbase-server: The patch generated 1 new + 34 unchanged - 1 fixed = 35 total (was 35) |
| +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 19m 29s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 2m 18s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. |
| | | 49m 36s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3777 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 5167b3a23e1d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / e0813e5402 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt |
| Max. process+thread count | 95 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/2/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#issuecomment-949331849
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 2s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 18s | master passed |
| +1 :green_heart: | compile | 1m 19s | master passed |
| +1 :green_heart: | shadedjars | 9m 9s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 44s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 3s | the patch passed |
| +1 :green_heart: | compile | 1m 23s | the patch passed |
| +1 :green_heart: | javac | 1m 23s | the patch passed |
| +1 :green_heart: | shadedjars | 8m 57s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 42s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 208m 18s | hbase-server in the patch passed. |
| | | 243m 51s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3777 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 521edc68eda0 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / e0813e5402 |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/2/testReport/ |
| Max. process+thread count | 3834 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/2/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#issuecomment-953050420
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 12s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 19s | master passed |
| +1 :green_heart: | compile | 1m 20s | master passed |
| +1 :green_heart: | shadedjars | 9m 16s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 43s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 16s | the patch passed |
| +1 :green_heart: | compile | 1m 24s | the patch passed |
| +1 :green_heart: | javac | 1m 24s | the patch passed |
| +1 :green_heart: | shadedjars | 9m 22s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 43s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 208m 17s | hbase-server in the patch passed. |
| | | 244m 51s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3777 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 5332a93b1d1a 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 9b7ba7f300 |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/6/testReport/ |
| Max. process+thread count | 2351 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/6/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] comnetwork commented on a change in pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
comnetwork commented on a change in pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#discussion_r737075750
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionPipeline.java
##########
@@ -95,7 +103,7 @@ public VersionedSegmentsList getVersionedList() {
public VersionedSegmentsList getVersionedTail() {
synchronized (pipeline){
- List<ImmutableSegment> segmentList = new ArrayList<>();
+ LinkedList<ImmutableSegment> segmentList = new LinkedList<>();
Review comment:
@Apache9 ， thank you very much for review，Here I change to `LinkedList `because I expect `VersionedSegmentsList.storeSegments` is `LinkedList`， so in the following `CompactionPipline.matchAndRemoveSuffixFromPipeline` I could use `LinkedList.descendingIterator` to simplify code.
I think here change to LinkedList is somwhat ok, because:
1. Here the `List `has only one element, `LinkedList `and `ArrayList `make no difference.
2. Furthermore, the` CompactionPipeline.pipeline` and `CompactionPipeline.readOnlyCopy` which used in` CompactionPipeline.getVersionedList` are all LinkedList, that is to say, `VersionedSegmentsList.storeSegments` is actually LinkedList in fact(`CompactionPipline.getVersionedTail` is only used for test), so here change to LinkedList could also make the code looks consistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] comnetwork commented on a change in pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
comnetwork commented on a change in pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#discussion_r737125535
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHStore.java
##########
@@ -1921,6 +1922,202 @@ private void doWriteTestLargeCellAndSmallCellConcurrently(
}
}
+ /**
+ * <pre>
+ * This test is for HBASE-26384,
+ * test {@link CompactingMemStore#flattenOneSegment} and {@link CompactingMemStore#snapshot()}
+ * execute concurrently.
+ * The threads sequence is:
Review comment:
This is the expected behavior,I would modify the comments to explictly state it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] comnetwork commented on a change in pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
comnetwork commented on a change in pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#discussion_r737340539
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHStore.java
##########
@@ -1921,6 +1922,202 @@ private void doWriteTestLargeCellAndSmallCellConcurrently(
}
}
+ /**
+ * <pre>
+ * This test is for HBASE-26384,
+ * test {@link CompactingMemStore#flattenOneSegment} and {@link CompactingMemStore#snapshot()}
+ * execute concurrently.
+ * The threads sequence is:
Review comment:
Ok, thank you for suggestion, I would fix the description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] comnetwork commented on a change in pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
comnetwork commented on a change in pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#discussion_r737340539
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHStore.java
##########
@@ -1921,6 +1922,202 @@ private void doWriteTestLargeCellAndSmallCellConcurrently(
}
}
+ /**
+ * <pre>
+ * This test is for HBASE-26384,
+ * test {@link CompactingMemStore#flattenOneSegment} and {@link CompactingMemStore#snapshot()}
+ * execute concurrently.
+ * The threads sequence is:
Review comment:
Ok, thank you, I would fix the description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] comnetwork commented on a change in pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
comnetwork commented on a change in pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#discussion_r737075750
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionPipeline.java
##########
@@ -95,7 +103,7 @@ public VersionedSegmentsList getVersionedList() {
public VersionedSegmentsList getVersionedTail() {
synchronized (pipeline){
- List<ImmutableSegment> segmentList = new ArrayList<>();
+ LinkedList<ImmutableSegment> segmentList = new LinkedList<>();
Review comment:
@Apache9 ， thank you very much for review，Here I change to `LinkedList `because I expect `VersionedSegmentsList.storeSegments` is `LinkedList`， so in the following `CompactionPipline.matchAndRemoveSuffixFromPipeline` I could use `LinkedList.descendingIterator` to simplify code.
I think here change to LinkedList is somwhat ok, because:
1. Here the returned `VersionedSegmentsList ` has only one tail element, `LinkedList `and `ArrayList `make no difference.
2. Furthermore, the` CompactionPipeline.pipeline` and `CompactionPipeline.readOnlyCopy` which used in` CompactionPipeline.getVersionedList` are all LinkedList, that is to say, `VersionedSegmentsList.storeSegments` is actually LinkedList in fact(`CompactionPipline.getVersionedTail` is only used for test), so here change to LinkedList could also make the code looks consistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#issuecomment-952753793
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 5s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 12s | master passed |
| +1 :green_heart: | compile | 1m 4s | master passed |
| +1 :green_heart: | shadedjars | 9m 6s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 37s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 20s | the patch passed |
| +1 :green_heart: | compile | 1m 5s | the patch passed |
| +1 :green_heart: | javac | 1m 5s | the patch passed |
| +1 :green_heart: | shadedjars | 9m 2s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 36s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 212m 3s | hbase-server in the patch passed. |
| | | 245m 4s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3777 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux f94be66560d6 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 9b7ba7f300 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/5/testReport/ |
| Max. process+thread count | 2777 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/5/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
Apache9 commented on a change in pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#discussion_r737330653
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHStore.java
##########
@@ -1921,6 +1922,202 @@ private void doWriteTestLargeCellAndSmallCellConcurrently(
}
}
+ /**
+ * <pre>
+ * This test is for HBASE-26384,
+ * test {@link CompactingMemStore#flattenOneSegment} and {@link CompactingMemStore#snapshot()}
+ * execute concurrently.
+ * The threads sequence is:
Review comment:
I prefer we explain the broken behavior first and then say what should be the correct behavior, so it will be easier for people to know what we actually fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#issuecomment-953051167
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 2s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 29s | master passed |
| +1 :green_heart: | compile | 1m 5s | master passed |
| +1 :green_heart: | shadedjars | 9m 10s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 40s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 19s | the patch passed |
| +1 :green_heart: | compile | 1m 4s | the patch passed |
| +1 :green_heart: | javac | 1m 4s | the patch passed |
| +1 :green_heart: | shadedjars | 9m 11s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 37s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 212m 8s | hbase-server in the patch passed. |
| | | 245m 40s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3777 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 8b81a089b5f3 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 9b7ba7f300 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/6/testReport/ |
| Max. process+thread count | 2667 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/6/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
Apache9 commented on a change in pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#discussion_r735716202
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionPipeline.java
##########
@@ -95,7 +103,7 @@ public VersionedSegmentsList getVersionedList() {
public VersionedSegmentsList getVersionedTail() {
synchronized (pipeline){
- List<ImmutableSegment> segmentList = new ArrayList<>();
+ LinkedList<ImmutableSegment> segmentList = new LinkedList<>();
Review comment:
Why change to LinkedList? Usually it will be much slower than ArrayList.
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHStore.java
##########
@@ -1921,6 +1922,202 @@ private void doWriteTestLargeCellAndSmallCellConcurrently(
}
}
+ /**
+ * <pre>
+ * This test is for HBASE-26384,
+ * test {@link CompactingMemStore#flattenOneSegment} and {@link CompactingMemStore#snapshot()}
+ * execute concurrently.
+ * The threads sequence is:
Review comment:
This is the expected behavior or the broken behavior?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#issuecomment-950050727
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 27s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 11s | master passed |
| +1 :green_heart: | compile | 3m 14s | master passed |
| +1 :green_heart: | checkstyle | 1m 3s | master passed |
| +1 :green_heart: | spotbugs | 2m 4s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 49s | the patch passed |
| +1 :green_heart: | compile | 3m 9s | the patch passed |
| +1 :green_heart: | javac | 3m 9s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 2s | hbase-server: The patch generated 0 new + 34 unchanged - 1 fixed = 34 total (was 35) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 19m 13s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 2m 14s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. |
| | | 48m 54s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/4/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3777 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 668337a15a88 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 9e47358ecb |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Max. process+thread count | 96 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/4/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] comnetwork commented on a change in pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
comnetwork commented on a change in pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#discussion_r737125535
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHStore.java
##########
@@ -1921,6 +1922,202 @@ private void doWriteTestLargeCellAndSmallCellConcurrently(
}
}
+ /**
+ * <pre>
+ * This test is for HBASE-26384,
+ * test {@link CompactingMemStore#flattenOneSegment} and {@link CompactingMemStore#snapshot()}
+ * execute concurrently.
+ * The threads sequence is:
Review comment:
This is the expected behavior,I modified the comments to explictly state it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#issuecomment-953096953
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 4s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 7s | master passed |
| +1 :green_heart: | compile | 3m 14s | master passed |
| +1 :green_heart: | checkstyle | 1m 3s | master passed |
| +1 :green_heart: | spotbugs | 2m 6s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 47s | the patch passed |
| +1 :green_heart: | compile | 3m 13s | the patch passed |
| +1 :green_heart: | javac | 3m 13s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 3s | hbase-server: The patch generated 0 new + 34 unchanged - 1 fixed = 34 total (was 35) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 19m 21s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 2m 16s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. |
| | | 49m 41s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/7/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3777 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 7518c84c3ffb 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 45f76a4104 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Max. process+thread count | 96 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/7/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#issuecomment-952764867
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 6m 45s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 10s | master passed |
| +1 :green_heart: | compile | 1m 20s | master passed |
| +1 :green_heart: | shadedjars | 9m 10s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 43s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 20s | the patch passed |
| +1 :green_heart: | compile | 1m 26s | the patch passed |
| +1 :green_heart: | javac | 1m 26s | the patch passed |
| +1 :green_heart: | shadedjars | 9m 32s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 48s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 215m 2s | hbase-server in the patch passed. |
| | | 257m 24s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3777 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 9e37fe7d596b 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 9b7ba7f300 |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/5/testReport/ |
| Max. process+thread count | 2371 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/5/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
Apache9 commented on a change in pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#discussion_r740062652
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHStore.java
##########
@@ -1921,6 +1922,225 @@ private void doWriteTestLargeCellAndSmallCellConcurrently(
}
}
+ /**
+ * <pre>
+ * This test is for HBASE-26384,
+ * test {@link CompactingMemStore#flattenOneSegment} and {@link CompactingMemStore#snapshot()}
+ * execute concurrently.
+ * The threads sequence before HBASE-26384 is(The bug only exists for branch-2,and I add UTs
+ * for both branch-2 and master):
+ * 1. The {@link CompactingMemStore} size exceeds
+ * {@link CompactingMemStore#getInmemoryFlushSize()},the write thread adds a new
+ * {@link ImmutableSegment} to the head of {@link CompactingMemStore#pipeline},and start a
+ * in memory compact thread to execute {@link CompactingMemStore#inMemoryCompaction}.
+ * 2. The in memory compact thread starts and then stopping before
+ * {@link CompactingMemStore#flattenOneSegment}.
+ * 3. The snapshot thread starts {@link CompactingMemStore#snapshot} concurrently,after the
+ * snapshot thread executing {@link CompactingMemStore#getImmutableSegments},the in memory
+ * compact thread continues.
+ * Assuming {@link VersionedSegmentsList#version} returned from
+ * {@link CompactingMemStore#getImmutableSegments} is v.
+ * 4. The snapshot thread stopping before {@link CompactingMemStore#swapPipelineWithNull}.
+ * 5. The in memory compact thread completes {@link CompactingMemStore#flattenOneSegment},
+ * {@link CompactionPipeline#version} is still v.
+ * 6. The snapshot thread continues {@link CompactingMemStore#swapPipelineWithNull}, and because
+ * {@link CompactionPipeline#version} is v, {@link CompactingMemStore#swapPipelineWithNull}
+ * thinks it is successful and continue flushing,but the {@link ImmutableSegment} in
+ * {@link CompactionPipeline} has changed because
+ * {@link CompactingMemStore#flattenOneSegment},so the {@link ImmutableSegment} is not
+ * removed in fact and still remaining in {@link CompactionPipeline}.
+ *
+ * After HBASE-26384, the 5-6 step is changed to following, which is expected behavior:
+ * 5. The in memory compact thread completes {@link CompactingMemStore#flattenOneSegment},
+ * {@link CompactingMemStore#flattenOneSegment} change {@link CompactionPipeline#version} to
+ * v+1.
Review comment:
We do not change the implementation for CompactingMemStore#flattenOneSegment? So why in the past we do not increase the version but now we will increase it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 merged pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
Apache9 merged pull request #3777:
URL: https://github.com/apache/hbase/pull/3777
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
Apache9 commented on a change in pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#discussion_r737331379
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionPipeline.java
##########
@@ -95,7 +103,7 @@ public VersionedSegmentsList getVersionedList() {
public VersionedSegmentsList getVersionedTail() {
synchronized (pipeline){
- List<ImmutableSegment> segmentList = new ArrayList<>();
+ LinkedList<ImmutableSegment> segmentList = new LinkedList<>();
Review comment:
If ArrayList is not suitable here, is it possible to make use of other array based data structure, like ArrayQueue or ArrayDeque? LinkedList is not recommanded for most cases...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#issuecomment-952594672
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 3m 54s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 17s | master passed |
| +1 :green_heart: | compile | 3m 9s | master passed |
| +1 :green_heart: | checkstyle | 1m 6s | master passed |
| +1 :green_heart: | spotbugs | 2m 8s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 48s | the patch passed |
| +1 :green_heart: | compile | 3m 14s | the patch passed |
| +1 :green_heart: | javac | 3m 14s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 4s | hbase-server: The patch generated 0 new + 34 unchanged - 1 fixed = 34 total (was 35) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 19m 42s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 2m 21s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. |
| | | 53m 15s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/5/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3777 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 7d6f3546d3ad 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 9b7ba7f300 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Max. process+thread count | 95 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/5/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#issuecomment-949369262
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 25s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 12s | master passed |
| +1 :green_heart: | compile | 3m 10s | master passed |
| +1 :green_heart: | checkstyle | 1m 5s | master passed |
| +1 :green_heart: | spotbugs | 2m 5s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 59s | the patch passed |
| +1 :green_heart: | compile | 3m 15s | the patch passed |
| +1 :green_heart: | javac | 3m 15s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 3s | hbase-server: The patch generated 0 new + 34 unchanged - 1 fixed = 34 total (was 35) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 19m 36s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 2m 14s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. |
| | | 49m 36s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/3/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3777 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux dab2da062720 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / e0813e5402 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Max. process+thread count | 96 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/3/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#issuecomment-950108980
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 14s | Docker mode activated. |
| -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 35s | master passed |
| +1 :green_heart: | compile | 1m 28s | master passed |
| +1 :green_heart: | shadedjars | 9m 36s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 47s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 25s | the patch passed |
| +1 :green_heart: | compile | 1m 30s | the patch passed |
| +1 :green_heart: | javac | 1m 30s | the patch passed |
| +1 :green_heart: | shadedjars | 9m 37s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 48s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 226m 27s | hbase-server in the patch passed. |
| | | 264m 23s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3777 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux ce779347972c 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 9e47358ecb |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/4/testReport/ |
| Max. process+thread count | 3433 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/4/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] comnetwork commented on a change in pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Posted by GitBox <gi...@apache.org>.
comnetwork commented on a change in pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#discussion_r740113588
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHStore.java
##########
@@ -1921,6 +1922,225 @@ private void doWriteTestLargeCellAndSmallCellConcurrently(
}
}
+ /**
+ * <pre>
+ * This test is for HBASE-26384,
+ * test {@link CompactingMemStore#flattenOneSegment} and {@link CompactingMemStore#snapshot()}
+ * execute concurrently.
+ * The threads sequence before HBASE-26384 is(The bug only exists for branch-2,and I add UTs
+ * for both branch-2 and master):
+ * 1. The {@link CompactingMemStore} size exceeds
+ * {@link CompactingMemStore#getInmemoryFlushSize()},the write thread adds a new
+ * {@link ImmutableSegment} to the head of {@link CompactingMemStore#pipeline},and start a
+ * in memory compact thread to execute {@link CompactingMemStore#inMemoryCompaction}.
+ * 2. The in memory compact thread starts and then stopping before
+ * {@link CompactingMemStore#flattenOneSegment}.
+ * 3. The snapshot thread starts {@link CompactingMemStore#snapshot} concurrently,after the
+ * snapshot thread executing {@link CompactingMemStore#getImmutableSegments},the in memory
+ * compact thread continues.
+ * Assuming {@link VersionedSegmentsList#version} returned from
+ * {@link CompactingMemStore#getImmutableSegments} is v.
+ * 4. The snapshot thread stopping before {@link CompactingMemStore#swapPipelineWithNull}.
+ * 5. The in memory compact thread completes {@link CompactingMemStore#flattenOneSegment},
+ * {@link CompactionPipeline#version} is still v.
+ * 6. The snapshot thread continues {@link CompactingMemStore#swapPipelineWithNull}, and because
+ * {@link CompactionPipeline#version} is v, {@link CompactingMemStore#swapPipelineWithNull}
+ * thinks it is successful and continue flushing,but the {@link ImmutableSegment} in
+ * {@link CompactionPipeline} has changed because
+ * {@link CompactingMemStore#flattenOneSegment},so the {@link ImmutableSegment} is not
+ * removed in fact and still remaining in {@link CompactionPipeline}.
+ *
+ * After HBASE-26384, the 5-6 step is changed to following, which is expected behavior:
+ * 5. The in memory compact thread completes {@link CompactingMemStore#flattenOneSegment},
+ * {@link CompactingMemStore#flattenOneSegment} change {@link CompactionPipeline#version} to
+ * v+1.
Review comment:
Yes ，I explain this at the end of the JIRA description， because this `version `bug for `CompactingMemStore#flattenOneSegment` is only exists for branch-2， for master，`version` is correctly increased for `CompactingMemStore#flattenOneSegment` ，so I made separate PRs for master(#3777) and branch-2(#3779) , so the code for master and brach-2 could keep consistent and master could also has UTs for this problem.
and I included folowing comments at the head of this test method comments:
```
/*....
* The threads sequence before HBASE-26384 is(The bug only exists for branch-2,and I add UTs
* for both branch-2 and master):
* .../
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org