You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Andrew Kyle Purtell (Jira)" <ji...@apache.org> on 2021/11/09 05:36:00 UTC
[jira] [Comment Edited] (HBASE-26384) Segment already flushed to
hfile may still be remained in CompactingMemStore
[ https://issues.apache.org/jira/browse/HBASE-26384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17440907#comment-17440907 ]
Andrew Kyle Purtell edited comment on HBASE-26384 at 11/9/21, 5:35 AM:
-----------------------------------------------------------------------
Recent branch-2.4 is showing dataloss in an integration test scenario with ADAPTIVE compacting memstore active.
{noformat}
2021-11-09 03:52:28,730 INFO [main] mapreduce.Job: tasks=22970712064
Total megabyte-milliseconds taken by all reduce tasks=282829824
Map-Reduce Framework
Map input records=27572937
Map output records=0
Map output bytes=0
Map output materialized bytes=600
Input split bytes=15500
Combine input records=0
Combine output records=0
Reduce input groups=0
Reduce shuffle bytes=600
Reduce input records=0
Reduce output records=0
Spilled Records=0
Shuffled Maps =100
Failed Shuffles=0
Merged Map outputs=100
GC time elapsed (ms)=13011
CPU time spent (ms)=5495120
Physical memory (bytes) snapshot=148964569088
Virtual memory (bytes) snapshot=355227025408
Total committed heap usage (bytes)=107583897600
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
org.apache.hadoop.hbase.test.IntegrationTestLoadCommonCrawl$Counts
REFERENCED=27475828
UNREFERENCED=97109
File Input Format Counters
Bytes Read=659739581
File Output Format Counters
Bytes Written=0
2021-11-09 03:52:28,732 INFO [main] test.IntegrationTestLoadCommonCrawl$Verify: REFERENCED: 27475828
2021-11-09 03:52:28,732 INFO [main] test.IntegrationTestLoadCommonCrawl$Verify: UNREFERENCED: 97109
2021-11-09 03:52:28,732 INFO [main] test.IntegrationTestLoadCommonCrawl$Verify: CORRUPT: 0
2021-11-09 03:52:28,733 ERROR [main] test.IntegrationTestLoadCommonCrawl$Verify: Nonzero UNREFERENCED count from job job_1636426618387_0004
{noformat}
was (Author: apurtell):
Recent branch-2.4 is showing dataloss in an integration test scenario with ADAPTIVE compacting memstore active.
> Segment already flushed to hfile may still be remained in CompactingMemStore
> -----------------------------------------------------------------------------
>
> Key: HBASE-26384
> URL: https://issues.apache.org/jira/browse/HBASE-26384
> Project: HBase
> Issue Type: Bug
> Components: in-memory-compaction
> Affects Versions: 3.0.0-alpha-1, 2.4.8
> Reporter: chenglei
> Assignee: chenglei
> Priority: Major
> Labels: branch-2
> Fix For: 2.5.0, 3.0.0-alpha-2
>
>
> When {{CompactingMemStore}} prepares flushing, {{CompactingMemStore.snapshot}} invokes following {{CompactingMemStore.pushPipelineToSnapshot}} method to get {{Snapshot}}, following line 570 and line 575 uses {{CompactionPipeline#version}} to track whether the Segments in {{CompactionPipeline#pipeline}} has changed since it gets {{VersionedSegmentsList}} in line 570 before emptying {{CompactionPipeline#pipeline}} in line 575.
> {code:java}
> 565 private void pushPipelineToSnapshot() {
> 566 int iterationsCnt = 0;
> 567 boolean done = false;
> 568 while (!done) {
> 569 iterationsCnt++;
> 570 VersionedSegmentsList segments = pipeline.getVersionedList();
> 571 pushToSnapshot(segments.getStoreSegments());
> 572 // swap can return false in case the pipeline was updated by ongoing compaction
> 573 // and the version increase, the chance of it happenning is very low
> 574 // In Swap: don't close segments (they are in snapshot now) and don't update the region size
> 575 done = pipeline.swap(segments, null, false, false);
> .......
> }
> {code}
> However, when {{CompactingMemStore#inMemoryCompaction}} executes {{CompactionPipeline#flattenOneSegment}}, it does not change {{CompactionPipeline#version}} , if there is an {{in memeory compaction}} which executes {{CompactingMemStore#flattenOneSegment}} between above line 570 and line 575, the {{CompactionPipeline#version}} not change, but the {{Segment}} in {{CompactionPipeline}} has changed. Because {{CompactionPipeline#version}} not change, {{pipeline.swap}} in above line 575 could think it is safe to invoke following {{CompactionPipeline#swapSuffix}} method to remove {{Segment}} in {{CompactionPipeline}} , but the {{Segment}} in {{CompactionPipeline}} has changed because of {{CompactingMemStore#flattenOneSegment}} , so the {{Segment}} not removed in following line 295 and still remaining in {{CompactionPipeline}}.
> {code:java}
> 293 private void swapSuffix(List<? extends Segment> suffix, ImmutableSegment segment,
> 294 boolean closeSegmentsInSuffix) {
> 295 pipeline.removeAll(suffix);
> 296 if(segment != null) pipeline.addLast(segment);
> ....
> {code}
> However {{CompactingMemStore.snapshot}} think it is successful and continues to flush the {{Segment}} got by {{CompactingMemStore.snapshot}} as normal, but the {{Segment}} with the same cells still be left in {{CompactingMemStore}}. Leaving {{Segment}} which already flushed in {{MemStore}} is dangerous: if a Major Compaction before the left {{Segment}} flushing, there may be data erroneous.
> My Fix in the PR is as following:
> # Increasing the {{CompactionPipeline#version}} in {{CompactingMemStore#flattenOneSegment}} .
> Branch-2 has this problem but master not， because the branch-2 patch for HBASE-18375 omitting this.
> # For {{CompactionPipeline#swapSuffix}} , explicitly checking that the {{Segment}} in {{suffix}} input parameter is same as the {{Segment}} in {{pipeline}} one by one from
> the last element to the first element of {{suffix}} ， I think explicitly throwing Exception is better than hiding error and causing subtle problem.
> I made separate PRs for master and branch-2 so the code for master and brach-2 could consistent and master could also has UTs for this problem.
> [PR#3777|https://github.com/apache/hbase/pull/3777] is for master and [PR#3779|https://github.com/apache/hbase/pull/3779] is for branch-2.The difference between them is patch for brach-2 including following code in {{CompactionPipeline.replaceAtIndex}} which not included in branch-2 patch for HBASE-18375:
> {code:java}
> // the version increment is indeed needed, because the swap uses removeAll() method of the
> // linked-list that compares the objects to find what to remove.
> // The flattening changes the segment object completely (creation pattern) and so
> // swap will not proceed correctly after concurrent flattening.
> version++;
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)