You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Vladimir Rodionov (JIRA)" <ji...@apache.org> on 2014/12/08 23:08:12 UTC
[jira] [Commented] (HBASE-12657) The Region is not being split and
far exceeds the desired maximum size.
[ https://issues.apache.org/jira/browse/HBASE-12657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14238546#comment-14238546 ]
Vladimir Rodionov commented on HBASE-12657:
-------------------------------------------
Region can't be split because, despite of its size, it still has reference files and this prevents region from splitting. The question : what does prevent reference files from being compacted?
Lowest sequence ID among all store files in a region is the reason that reference files are constantly getting removed from compaction selections if there are newer files in a compaction queue. This is what is happening under high load when there are too many minor compaction requests in a queue, reference files do not have a chance to be compacted. Interestingly, that current 0.94 and 0.98 code have different issues here and require different patches.
0.94 does not treat compaction request with reference files as major one, but ignores *hbase.hstore.compaction.max* for major compactions, 0.98 consider compaction of reference files as a major one, but consults *hbase.hstore.compaction.max* and downgrades request when # of files exceeds this limit.
Low (default) value for *hbase.hstore.compaction.max* and prolonged write activity w/o throttling will get you into this region is out of bounds situation. Because, when compaction can't keep up with writes, number of store files eventually exceeds *hbase.hstore.compaction.max*.
Here what is happening in 0.94:
*0.94 - Does not enforce major compaction flag on selections with reference files, applies selection algorithm and applies the limit by removing first K = N - max files from candidate list. N - # of files in selection list, max - is *hbase.hstore.compaction.max* value.
*0.98 - It marks selection as major but after that applies limit and removes first K = N - max files from a candidate list.
In both cases, if # of references files > max, some reference files will be excluded.
What happens later, depends on what we have in a compaction file list for this Store (HStore). If pending files list has at least one non-reference file, all reference files will be excluded from above selection. What I have to say here, it seems that the only way to compact all reference files is to enforce major compaction immediately after region split. If we fail to do this, with a very high probability, reference files will be pushing out of compaction until the write load decrease substantially and # of store files becomes less than *hbase.hstore.compaction.max*.
> The Region is not being split and far exceeds the desired maximum size.
> -----------------------------------------------------------------------
>
> Key: HBASE-12657
> URL: https://issues.apache.org/jira/browse/HBASE-12657
> Project: HBase
> Issue Type: Bug
> Components: Compaction
> Affects Versions: 0.98.8, 0.94.25, 0.99.2
> Reporter: Vladimir Rodionov
> Assignee: Vladimir Rodionov
> Fix For: 1.0.0, 2.0.0, 0.94.26, 0.98.9
>
>
> We are seeing this behavior when creating indexes in one of our environment.
> When an index is being created, most of the "requests" go into a single region. The amount of time to create an index seems to take longer than usual and it can take days for the regions to compact and split after the index is created.
> Here is a du of the HBase index table:
> {code}
> -bash-4.1$ sudo -su hdfs hadoop fs -du /hbase/43681
> 705 /hbase/43681/.tableinfo.0000000001
> 0 /hbase/43681/.tmp
> 27981697293 /hbase/43681/0492e22092e21d35fca8e779b21ec797
> 539687093 /hbase/43681/832298c4e975fc47210feb6bac3d2f71
> 560660531 /hbase/43681/be9bdb3bdf9365afe5fe90db4247d82c
> 7081938297 /hbase/43681/cd440e524f96fbe0719b2fe969848560
> 6297860287 /hbase/43681/dc893a2d8daa08c689dc69e6bb2c5b50
> 7189607722 /hbase/43681/ffbceaea5e2f142dbe6cd4cbeacc00e8
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)