[jira] [Updated] (HBASE-25302) [Umbrella]Fast split regions and lightweight compaction with stripe store engine

["Xiaolin Ha (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-25302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xiaolin Ha updated HBASE-25302:
-------------------------------
    Affects Version/s: 3.0.0-alpha-2

> [Umbrella]Fast split regions and lightweight compaction with stripe store engine
> --------------------------------------------------------------------------------
>
>                 Key: HBASE-25302
>                 URL: https://issues.apache.org/jira/browse/HBASE-25302
>             Project: HBase
>          Issue Type: Umbrella
>    Affects Versions: 3.0.0-alpha-2
>            Reporter: Xiaolin Ha
>            Assignee: Xiaolin Ha
>            Priority: Major
>         Attachments: after-split.png, before-split.png
>
>
> In our company MeiTuan, we have encountered a problem, that is a production cluster with about more than 10000 regions set off the split switch by some mistakes. And more than 10000 regions are larger than 600GB before we can do something to split them. Because online real time R+W QPS of each RS is higher than 80000 (8000+ per region), if we just switch on split, the IO pressure brought by compactions will influence the availabilities, and all large regions need repeatedly compactions until the size of child regions are normal.
> Different with the normal circumstances, we used stripe store engine in this cluster, after looking through docs of https://issues.apache.org/jira/browse/HBASE-7667 ,  we have implemented a fast split region method using the similar ideas of HFileLink and move store files between regions when compacting. Actually, this idea has already been mentioned in HBASE-7667 , it said that `region splits become marvelously simple (if we could move files between regions, no references would be needed)`.
> We have implemented the ideas and practiced them on that cluster. The fact results show that, a region of about 600G can  be splitted to  75G*8 regions in about five minutes, with less than 5G total rewrite size(all are L0) in the whole process, while normal continuous split needs 600G*3=1800G. By using movement of store files, the rewritten size is less than 50G(two stripe size, because the rebuild of HFileLinks to stripes may insert some files to L0.).I have attached two pictures about a RS before and after splitting all the regions using this method.
> This idea can not only be used in stripe store engine, but also default store engine, and the store movement methods are very benefit to region MERGE.  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)