[jira] [Commented] (HBASE-20542) Better heap utilization for IMC with MSLABs

["Eshcar Hillel (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516156#comment-16516156 ] 

Eshcar Hillel commented on HBASE-20542:
---------------------------------------

Patch is attached.
To reduce internal fragmentation the size of the active segment is set to be the size of one MSLAB chunk (by default 2MB).
An add operation is supplemented with pre-update and post update procedures.
The pre-update procedure atomically increases the size of the segment if this increment does not exceed the segment size threshold, and then continues with the normal path of updating the memstore.
If the increment will exceed the segment size threshold then the size is not increased and instead 
(1) the segment is flushed into the compaction pipeline,
(2) a new active segment is created, 
(3) an IMC task is scheduled in the background,
(4) the operation re-runs the pre-update procedure, this time with the new active segment.

This changes calls for an additional optimization.
The IMC no longer needs to acquire the region level updates lock. Instead we use segment level read-write lock to synchronize IMC with concurrent update operations. This is better since with the new solution IMC only needs to wait only for those few operations that already updated the size of the segment in the pre-update procedure but are still updating the segment skip list, and does not need to wait for operations of other stores. Moreover, update operation do not wait for in-memory flush to complete as before.
To synchronize, update operation take the read lock of the segment they are updating in the pre-update procedure, and release it in the post-update procedure. IMC thread take the write lock of each segment it is compacting. This ensures all updates that started before the in-memory flush have completed.

I will upload the patch also in RB.
Feel free to ask questions and comment.


> Better heap utilization for IMC with MSLABs
> -------------------------------------------
>
>                 Key: HBASE-20542
>                 URL: https://issues.apache.org/jira/browse/HBASE-20542
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: Eshcar Hillel
>            Assignee: Eshcar Hillel
>            Priority: Major
>         Attachments: HBASE-20542.branch-2.001.patch
>
>
> Following HBASE-20188 we realized in-memory compaction combined with MSLABs may suffer from heap under-utilization due to internal fragmentation. This jira presents a solution to circumvent this problem. The main idea is to have each update operation check if it will cause overflow in the active segment *before* it is writing the new value (instead of checking the size after the write is completed), and if it is then the active segment is atomically swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to the compaction pipeline. Later on the IMC deamon will run its compaction operation (flatten index/merge indices/data compaction) in the background. Some subtle concurrency issues should be handled with care. We next elaborate on them.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)