[jira] [Reopened] (HBASE-20542) Better heap utilization for IMC with MSLABs

["Duo Zhang (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Duo Zhang reopened HBASE-20542:
-------------------------------

This breaks TestWriteHeavyIncrementObserverWithMemStoreCompaction.

The UT assumes that with in-memory compaction, we can do all the aggregation in memory so there will be no store file generated, but with the patch here we do have some store files generated.

So what's the problem? With this patch we slow down the in memory compaction?

> Better heap utilization for IMC with MSLABs
> -------------------------------------------
>
>                 Key: HBASE-20542
>                 URL: https://issues.apache.org/jira/browse/HBASE-20542
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: Eshcar Hillel
>            Assignee: Eshcar Hillel
>            Priority: Major
>             Fix For: 3.0.0, 2.2.0
>
>         Attachments: HBASE-20542.branch-2.001.patch, HBASE-20542.branch-2.003.patch, HBASE-20542.branch-2.004.patch, HBASE-20542.branch-2.005.patch, HBASE-20542.master.003.patch, run.sh, workloada, workloadc, workloadx, workloady
>
>
> Following HBASE-20188 we realized in-memory compaction combined with MSLABs may suffer from heap under-utilization due to internal fragmentation. This jira presents a solution to circumvent this problem. The main idea is to have each update operation check if it will cause overflow in the active segment *before* it is writing the new value (instead of checking the size after the write is completed), and if it is then the active segment is atomically swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to the compaction pipeline. Later on the IMC deamon will run its compaction operation (flatten index/merge indices/data compaction) in the background. Some subtle concurrency issues should be handled with care. We next elaborate on them.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)