You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "chenxu (JIRA)" <ji...@apache.org> on 2018/11/12 08:42:00 UTC
[jira] [Commented] (HBASE-20542) Better heap utilization for IMC
with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683410#comment-16683410 ]
chenxu commented on HBASE-20542:
--------------------------------
{code:java}
protected boolean shouldFlushInMemory(MutableSegment currActive, Cell cellToAdd,
MemStoreSizing memstoreSizing) {
long cellSize = currActive.getCellLength(cellToAdd);
long segmentDataSize = currActive.getDataSize();
while (segmentDataSize + cellSize < inmemoryFlushSize || inWalReplay) {
// when replaying edits from WAL there is no need in in-memory flush regardless the size
{code}
if _cellSize_ larger than _inmemoryFlushSize_, AbstractMemStore#doAddOrUpsert will never break the while loop?
> Better heap utilization for IMC with MSLABs
> -------------------------------------------
>
> Key: HBASE-20542
> URL: https://issues.apache.org/jira/browse/HBASE-20542
> Project: HBase
> Issue Type: Sub-task
> Components: in-memory-compaction
> Reporter: Eshcar Hillel
> Assignee: Eshcar Hillel
> Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-20542-addendum.master.005.patch, HBASE-20542.branch-2.001.patch, HBASE-20542.branch-2.003.patch, HBASE-20542.branch-2.004.patch, HBASE-20542.branch-2.005.patch, HBASE-20542.master.003.patch, HBASE-20542.master.005-addendum.patch, run.sh, workloada, workloadc, workloadx, workloady
>
>
> Following HBASE-20188 we realized in-memory compaction combined with MSLABs may suffer from heap under-utilization due to internal fragmentation. This jira presents a solution to circumvent this problem. The main idea is to have each update operation check if it will cause overflow in the active segment *before* it is writing the new value (instead of checking the size after the write is completed), and if it is then the active segment is atomically swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to the compaction pipeline. Later on the IMC deamon will run its compaction operation (flatten index/merge indices/data compaction) in the background. Some subtle concurrency issues should be handled with care. We next elaborate on them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)