[jira] [Commented] (HBASE-17338) Treat Cell data size under global memstore heap size only when that Cell can not be copied to MSLAB

["Anoop Sam John (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-17338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891676#comment-15891676 ] 

Anoop Sam John commented on HBASE-17338:
----------------------------------------

bq.Dumb question. dataSize is KV infrastructure + key content + value + trailing tags and sequenceid if any? i.e. the whole KV? And CellSize is infrastructure only or rather key+infrastructure?
Not sure whether used dataSize/cellSize here and there. Its data size only which is the way as u said above. There is no cellSize as such. There is only cell heap size which is the heap size (total) occupied by the cell impl object
bq.There is one global threshold whether data is onheap or offheap (I probably got this wrong?)
Ya there is one global threshold in both cases. Moreover in case of off heap, there is a heap global threshold also. ie. By def 40% of xmx above which we will force flush. In case of offheap, we have to check this extra thing also or else there is possibility of global memstore size getting oversized and GC impacts/OOME.
bq.We could probably but the direct memory would remain allocated until we restart.
Sorry for not clear here. I dont mean ON/OFF MSLAB over the RS run time. Now we have a way to turn it ON at cluster level means at RS level. All the regions in this RS will use this MSLAB then. (Ya we know some cells wont get copied to MSLAB - increment/append) My Q was whether there is a way for turning off MSLAB usage for a specific table. Say in RS MSLAB is ON.  But the regions from this particular table wont use MSLAB at all.

> Treat Cell data size under global memstore heap size only when that Cell can not be copied to MSLAB
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-17338
>                 URL: https://issues.apache.org/jira/browse/HBASE-17338
>             Project: HBase
>          Issue Type: Sub-task
>          Components: regionserver
>    Affects Versions: 2.0.0
>            Reporter: Anoop Sam John
>            Assignee: Anoop Sam John
>             Fix For: 2.0.0
>
>         Attachments: HBASE-17338.patch, HBASE-17338_V2.patch, HBASE-17338_V2.patch, HBASE-17338_V4.patch, HBASE-17338_V5.patch
>
>
> We have only data size and heap overhead being tracked globally.  Off heap memstore works with off heap backed MSLAB pool.  But a cell, when added to memstore, not always getting copied to MSLAB.  Append/Increment ops doing an upsert, dont use MSLAB.  Also based on the Cell size, we sometimes avoid MSLAB copy.  But now we track these cell data size also under the global memstore data size which indicated off heap size in case of off heap memstore.  For global checks for flushes (against lower/upper watermark levels), we check this size against max off heap memstore size.  We do check heap overhead against global heap memstore size (Defaults to 40% of xmx)  But for such cells the data size also should be accounted under the heap overhead.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)