[jira] Commented: (HBASE-2500) Block indexes and meta blocks should be put into the LRU

["Jonathan Gray (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861948#action_12861948 ] 

Jonathan Gray commented on HBASE-2500:
--------------------------------------

My preference would be to add all of these to the existing block cache LRU.  The open question then would be whether they are treated like other normal data blocks, or whether we add additional block types.  The design of the LRU actually makes it quite easy to add additional types (currently there is single access block, multiple access block, and in-memory block), and there is not much overhead involved.  The additional overhead is done in the background eviction thread so doesn't add anything to the put/get methods.

If there was a new block type or two, like "index" and/or "meta", then these would be able to have their own slice of the total block cache pie.  The LRU gives each of the different block types their own percentage of the total LRU size.  Each is able to grow above it's slice if others are under-utilized, the LRU just guarantees that during an eviction that block type will be given at least the specified slice size.

I think we'll also need to add code to re-fetch block indexes / meta blocks if they are not in the block cache.

> Block indexes and meta blocks should be put into the LRU
> --------------------------------------------------------
>
>                 Key: HBASE-2500
>                 URL: https://issues.apache.org/jira/browse/HBASE-2500
>             Project: Hadoop HBase
>          Issue Type: Improvement
>          Components: regionserver
>            Reporter: Jonathan Gray
>             Fix For: 0.20.5, 0.21.0
>
>
> Currently we are not tracking the heap usage of block indexes.  We're just relying on the fact that we leave a large percentage of total heap as part of "everything else" not in memstores or the block cache.  The "everything else" portion should be left for ephemeral/transient memory usage, not long-lived objects like block indexes.  Introduction of blooms will extend this issue to meta blocks in general.
> All of this should be moved into an LRU.  Either the existing block cache LRU or a new LRU.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.