You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@hbase.apache.org by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org> on 2012/02/01 05:58:58 UTC

[jira] [Commented] (HBASE-5311) Allow inmemory Memstore compactions

    [ https://issues.apache.org/jira/browse/HBASE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197581#comment-13197581 ] 

Todd Lipcon commented on HBASE-5311:
------------------------------------

Another thing I've considered and mentioned to people, but don't think is tracked on JIRA, is to do "in-memory flushes". Our memstores are ConcurrentSkipListMaps, which have the nice property of being lock-free for contending updates, but the poor property of having bad cache locality and being less efficient on the read side (beware: unsubstantiated claim). 

If we periodically flushed to in-memory "dense" storage, we'd probably get a bunch of benefits:
- scans should be faster since we could order the KVs in memory according to their sort order, in a dense array (taking advantage of sequential memory access being much faster than random)
- reads in general should be faster since there would be less contention on the datastructure
- we could use less memory by using a more tightly packed data structure for the immutable data, or even using HFile compression methods there
                
> Allow inmemory Memstore compactions
> -----------------------------------
>
>                 Key: HBASE-5311
>                 URL: https://issues.apache.org/jira/browse/HBASE-5311
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Lars Hofhansl
>
> Just like we periodically compact the StoreFiles we should also periodically compact the MemStore.
> During these compactions we eliminate deleted cells, expired cells, cells to removed because of version count, etc, before we even do a memstore flush.
> Besides the optimization that we could get from this, it should also allow us to remove the special handling of ICV, Increment, and Append (all of which use upsert logic to avoid accumulating excessive cells in the Memstore).
> Not targeting this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira