[jira] [Resolved] (HBASE-5476) Merge flush/merging compaction; save on i/o by merging an existing file with content memstore

["Andrew Purtell (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-5476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Purtell resolved HBASE-5476.
-----------------------------------
    Resolution: Incomplete

> Merge flush/merging compaction; save on i/o by merging an existing file with content memstore
> ---------------------------------------------------------------------------------------------
>
>                 Key: HBASE-5476
>                 URL: https://issues.apache.org/jira/browse/HBASE-5476
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: stack
>
> Compactions are slow.  To improve, we can work on  making compactions faster (TODO) or make it so compactions have less work to do.  This issue is about the later; doing something the bigtable paper talks off where the content of memory is merged with existing content in the filesystem (I'm sure we've discussed this a bunch in the past but can't find an explicit issue for it).
> We save on the reading of the size of a memstore from the filesystem if we merge flush/merge compact.
> BT seems to include memstore content in minor compactions.  We should look at doing that (would snapshot at start of compaction and then would integrate the snapshot into resultant compacted file).  Alternatively we could flush into a small file that is already in the filesystem (this could be tougher given that we have this flush-compaction separation at the moment... how would we make it so the file we're merging into is not picked up for a normal compaction).
> Doing the merge compaction/flush merge would slow down flushes.  It could back up memory into the global barrier such that we stop taking on writes altogether.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)