You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2011/03/16 18:03:29 UTC
[jira] Created: (HBASE-3656) Merging flush; merge a flush with one
of the existing store files (the smallest?) so we skip creating a new store
file on each flush
Merging flush; merge a flush with one of the existing store files (the smallest?) so we skip creating a new store file on each flush
------------------------------------------------------------------------------------------------------------------------------------
Key: HBASE-3656
URL: https://issues.apache.org/jira/browse/HBASE-3656
Project: HBase
Issue Type: Task
Reporter: stack
This behavior is described in the BT paper. Years ago I had a go at it but at the time it slowed flushing significantly -- and IIRC we had no barriers on writes when the memory pressue was high -- so it brought on OOMEs... so punted on it. Its time to consider this feature again.
Would we always do it? Maybe not if its a close? If a close we want stuff to run quickly so we should skip the merge. But any other time, we should do it?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HBASE-3656) Merging flush; merge a flush with
one of the existing store files (the smallest?) so we skip creating a new
store file on each flush
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007686#comment-13007686 ]
Jonathan Gray commented on HBASE-3656:
--------------------------------------
I think we'd need some pretty well-thought-out heuristics for when to do it and when not to do it.
I also had a stab at this when I worked on HBASE-2375 way back when (yeah, need to get back on that one soon).
When I looked at this back then, a much bigger win was to upgrade a "flush then compaction" to a "merge flush with files that would have been selected for compaction". (we only compact after a flush today, but before we do the flush, we can actually know whether a compaction will be requested afterwards). In this case, I think it would be a clear win in this case.
Otherwise, I think it would mostly make sense if the file it is merging with was relatively small and the amount being flushed is relatively small (the kind of stuff that happens when you have too many regions on each RS or not enough total HLog capacity --> small flushes).
> Merging flush; merge a flush with one of the existing store files (the smallest?) so we skip creating a new store file on each flush
> ------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3656
> URL: https://issues.apache.org/jira/browse/HBASE-3656
> Project: HBase
> Issue Type: Task
> Reporter: stack
>
> This behavior is described in the BT paper. Years ago I had a go at it but at the time it slowed flushing significantly -- and IIRC we had no barriers on writes when the memory pressue was high -- so it brought on OOMEs... so punted on it. Its time to consider this feature again.
> Would we always do it? Maybe not if its a close? If a close we want stuff to run quickly so we should skip the merge. But any other time, we should do it?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3656) Merging flush; merge a flush with
one of the existing store files (the smallest?) so we skip creating a new
store file on each flush
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469697#comment-13469697 ]
Lars Hofhansl commented on HBASE-3656:
--------------------------------------
This would (potentially) make certain backup solution more difficult.
Imagine a backup scheme that backs up each HFile resulting from a major compaction and each HFile resulting from a flush (HFiles from minor compactions are not needed and just complicate things).
This would no longer be possible since the flushed KVs would then be intermingled with other KVs, which may or may not be in an HFile that was copied already.
Not saying this is a reason for not doing this.
> Merging flush; merge a flush with one of the existing store files (the smallest?) so we skip creating a new store file on each flush
> ------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3656
> URL: https://issues.apache.org/jira/browse/HBASE-3656
> Project: HBase
> Issue Type: Task
> Reporter: stack
>
> This behavior is described in the BT paper. Years ago I had a go at it but at the time it slowed flushing significantly -- and IIRC we had no barriers on writes when the memory pressue was high -- so it brought on OOMEs... so punted on it. Its time to consider this feature again.
> Would we always do it? Maybe not if its a close? If a close we want stuff to run quickly so we should skip the merge. But any other time, we should do it?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3656) Merging flush; merge a flush with
one of the existing store files (the smallest?) so we skip creating a new
store file on each flush
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018439#comment-13018439 ]
Jean-Daniel Cryans commented on HBASE-3656:
-------------------------------------------
Talking on IRC with aph__ (yeah that's his handle), he was running imports on a small 3 nodes cluster with heap set at 1.5GB and saw abysmal performance. While we told him to get some real hardware because tuning his current setup would be a massive waste of time, I was thinking that HBase could still do a much better job out of the box. One thing that would make it better is somehow automatically controlling regions size/number to not run into the situation where you basically are filling up 1000 regions at the same rate and flush tiny files (which then triggers compactions _ad vitam aeternam_).
The other thing that would help is this jira since you wouldn't be compacting so much in the first place.
> Merging flush; merge a flush with one of the existing store files (the smallest?) so we skip creating a new store file on each flush
> ------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3656
> URL: https://issues.apache.org/jira/browse/HBASE-3656
> Project: HBase
> Issue Type: Task
> Reporter: stack
>
> This behavior is described in the BT paper. Years ago I had a go at it but at the time it slowed flushing significantly -- and IIRC we had no barriers on writes when the memory pressue was high -- so it brought on OOMEs... so punted on it. Its time to consider this feature again.
> Would we always do it? Maybe not if its a close? If a close we want stuff to run quickly so we should skip the merge. But any other time, we should do it?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira