You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Billy Pearson (JIRA)" <ji...@apache.org> on 2008/07/25 22:11:31 UTC
[jira] Created: (HBASE-775) max mapfiles to compact at one time
max mapfiles to compact at one time
-----------------------------------
Key: HBASE-775
URL: https://issues.apache.org/jira/browse/HBASE-775
Project: Hadoop HBase
Issue Type: Improvement
Components: regionserver
Reporter: Billy Pearson
Priority: Minor
Fix For: 0.3.0
Currently we compact all map files with no upper limit this could cause a regionserver to OOME if the compaction get behind and the number of mapfiles build up.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-775) max mapfiles to compact at one time
Posted by "Billy Pearson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12623046#action_12623046 ]
Billy Pearson commented on HBASE-775:
-------------------------------------
If there is some other reason any one would see us needing it let me know and I will make a patch based on the count of file compacted at one time
> max mapfiles to compact at one time
> -----------------------------------
>
> Key: HBASE-775
> URL: https://issues.apache.org/jira/browse/HBASE-775
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: Billy Pearson
> Assignee: Billy Pearson
> Priority: Blocker
> Fix For: 0.2.1, 0.3.0
>
> Attachments: 775-patch.txt
>
>
> Currently we compact all map files with no upper limit this could cause a regionserver to OOME if the compaction get behind and the number of mapfiles build up.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-775) max mapfiles to compact at one time
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack reassigned HBASE-775:
---------------------------
Assignee: Billy Pearson
Billy, you think that you could have a go at this one? Doesn't look too hard. Add a new configuration into hbase-default.xml named something like hbase.hstore.compaction.max In the HStore constructor, you would read its setting and save it off to a data member. See 'hbase.hstore.compactionThreshold' for an example. Then you'd add something around line #785 in HStore where you'd only compact the hbase.hstore.compaction.max amount.
Also, checkout HBASE-64. So much has changed since that issue was originally filed but there are a load of your ideas in there about how we should do compactions. Perhaps distill out the good stuff into new, more pointed issues?
> max mapfiles to compact at one time
> -----------------------------------
>
> Key: HBASE-775
> URL: https://issues.apache.org/jira/browse/HBASE-775
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: Billy Pearson
> Assignee: Billy Pearson
> Priority: Blocker
> Fix For: 0.3.0
>
>
> Currently we compact all map files with no upper limit this could cause a regionserver to OOME if the compaction get behind and the number of mapfiles build up.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-775) max mapfiles to compact at one time
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-775:
------------------------
Resolution: Won't Fix
Status: Resolved (was: Patch Available)
I'm tempted to leave hbase-64 open for now since the original justification in hbase-64 on why there should be a maximum number of files to compact at any one time is convincing, to me at least.
But HBASE-64 became a mess pulled in all directions. Because of this, I closed hbase-64 and opened a new issue on a maximum number of files per compaction -- perferably the smaller ones first (HBASE-834).
I'll close this one since it was based on a misunderstanding.
Thanks Billy.
> max mapfiles to compact at one time
> -----------------------------------
>
> Key: HBASE-775
> URL: https://issues.apache.org/jira/browse/HBASE-775
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: Billy Pearson
> Assignee: Billy Pearson
> Priority: Blocker
> Fix For: 0.2.1, 0.3.0
>
> Attachments: 775-patch.txt
>
>
> Currently we compact all map files with no upper limit this could cause a regionserver to OOME if the compaction get behind and the number of mapfiles build up.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-775) max mapfiles to compact at one time
Posted by "Billy Pearson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Billy Pearson updated HBASE-775:
--------------------------------
Fix Version/s: 0.2.1
Status: Patch Available (was: Open)
> max mapfiles to compact at one time
> -----------------------------------
>
> Key: HBASE-775
> URL: https://issues.apache.org/jira/browse/HBASE-775
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: Billy Pearson
> Assignee: Billy Pearson
> Priority: Blocker
> Fix For: 0.2.1, 0.3.0
>
> Attachments: 775-patch.txt
>
>
> Currently we compact all map files with no upper limit this could cause a regionserver to OOME if the compaction get behind and the number of mapfiles build up.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-775) max mapfiles to compact at one time
Posted by "Billy Pearson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12623043#action_12623043 ]
Billy Pearson commented on HBASE-775:
-------------------------------------
I thought we did it in Ram if its just streaming then there should not be a max. we should do all we can on each compaction that would be less work on the next compaction. I assume there some kind of index that is in memory so we keep track of max versions etc so that would be the only factor to limit the count or size of the compaction.
Sense we do not compact in memory I do not see any reason for this issue then so it would be resolved as won't fix
HBASE-64 should also be resolved as HBASE 785 solved that problem
> max mapfiles to compact at one time
> -----------------------------------
>
> Key: HBASE-775
> URL: https://issues.apache.org/jira/browse/HBASE-775
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: Billy Pearson
> Assignee: Billy Pearson
> Priority: Blocker
> Fix For: 0.2.1, 0.3.0
>
> Attachments: 775-patch.txt
>
>
> Currently we compact all map files with no upper limit this could cause a regionserver to OOME if the compaction get behind and the number of mapfiles build up.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-775) max mapfiles to compact at one time
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-775:
------------------------
Priority: Blocker (was: Minor)
This should not be hard to do. Making a blocker for 0.3.0. We can move it out later.
> max mapfiles to compact at one time
> -----------------------------------
>
> Key: HBASE-775
> URL: https://issues.apache.org/jira/browse/HBASE-775
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: Billy Pearson
> Priority: Blocker
> Fix For: 0.3.0
>
>
> Currently we compact all map files with no upper limit this could cause a regionserver to OOME if the compaction get behind and the number of mapfiles build up.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-775) max mapfiles to compact at one time
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12622928#action_12622928 ]
stack commented on HBASE-775:
-----------------------------
Hey Billy: Thanks for the patch. You made size the determinant. I may be wrong, but I didn't think size mattered; the compaction is not done in RAM but by streaming a set of inputs to an output.
> max mapfiles to compact at one time
> -----------------------------------
>
> Key: HBASE-775
> URL: https://issues.apache.org/jira/browse/HBASE-775
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: Billy Pearson
> Assignee: Billy Pearson
> Priority: Blocker
> Fix For: 0.2.1, 0.3.0
>
> Attachments: 775-patch.txt
>
>
> Currently we compact all map files with no upper limit this could cause a regionserver to OOME if the compaction get behind and the number of mapfiles build up.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-775) max mapfiles to compact at one time
Posted by "Billy Pearson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Billy Pearson updated HBASE-775:
--------------------------------
Attachment: 775-patch.txt
I went with compaction size based on file sizes not count. count is to hard to figure a default for
when you do not know what the column count will be on any users setup.
This should solve the problem when we are running a normal compaction but this will not solve
The issue when it comes to compacting and force is true or if we are compacting after a split and have references.
Please review stack
The only thing missing I would like to have a debug line in there to let us know when we are hitting the compaction limit
In case a user is setting it to low and running in to problems down the road but not sure if its work the extra code and vars to log that.
> max mapfiles to compact at one time
> -----------------------------------
>
> Key: HBASE-775
> URL: https://issues.apache.org/jira/browse/HBASE-775
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: Billy Pearson
> Assignee: Billy Pearson
> Priority: Blocker
> Fix For: 0.3.0
>
> Attachments: 775-patch.txt
>
>
> Currently we compact all map files with no upper limit this could cause a regionserver to OOME if the compaction get behind and the number of mapfiles build up.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.