You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Nicolas Spiegelberg (JIRA)" <ji...@apache.org> on 2010/11/02 22:23:07 UTC
[jira] Created: (HBASE-3189) Stagger Major Compactions
Stagger Major Compactions
-------------------------
Key: HBASE-3189
URL: https://issues.apache.org/jira/browse/HBASE-3189
Project: HBase
Issue Type: Bug
Components: regionserver
Reporter: Nicolas Spiegelberg
Assignee: Nicolas Spiegelberg
Priority: Minor
For pre-split regions, we can get into a case where the oldest HFile in a Store is pretty large and will not encounter a compaction within the 24hr major compact window. If that's the case, we don't want multiple multi-GB major compactions being triggered at the same time. Add ability to stagger the major compaction expiration window.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-3189) Stagger Major Compactions
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-3189.
--------------------------
Resolution: Fixed
Fix Version/s: 0.90.0
Hadoop Flags: [Reviewed]
Committed. Thanks for the patch Nicolas.
> Stagger Major Compactions
> -------------------------
>
> Key: HBASE-3189
> URL: https://issues.apache.org/jira/browse/HBASE-3189
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Priority: Minor
> Fix For: 0.90.0
>
> Attachments: HBASE-3189.patch
>
>
> For pre-split regions, we can get into a case where the oldest HFile in a Store is pretty large and will not encounter a compaction within the 24hr major compact window. If that's the case, we don't want multiple multi-GB major compactions being triggered at the same time. Add ability to stagger the major compaction expiration window.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3189) Stagger Major Compactions
Posted by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928122#action_12928122 ]
Kannan Muthukkaruppan commented on HBASE-3189:
----------------------------------------------
I think, usability wise, jitter (and it's default) should be specified as fraction (% value) of the major compaction cycle time, instead of absolute terms (like 4 hours)/
Otherwise, you have a backward compat issue with this change for someone who is running a major compaction say every three hours, but has forgotten to set the jitter parameter when they upgrade to 0.90. And they'll be compacting anywhere from 3hrs +/- (2* 4 hours jitter default). This approach will also ensure you don't return -ve values for "get next compaction time".
> Stagger Major Compactions
> -------------------------
>
> Key: HBASE-3189
> URL: https://issues.apache.org/jira/browse/HBASE-3189
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Priority: Minor
> Fix For: 0.90.0
>
> Attachments: HBASE-3189.patch
>
>
> For pre-split regions, we can get into a case where the oldest HFile in a Store is pretty large and will not encounter a compaction within the 24hr major compact window. If that's the case, we don't want multiple multi-GB major compactions being triggered at the same time. Add ability to stagger the major compaction expiration window.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-3189) Stagger Major Compactions
Posted by "Nicolas Spiegelberg (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nicolas Spiegelberg updated HBASE-3189:
---------------------------------------
Attachment: HBASE-3189-addendum.patch
addresses Kannan's issue. this has been accepted/committed internally.
> Stagger Major Compactions
> -------------------------
>
> Key: HBASE-3189
> URL: https://issues.apache.org/jira/browse/HBASE-3189
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Priority: Minor
> Fix For: 0.90.0
>
> Attachments: HBASE-3189-addendum.patch, HBASE-3189.patch
>
>
> For pre-split regions, we can get into a case where the oldest HFile in a Store is pretty large and will not encounter a compaction within the 24hr major compact window. If that's the case, we don't want multiple multi-GB major compactions being triggered at the same time. Add ability to stagger the major compaction expiration window.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3189) Stagger Major Compactions
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928423#action_12928423 ]
stack commented on HBASE-3189:
------------------------------
Committed the addendum to TRUNK just now.
> Stagger Major Compactions
> -------------------------
>
> Key: HBASE-3189
> URL: https://issues.apache.org/jira/browse/HBASE-3189
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Priority: Minor
> Fix For: 0.90.0
>
> Attachments: HBASE-3189-addendum.patch, HBASE-3189.patch
>
>
> For pre-split regions, we can get into a case where the oldest HFile in a Store is pretty large and will not encounter a compaction within the 24hr major compact window. If that's the case, we don't want multiple multi-GB major compactions being triggered at the same time. Add ability to stagger the major compaction expiration window.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-3189) Stagger Major Compactions
Posted by "Nicolas Spiegelberg (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nicolas Spiegelberg updated HBASE-3189:
---------------------------------------
Attachment: HBASE-3189.patch
Fix for 0.90. Adds a 'jitter' config option to stagger compactions on a per-store basis.
> Stagger Major Compactions
> -------------------------
>
> Key: HBASE-3189
> URL: https://issues.apache.org/jira/browse/HBASE-3189
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Priority: Minor
> Attachments: HBASE-3189.patch
>
>
> For pre-split regions, we can get into a case where the oldest HFile in a Store is pretty large and will not encounter a compaction within the 24hr major compact window. If that's the case, we don't want multiple multi-GB major compactions being triggered at the same time. Add ability to stagger the major compaction expiration window.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3189) Stagger Major Compactions
Posted by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928123#action_12928123 ]
Kannan Muthukkaruppan commented on HBASE-3189:
----------------------------------------------
I think, usability wise, jitter (and it's default) should be specified as fraction (% value) of the major compaction cycle time, instead of absolute terms (like 4 hours)/
Otherwise, you have a backward compat issue with this change for someone who is running a major compaction say every three hours, but has forgotten to set the jitter parameter when they upgrade to 0.90. And they'll be compacting anywhere from 3hrs +/- (2* 4 hours jitter default). This approach will also ensure you don't return -ve values for "get next compaction time".
> Stagger Major Compactions
> -------------------------
>
> Key: HBASE-3189
> URL: https://issues.apache.org/jira/browse/HBASE-3189
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Priority: Minor
> Fix For: 0.90.0
>
> Attachments: HBASE-3189.patch
>
>
> For pre-split regions, we can get into a case where the oldest HFile in a Store is pretty large and will not encounter a compaction within the 24hr major compact window. If that's the case, we don't want multiple multi-GB major compactions being triggered at the same time. Add ability to stagger the major compaction expiration window.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.