You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Matteo Bertozzi (JIRA)" <ji...@apache.org> on 2012/12/05 18:29:59 UTC
[jira] [Created] (HBASE-7282) Backport Compaction Tool to 0.94
Matteo Bertozzi created HBASE-7282:
--------------------------------------
Summary: Backport Compaction Tool to 0.94
Key: HBASE-7282
URL: https://issues.apache.org/jira/browse/HBASE-7282
Project: HBase
Issue Type: Sub-task
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
Fix For: 0.94.4
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7282) Backport Compaction Tool to 0.94
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511178#comment-13511178 ]
stack commented on HBASE-7282:
------------------------------
I made the fat patch by applying one after the other. Patch hiccuped? Or I did applying them. I could redo?
You'd set "hbase.hstore.compaction.complete" to false when you want to compact once only (rather than compact and then when done, if we find we could compact again, go ahead and do it). Correct me if I'm wrong [~mbertozzi]
> Backport Compaction Tool to 0.94
> --------------------------------
>
> Key: HBASE-7282
> URL: https://issues.apache.org/jira/browse/HBASE-7282
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction
> Reporter: Matteo Bertozzi
> Assignee: Matteo Bertozzi
> Priority: Minor
> Fix For: 0.94.4
>
> Attachments: 7253.txt, 7253-v2.txt, HBASE-5616-0.94.patch, HBASE-5693-0.94.patch, HBASE-6327-0.94.patch, HBASE-7253-0.94.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7282) Backport Compaction Tool to 0.94
Posted by "Matteo Bertozzi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matteo Bertozzi updated HBASE-7282:
-----------------------------------
Attachment: HBASE-7253-0.94.patch
HBASE-6327-0.94.patch
HBASE-5693-0.94.patch
HBASE-5616-0.94.patch
Added CompactionTool (HBASE-7253) and dependencies patch for 0.94
* HBASE-5693 - When creating a region, the master initializes it and creates a memstore within the master server
* HBASE-6327 - HLog can be null when create table
* HBASE-5616 - Make compaction code standalone
* HBASE-7253 - Compaction Tool
HBASE-5693 and HBASE-6327 are included just because is a refactor of the HRegion.createHRegion(), used by the CompactionTool, and the patch is small.
> Backport Compaction Tool to 0.94
> --------------------------------
>
> Key: HBASE-7282
> URL: https://issues.apache.org/jira/browse/HBASE-7282
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction
> Reporter: Matteo Bertozzi
> Assignee: Matteo Bertozzi
> Priority: Minor
> Fix For: 0.94.4
>
> Attachments: HBASE-5616-0.94.patch, HBASE-5693-0.94.patch, HBASE-6327-0.94.patch, HBASE-7253-0.94.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7282) Backport Compaction Tool to 0.94
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511181#comment-13511181 ]
Lars Hofhansl commented on HBASE-7282:
--------------------------------------
I see. +1 commit v2 to 0.94. This is too nice of a tool to pass on.
(v2 is almost identical to your version; it just re-adds the call to preCompactScannerOpen that the first one accidentally removed).
> Backport Compaction Tool to 0.94
> --------------------------------
>
> Key: HBASE-7282
> URL: https://issues.apache.org/jira/browse/HBASE-7282
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction
> Reporter: Matteo Bertozzi
> Assignee: Matteo Bertozzi
> Priority: Minor
> Fix For: 0.94.4
>
> Attachments: 7253.txt, 7253-v2.txt, HBASE-5616-0.94.patch, HBASE-5693-0.94.patch, HBASE-6327-0.94.patch, HBASE-7253-0.94.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7282) Backport Compaction Tool to 0.94
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511143#comment-13511143 ]
Lars Hofhansl commented on HBASE-7282:
--------------------------------------
Looking at the patch again, it seems that just the call to preCompactScannerOpen got lost in the patch.
> Backport Compaction Tool to 0.94
> --------------------------------
>
> Key: HBASE-7282
> URL: https://issues.apache.org/jira/browse/HBASE-7282
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction
> Reporter: Matteo Bertozzi
> Assignee: Matteo Bertozzi
> Priority: Minor
> Fix For: 0.94.4
>
> Attachments: 7253.txt, HBASE-5616-0.94.patch, HBASE-5693-0.94.patch, HBASE-6327-0.94.patch, HBASE-7253-0.94.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7282) Backport Compaction Tool to 0.94
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-7282:
---------------------------------
Attachment: 7253-v2.txt
Here's a patch that fixes the issue.
For this I copied the original compaction from Store.java into Compactor.compact(...) and changed it accordingly. Then I checked that I did come with something similar to what you had.
The only difference was the call the preCompactScannerOpen.
With this the three above mentioned tests pass.
Of course now I am a bit less confident in the rest of the patch.
> Backport Compaction Tool to 0.94
> --------------------------------
>
> Key: HBASE-7282
> URL: https://issues.apache.org/jira/browse/HBASE-7282
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction
> Reporter: Matteo Bertozzi
> Assignee: Matteo Bertozzi
> Priority: Minor
> Fix For: 0.94.4
>
> Attachments: 7253.txt, 7253-v2.txt, HBASE-5616-0.94.patch, HBASE-5693-0.94.patch, HBASE-6327-0.94.patch, HBASE-7253-0.94.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7282) Backport Compaction Tool to 0.94
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511136#comment-13511136 ]
stack commented on HBASE-7282:
------------------------------
Reverted. Sorry for breakage. Confirmed at least TestZooKeeperScanPolicyObserver passes again. This is a nice facility and can get fellows who have overrun their hbase w/ too many files out of their hole. Will see what Matteo says.
> Backport Compaction Tool to 0.94
> --------------------------------
>
> Key: HBASE-7282
> URL: https://issues.apache.org/jira/browse/HBASE-7282
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction
> Reporter: Matteo Bertozzi
> Assignee: Matteo Bertozzi
> Priority: Minor
> Fix For: 0.94.4
>
> Attachments: 7253.txt, HBASE-5616-0.94.patch, HBASE-5693-0.94.patch, HBASE-6327-0.94.patch, HBASE-7253-0.94.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7282) Backport Compaction Tool to 0.94
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511167#comment-13511167 ]
Lars Hofhansl commented on HBASE-7282:
--------------------------------------
Rest of the patch looks good. What is this:
{code}
// Move the compaction into place.
- sf = completeCompaction(filesToCompact, writer);
- if (region.getCoprocessorHost() != null) {
- region.getCoprocessorHost().postCompact(this, sf);
+ if (this.conf.getBoolean("hbase.hstore.compaction.complete", true)) {
+ sf = completeCompaction(filesToCompact, writer);
+ if (region.getCoprocessorHost() != null) {
+ region.getCoprocessorHost().postCompact(this, sf);
+ }
+ } else {
+ // Create storefile around what we wrote with a reader on it.
+ sf = new StoreFile(this.fs, writer.getPath(), this.conf, this.cacheConf,
+ this.family.getBloomFilterType(), this.dataBlockEncoder);
+ sf.createReader();
}
{code}
What is "hbase.hstore.compaction.complete"? And why would ever not want to have this true? Was that wrongly pulled in from trunk?
> Backport Compaction Tool to 0.94
> --------------------------------
>
> Key: HBASE-7282
> URL: https://issues.apache.org/jira/browse/HBASE-7282
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction
> Reporter: Matteo Bertozzi
> Assignee: Matteo Bertozzi
> Priority: Minor
> Fix For: 0.94.4
>
> Attachments: 7253.txt, 7253-v2.txt, HBASE-5616-0.94.patch, HBASE-5693-0.94.patch, HBASE-6327-0.94.patch, HBASE-7253-0.94.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HBASE-7282) Backport Compaction Tool to 0.94
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl reopened HBASE-7282:
----------------------------------
> Backport Compaction Tool to 0.94
> --------------------------------
>
> Key: HBASE-7282
> URL: https://issues.apache.org/jira/browse/HBASE-7282
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction
> Reporter: Matteo Bertozzi
> Assignee: Matteo Bertozzi
> Priority: Minor
> Fix For: 0.94.4
>
> Attachments: 7253.txt, HBASE-5616-0.94.patch, HBASE-5693-0.94.patch, HBASE-6327-0.94.patch, HBASE-7253-0.94.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7282) Backport Compaction Tool to 0.94
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511141#comment-13511141 ]
Lars Hofhansl commented on HBASE-7282:
--------------------------------------
Probably just needs some verification that the code in the new Compactor class is the same as it was before... Maybe on RB it is easier to verify? (Is it even possible to put a 0.94 up on RB?)
Yeah, I loved the M/R compaction and compaction tool when I read the description/patch.
In fact I like that idea so much that we should still try to get into 0.94.
> Backport Compaction Tool to 0.94
> --------------------------------
>
> Key: HBASE-7282
> URL: https://issues.apache.org/jira/browse/HBASE-7282
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction
> Reporter: Matteo Bertozzi
> Assignee: Matteo Bertozzi
> Priority: Minor
> Fix For: 0.94.4
>
> Attachments: 7253.txt, HBASE-5616-0.94.patch, HBASE-5693-0.94.patch, HBASE-6327-0.94.patch, HBASE-7253-0.94.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7282) Backport Compaction Tool to 0.94
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511125#comment-13511125 ]
Lars Hofhansl commented on HBASE-7282:
--------------------------------------
This change breaks at least these tests:
* TestZooKeeperScanPolicyObserver.testScanPolicyObserver
* TestCoprocessorScanPolicy.testBaseCases
* TestCoprocessorScanPolicy.testTTL
Verified by selectively reverting the recent patches.
> Backport Compaction Tool to 0.94
> --------------------------------
>
> Key: HBASE-7282
> URL: https://issues.apache.org/jira/browse/HBASE-7282
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction
> Reporter: Matteo Bertozzi
> Assignee: Matteo Bertozzi
> Priority: Minor
> Fix For: 0.94.4
>
> Attachments: 7253.txt, HBASE-5616-0.94.patch, HBASE-5693-0.94.patch, HBASE-6327-0.94.patch, HBASE-7253-0.94.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7282) Backport Compaction Tool to 0.94
Posted by "Matteo Bertozzi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matteo Bertozzi updated HBASE-7282:
-----------------------------------
Status: Patch Available (was: Open)
> Backport Compaction Tool to 0.94
> --------------------------------
>
> Key: HBASE-7282
> URL: https://issues.apache.org/jira/browse/HBASE-7282
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction
> Reporter: Matteo Bertozzi
> Assignee: Matteo Bertozzi
> Priority: Minor
> Fix For: 0.94.4
>
> Attachments: HBASE-5616-0.94.patch, HBASE-5693-0.94.patch, HBASE-6327-0.94.patch, HBASE-7253-0.94.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7282) Backport Compaction Tool to 0.94
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511130#comment-13511130 ]
Lars Hofhansl commented on HBASE-7282:
--------------------------------------
This is quite large change for 0.94 anyway, methinks.
Maybe we should revert for now and upload to RB?
> Backport Compaction Tool to 0.94
> --------------------------------
>
> Key: HBASE-7282
> URL: https://issues.apache.org/jira/browse/HBASE-7282
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction
> Reporter: Matteo Bertozzi
> Assignee: Matteo Bertozzi
> Priority: Minor
> Fix For: 0.94.4
>
> Attachments: 7253.txt, HBASE-5616-0.94.patch, HBASE-5693-0.94.patch, HBASE-6327-0.94.patch, HBASE-7253-0.94.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7282) Backport Compaction Tool to 0.94
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-7282:
-------------------------
Attachment: 7253.txt
This is what I committed. Its all the patches applied one after the other and some fix up around the HRegion data members to address a patch reject.
> Backport Compaction Tool to 0.94
> --------------------------------
>
> Key: HBASE-7282
> URL: https://issues.apache.org/jira/browse/HBASE-7282
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction
> Reporter: Matteo Bertozzi
> Assignee: Matteo Bertozzi
> Priority: Minor
> Fix For: 0.94.4
>
> Attachments: 7253.txt, HBASE-5616-0.94.patch, HBASE-5693-0.94.patch, HBASE-6327-0.94.patch, HBASE-7253-0.94.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7282) Backport Compaction Tool to 0.94
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-7282:
-------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to 0.94. Thanks for the patches Matteo.
> Backport Compaction Tool to 0.94
> --------------------------------
>
> Key: HBASE-7282
> URL: https://issues.apache.org/jira/browse/HBASE-7282
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction
> Reporter: Matteo Bertozzi
> Assignee: Matteo Bertozzi
> Priority: Minor
> Fix For: 0.94.4
>
> Attachments: HBASE-5616-0.94.patch, HBASE-5693-0.94.patch, HBASE-6327-0.94.patch, HBASE-7253-0.94.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira