You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "stack (Created) (JIRA)" <ji...@apache.org> on 2012/03/22 16:12:22 UTC
[jira] [Created] (HBASE-5616) Make compaction code standalone
Make compaction code standalone
-------------------------------
Key: HBASE-5616
URL: https://issues.apache.org/jira/browse/HBASE-5616
Project: HBase
Issue Type: Improvement
Reporter: stack
Attachments: standalone.txt
This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5616) Make compaction code standalone
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5616:
-------------------------
Attachment: standalone.txt
First cut. Not finished.
> Make compaction code standalone
> -------------------------------
>
> Key: HBASE-5616
> URL: https://issues.apache.org/jira/browse/HBASE-5616
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Attachments: standalone.txt
>
>
> This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5616) Make compaction code standalone
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236989#comment-13236989 ]
Hadoop QA commented on HBASE-5616:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12519692/5616v7.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 5 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestImportTsv
org.apache.hadoop.hbase.mapred.TestTableMapReduce
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1282//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1282//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1282//console
This message is automatically generated.
> Make compaction code standalone
> -------------------------------
>
> Key: HBASE-5616
> URL: https://issues.apache.org/jira/browse/HBASE-5616
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: stack
> Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, 5616v7.txt, 5616v7.txt, standalone.txt
>
>
> This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5616) Make compaction code standalone
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236200#comment-13236200 ]
Hadoop QA commented on HBASE-5616:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12519542/5616v3.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.io.TestHeapSize
org.apache.hadoop.hbase.regionserver.TestCompaction
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1267//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1267//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1267//console
This message is automatically generated.
> Make compaction code standalone
> -------------------------------
>
> Key: HBASE-5616
> URL: https://issues.apache.org/jira/browse/HBASE-5616
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Attachments: 5616.txt, 5616v3.txt, standalone.txt
>
>
> This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5616) Make compaction code standalone
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236836#comment-13236836 ]
Hadoop QA commented on HBASE-5616:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12519660/5616v6.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 5 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1278//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1278//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1278//console
This message is automatically generated.
> Make compaction code standalone
> -------------------------------
>
> Key: HBASE-5616
> URL: https://issues.apache.org/jira/browse/HBASE-5616
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: stack
> Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, standalone.txt
>
>
> This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5616) Make compaction code standalone
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5616:
-------------------------
Status: Open (was: Patch Available)
> Make compaction code standalone
> -------------------------------
>
> Key: HBASE-5616
> URL: https://issues.apache.org/jira/browse/HBASE-5616
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Attachments: 5616.txt, 5616v3.txt, standalone.txt
>
>
> This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5616) Make compaction code standalone
Posted by "Matt Corgan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236825#comment-13236825 ]
Matt Corgan commented on HBASE-5616:
------------------------------------
btw - this feature is exactly what i need to continue optimizing the Trie DataBlockEncoder in HBASE-4676. Thanks Stack!
> Make compaction code standalone
> -------------------------------
>
> Key: HBASE-5616
> URL: https://issues.apache.org/jira/browse/HBASE-5616
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: stack
> Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, standalone.txt
>
>
> This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5616) Make compaction code standalone
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5616:
-------------------------
Status: Patch Available (was: Open)
> Make compaction code standalone
> -------------------------------
>
> Key: HBASE-5616
> URL: https://issues.apache.org/jira/browse/HBASE-5616
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Attachments: 5616.txt, standalone.txt
>
>
> This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5616) Make compaction code standalone
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511095#comment-13511095 ]
Hudson commented on HBASE-5616:
-------------------------------
Integrated in HBase-0.94 #611 (See [https://builds.apache.org/job/HBase-0.94/611/])
HBASE-7253 Backport Compaction Tool to 0.94; includes HBASE-5616, HBASE-5693, HBASE-6327, HBASE-7253 (Revision 1417559)
Result = FAILURE
stack :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionProgress.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
> Make compaction code standalone
> -------------------------------
>
> Key: HBASE-5616
> URL: https://issues.apache.org/jira/browse/HBASE-5616
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: stack
> Fix For: 0.96.0, 0.94.4
>
> Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, 5616v7.txt, 5616v7.txt, addlicense.txt, standalone.txt
>
>
> This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5616) Make compaction code standalone
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5616:
-------------------------
Attachment: 5616v7.txt
Incorporate Lars review. Shutdown access to Compactor so can be easily evolved in future.
> Make compaction code standalone
> -------------------------------
>
> Key: HBASE-5616
> URL: https://issues.apache.org/jira/browse/HBASE-5616
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: stack
> Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, 5616v7.txt, standalone.txt
>
>
> This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5616) Make compaction code standalone
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5616:
-------------------------
Attachment: addlicense.txt
Addendum. Committed. Adds missing licenses (spotted by Ted).
> Make compaction code standalone
> -------------------------------
>
> Key: HBASE-5616
> URL: https://issues.apache.org/jira/browse/HBASE-5616
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: stack
> Fix For: 0.96.0
>
> Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, 5616v7.txt, 5616v7.txt, addlicense.txt, standalone.txt
>
>
> This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5616) Make compaction code standalone
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5616:
-------------------------
Resolution: Fixed
Fix Version/s: 0.96.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to trunk.
> Make compaction code standalone
> -------------------------------
>
> Key: HBASE-5616
> URL: https://issues.apache.org/jira/browse/HBASE-5616
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: stack
> Fix For: 0.96.0
>
> Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, 5616v7.txt, 5616v7.txt, standalone.txt
>
>
> This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5616) Make compaction code standalone
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236842#comment-13236842 ]
jiraposter@reviews.apache.org commented on HBASE-5616:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4469/
-----------------------------------------------------------
(Updated 2012-03-23 17:53:11.446024)
Review request for hbase.
Changes
-------
Shutdown access to new Compactor class so can evolve it easier in future.
Summary
-------
Introduces a standalone CompactionTool under src/test. Can call it from its main
and have it run compactions on arbitrary storefiles.
Compaction code is moved out of Store into a new Compactor class.
CompactionTool needs a Store and a mocked up Region to run because
compacting uses a StoreScanner (A StoreScanner needs a Store. A
Store needs an HRegion). Rather than expect to be passed a coherent
HRegion pointer, instead, we fake up one using Mockito.
I tried to break out of HRegion a "Region" Interface. This Region Interface
would have a basic subset of HRegion functionality and we'd pass this
instead of HRegion to SplitTransaction, Store, CompactionRequest, etc.
but the change would be massive. Everything expects to be able to do
anything on an HRegion. This is work we need to do but I"m not doing
it as part of this patch.
M src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
Use new CompactionTool instead of asking Store directly to compact.
M src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java
Formatting.
M src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
Format the log message so properly spaced.
A src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java
New compactiontool. Runs all compactions.
M src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Make getOpenAndCloseThreadPool static.
M src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
Move compaction code out to CompactionTool.
Refactor so can override ttl and Store home directory so Store
is more mockable; can now stand up a Store on its own w/o real
HRegion context.
M src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
Formatting.
This addresses bug hbase-5616.
https://issues.apache.org/jira/browse/hbase-5616
Diffs (updated)
-----
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java b52e5d3
src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 4efdc6b
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java dcede5a
src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionProgress.java 553eee0
src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java d2329e1
src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 4bfd42f
src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java 91ac652
Diff: https://reviews.apache.org/r/4469/diff
Testing
-------
Thanks,
Michael
> Make compaction code standalone
> -------------------------------
>
> Key: HBASE-5616
> URL: https://issues.apache.org/jira/browse/HBASE-5616
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: stack
> Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, 5616v7.txt, standalone.txt
>
>
> This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5616) Make compaction code standalone
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236788#comment-13236788 ]
jiraposter@reviews.apache.org commented on HBASE-5616:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4469/
-----------------------------------------------------------
Review request for hbase.
Summary
-------
Introduces a standalone CompactionTool under src/test. Can call it from its main
and have it run compactions on arbitrary storefiles.
Compaction code is moved out of Store into a new Compactor class.
CompactionTool needs a Store and a mocked up Region to run because
compacting uses a StoreScanner (A StoreScanner needs a Store. A
Store needs an HRegion). Rather than expect to be passed a coherent
HRegion pointer, instead, we fake up one using Mockito.
I tried to break out of HRegion a "Region" Interface. This Region Interface
would have a basic subset of HRegion functionality and we'd pass this
instead of HRegion to SplitTransaction, Store, CompactionRequest, etc.
but the change would be massive. Everything expects to be able to do
anything on an HRegion. This is work we need to do but I"m not doing
it as part of this patch.
M src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
Use new CompactionTool instead of asking Store directly to compact.
M src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java
Formatting.
M src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
Format the log message so properly spaced.
A src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java
New compactiontool. Runs all compactions.
M src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Make getOpenAndCloseThreadPool static.
M src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
Move compaction code out to CompactionTool.
Refactor so can override ttl and Store home directory so Store
is more mockable; can now stand up a Store on its own w/o real
HRegion context.
M src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
Formatting.
This addresses bug hbase-5616.
https://issues.apache.org/jira/browse/hbase-5616
Diffs
-----
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java b52e5d3
src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 4efdc6b
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java dcede5a
src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionProgress.java 553eee0
src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java d2329e1
src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 4bfd42f
src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java 91ac652
Diff: https://reviews.apache.org/r/4469/diff
Testing
-------
Thanks,
Michael
> Make compaction code standalone
> -------------------------------
>
> Key: HBASE-5616
> URL: https://issues.apache.org/jira/browse/HBASE-5616
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: stack
> Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, standalone.txt
>
>
> This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5616) Make compaction code standalone
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5616:
-------------------------
Status: Patch Available (was: Open)
> Make compaction code standalone
> -------------------------------
>
> Key: HBASE-5616
> URL: https://issues.apache.org/jira/browse/HBASE-5616
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: stack
> Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, 5616v7.txt, 5616v7.txt, standalone.txt
>
>
> This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5616) Make compaction code standalone
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5616:
-------------------------
Attachment: 5616v7.txt
Retry. TestSplitLogManager passes locally.
> Make compaction code standalone
> -------------------------------
>
> Key: HBASE-5616
> URL: https://issues.apache.org/jira/browse/HBASE-5616
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: stack
> Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, 5616v7.txt, 5616v7.txt, standalone.txt
>
>
> This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5616) Make compaction code standalone
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5616:
-------------------------
Status: Open (was: Patch Available)
> Make compaction code standalone
> -------------------------------
>
> Key: HBASE-5616
> URL: https://issues.apache.org/jira/browse/HBASE-5616
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: stack
> Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, 5616v7.txt, 5616v7.txt, standalone.txt
>
>
> This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5616) Make compaction code standalone
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5616:
-------------------------
Status: Patch Available (was: Open)
> Make compaction code standalone
> -------------------------------
>
> Key: HBASE-5616
> URL: https://issues.apache.org/jira/browse/HBASE-5616
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: stack
> Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, standalone.txt
>
>
> This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5616) Make compaction code standalone
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236819#comment-13236819 ]
jiraposter@reviews.apache.org commented on HBASE-5616:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4469/#review6293
-----------------------------------------------------------
CompactionTool is a cool idea.
It's early, so I don't follow completely why this required moving the compaction code into a separate class, especially because you need an instance of a store to run its methods.
src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java
<https://reviews.apache.org/r/4469/#comment13643>
If you need to pass in an instance of Store, what is the advantage?
Could also make it a public static helper method.
src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java
<https://reviews.apache.org/r/4469/#comment13645>
That would have been possible before separating the compaction code out, right?
- Lars
On 2012-03-23 17:02:01, Michael Stack wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4469/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-23 17:02:01)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Introduces a standalone CompactionTool under src/test. Can call it from its main
bq. and have it run compactions on arbitrary storefiles.
bq.
bq. Compaction code is moved out of Store into a new Compactor class.
bq.
bq. CompactionTool needs a Store and a mocked up Region to run because
bq. compacting uses a StoreScanner (A StoreScanner needs a Store. A
bq. Store needs an HRegion). Rather than expect to be passed a coherent
bq. HRegion pointer, instead, we fake up one using Mockito.
bq.
bq. I tried to break out of HRegion a "Region" Interface. This Region Interface
bq. would have a basic subset of HRegion functionality and we'd pass this
bq. instead of HRegion to SplitTransaction, Store, CompactionRequest, etc.
bq. but the change would be massive. Everything expects to be able to do
bq. anything on an HRegion. This is work we need to do but I"m not doing
bq. it as part of this patch.
bq.
bq. M src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
bq. Use new CompactionTool instead of asking Store directly to compact.
bq. M src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java
bq. Formatting.
bq. M src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
bq. Format the log message so properly spaced.
bq. A src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java
bq. New compactiontool. Runs all compactions.
bq. M src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
bq. Make getOpenAndCloseThreadPool static.
bq. M src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
bq. Move compaction code out to CompactionTool.
bq. Refactor so can override ttl and Store home directory so Store
bq. is more mockable; can now stand up a Store on its own w/o real
bq. HRegion context.
bq. M src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
bq. Formatting.
bq.
bq.
bq. This addresses bug hbase-5616.
bq. https://issues.apache.org/jira/browse/hbase-5616
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java b52e5d3
bq. src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 4efdc6b
bq. src/main/java/org/apache/hadoop/hbase/regionserver/Store.java dcede5a
bq. src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionProgress.java 553eee0
bq. src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java d2329e1
bq. src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 4bfd42f
bq. src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java 91ac652
bq.
bq. Diff: https://reviews.apache.org/r/4469/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Michael
bq.
bq.
> Make compaction code standalone
> -------------------------------
>
> Key: HBASE-5616
> URL: https://issues.apache.org/jira/browse/HBASE-5616
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: stack
> Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, standalone.txt
>
>
> This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5616) Make compaction code standalone
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5616:
-------------------------
Attachment: 5616.txt
Patch that will run compactions from command line. By default does not destroy the input so can run it again.
> Make compaction code standalone
> -------------------------------
>
> Key: HBASE-5616
> URL: https://issues.apache.org/jira/browse/HBASE-5616
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Attachments: 5616.txt, standalone.txt
>
>
> This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5616) Make compaction code standalone
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5616:
-------------------------
Assignee: stack
Status: Open (was: Patch Available)
> Make compaction code standalone
> -------------------------------
>
> Key: HBASE-5616
> URL: https://issues.apache.org/jira/browse/HBASE-5616
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: stack
> Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, standalone.txt
>
>
> This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5616) Make compaction code standalone
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237167#comment-13237167 ]
jiraposter@reviews.apache.org commented on HBASE-5616:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4469/#review6309
-----------------------------------------------------------
Ship it!
src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java
<https://reviews.apache.org/r/4469/#comment13696>
License.
src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java
<https://reviews.apache.org/r/4469/#comment13700>
License
- Ted
On 2012-03-23 17:53:11, Michael Stack wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4469/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-23 17:53:11)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Introduces a standalone CompactionTool under src/test. Can call it from its main
bq. and have it run compactions on arbitrary storefiles.
bq.
bq. Compaction code is moved out of Store into a new Compactor class.
bq.
bq. CompactionTool needs a Store and a mocked up Region to run because
bq. compacting uses a StoreScanner (A StoreScanner needs a Store. A
bq. Store needs an HRegion). Rather than expect to be passed a coherent
bq. HRegion pointer, instead, we fake up one using Mockito.
bq.
bq. I tried to break out of HRegion a "Region" Interface. This Region Interface
bq. would have a basic subset of HRegion functionality and we'd pass this
bq. instead of HRegion to SplitTransaction, Store, CompactionRequest, etc.
bq. but the change would be massive. Everything expects to be able to do
bq. anything on an HRegion. This is work we need to do but I"m not doing
bq. it as part of this patch.
bq.
bq. M src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
bq. Use new CompactionTool instead of asking Store directly to compact.
bq. M src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java
bq. Formatting.
bq. M src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
bq. Format the log message so properly spaced.
bq. A src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java
bq. New compactiontool. Runs all compactions.
bq. M src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
bq. Make getOpenAndCloseThreadPool static.
bq. M src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
bq. Move compaction code out to CompactionTool.
bq. Refactor so can override ttl and Store home directory so Store
bq. is more mockable; can now stand up a Store on its own w/o real
bq. HRegion context.
bq. M src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
bq. Formatting.
bq.
bq.
bq. This addresses bug hbase-5616.
bq. https://issues.apache.org/jira/browse/hbase-5616
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java b52e5d3
bq. src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 4efdc6b
bq. src/main/java/org/apache/hadoop/hbase/regionserver/Store.java dcede5a
bq. src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionProgress.java 553eee0
bq. src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java d2329e1
bq. src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 4bfd42f
bq. src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java 91ac652
bq.
bq. Diff: https://reviews.apache.org/r/4469/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Michael
bq.
bq.
> Make compaction code standalone
> -------------------------------
>
> Key: HBASE-5616
> URL: https://issues.apache.org/jira/browse/HBASE-5616
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: stack
> Fix For: 0.96.0
>
> Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, 5616v7.txt, 5616v7.txt, standalone.txt
>
>
> This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5616) Make compaction code standalone
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236054#comment-13236054 ]
Hadoop QA commented on HBASE-5616:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12519506/5616.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The patch appears to cause mvn compile goal to fail.
-1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1265//testReport/
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1265//console
This message is automatically generated.
> Make compaction code standalone
> -------------------------------
>
> Key: HBASE-5616
> URL: https://issues.apache.org/jira/browse/HBASE-5616
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Attachments: 5616.txt, standalone.txt
>
>
> This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5616) Make compaction code standalone
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236860#comment-13236860 ]
jiraposter@reviews.apache.org commented on HBASE-5616:
------------------------------------------------------
bq. On 2012-03-23 17:32:04, Lars Hofhansl wrote:
bq. > CompactionTool is a cool idea.
bq. > It's early, so I don't follow completely why this required moving the compaction code into a separate class, especially because you need an instance of a store to run its methods.
bq.
bq. Michael Stack wrote:
bq. Store is 2200 lines long.
bq.
bq. Need to make a start somewhere breaking up these Store and HRegion, etc., behemoths. The tendency otherwise is that they get bigger with time.
bq.
bq. Also, arguing its all a ball of wax so we should shut down any attempt at untangling is probably not what you were reasoning (smile).
bq.
bq. My main motive breaking out CompactionTool made is so I could run compactions in a profiler w/o a bunch of other moving parts in the way. I did some of that yesterday. Nothing obviously dumb going on at first glance. Requires more study figuring optimizations.
bq.
bq. Thanks for the review.
Fair enough.
- Lars
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4469/#review6293
-----------------------------------------------------------
On 2012-03-23 17:53:11, Michael Stack wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4469/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-23 17:53:11)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Introduces a standalone CompactionTool under src/test. Can call it from its main
bq. and have it run compactions on arbitrary storefiles.
bq.
bq. Compaction code is moved out of Store into a new Compactor class.
bq.
bq. CompactionTool needs a Store and a mocked up Region to run because
bq. compacting uses a StoreScanner (A StoreScanner needs a Store. A
bq. Store needs an HRegion). Rather than expect to be passed a coherent
bq. HRegion pointer, instead, we fake up one using Mockito.
bq.
bq. I tried to break out of HRegion a "Region" Interface. This Region Interface
bq. would have a basic subset of HRegion functionality and we'd pass this
bq. instead of HRegion to SplitTransaction, Store, CompactionRequest, etc.
bq. but the change would be massive. Everything expects to be able to do
bq. anything on an HRegion. This is work we need to do but I"m not doing
bq. it as part of this patch.
bq.
bq. M src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
bq. Use new CompactionTool instead of asking Store directly to compact.
bq. M src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java
bq. Formatting.
bq. M src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
bq. Format the log message so properly spaced.
bq. A src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java
bq. New compactiontool. Runs all compactions.
bq. M src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
bq. Make getOpenAndCloseThreadPool static.
bq. M src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
bq. Move compaction code out to CompactionTool.
bq. Refactor so can override ttl and Store home directory so Store
bq. is more mockable; can now stand up a Store on its own w/o real
bq. HRegion context.
bq. M src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
bq. Formatting.
bq.
bq.
bq. This addresses bug hbase-5616.
bq. https://issues.apache.org/jira/browse/hbase-5616
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java b52e5d3
bq. src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 4efdc6b
bq. src/main/java/org/apache/hadoop/hbase/regionserver/Store.java dcede5a
bq. src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionProgress.java 553eee0
bq. src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java d2329e1
bq. src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 4bfd42f
bq. src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java 91ac652
bq.
bq. Diff: https://reviews.apache.org/r/4469/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Michael
bq.
bq.
> Make compaction code standalone
> -------------------------------
>
> Key: HBASE-5616
> URL: https://issues.apache.org/jira/browse/HBASE-5616
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: stack
> Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, 5616v7.txt, standalone.txt
>
>
> This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5616) Make compaction code standalone
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5616:
-------------------------
Attachment: 5616v6.txt
Fixed the test failure.
Also changed my mind moving CompactionTool out to src/test rather than have it in main code base. I kept the breaking out of the compaction code from Store into its own class and made it so its possible to stand up a single Store in a unit test.
Until CompactionTool can be done w/o my need of the Mockito crutch because the code is a hairball, I think CompactionTool has to be out in test.
> Make compaction code standalone
> -------------------------------
>
> Key: HBASE-5616
> URL: https://issues.apache.org/jira/browse/HBASE-5616
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, standalone.txt
>
>
> This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5616) Make compaction code standalone
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5616:
-------------------------
Status: Patch Available (was: Open)
> Make compaction code standalone
> -------------------------------
>
> Key: HBASE-5616
> URL: https://issues.apache.org/jira/browse/HBASE-5616
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Attachments: 5616.txt, 5616v3.txt, standalone.txt
>
>
> This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5616) Make compaction code standalone
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237532#comment-13237532 ]
Hudson commented on HBASE-5616:
-------------------------------
Integrated in HBase-TRUNK #2694 (See [https://builds.apache.org/job/HBase-TRUNK/2694/])
HBASE-5616 Make compaction code standalone; ADDENDUM -- ADD LICENSES (Revision 1304624)
HBASE-5616 Make compaction code standalone (Revision 1304616)
Result = SUCCESS
stack :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java
stack :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionProgress.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
> Make compaction code standalone
> -------------------------------
>
> Key: HBASE-5616
> URL: https://issues.apache.org/jira/browse/HBASE-5616
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: stack
> Fix For: 0.96.0
>
> Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, 5616v7.txt, 5616v7.txt, addlicense.txt, standalone.txt
>
>
> This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5616) Make compaction code standalone
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5616:
-------------------------
Attachment: 5616v3.txt
Introduces a standalone CompactionTool. Can call it from its main
and have it run compactions on arbitrary storefiles. Compaction code
is moved out of Store into this new class.
CompactionTool needs a Store and a mocked up Region to run because
compacting uses a StoreScanner (A StoreScanner needs a Store. A
Store needs an HRegion). Rather than expect to be passed a coherent
HRegion pointer, instead, we fake up one using Mockito (this means
Mockito is no longer just a test dependency).
I tried to break out of HRegion a "Region" Interface. This Region Interface
would have a basic subset of HRegion functionality and we'd pass this
instead of HRegion to SplitTransaction, Store, CompactionRequest, etc.
but the change would be massive. Everything expects to be able to do
anything on an HRegion. This is work we need to do but I"m not doing
it as part of this patch.
M src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
Use new CompactionTool instead of asking Store directly to compact.
M src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java
Formatting.
M src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
Format the log message so properly spaced.
A src/main/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java
New compactiontool. Runs all compactions.
M src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Make getOpenAndCloseThreadPool static.
M src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
Move compaction code out to CompactionTool.
Refactor so can override ttl and Store home directory so Store
is more mockable; can now stand up a Store on its own w/o real
HRegion context.
M src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
Formatting.
M pom.xml
Need mockito at runtime now too.
> Make compaction code standalone
> -------------------------------
>
> Key: HBASE-5616
> URL: https://issues.apache.org/jira/browse/HBASE-5616
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Attachments: 5616.txt, 5616v3.txt, standalone.txt
>
>
> This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5616) Make compaction code standalone
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236921#comment-13236921 ]
Hadoop QA commented on HBASE-5616:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12519671/5616v7.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 5 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplication
org.apache.hadoop.hbase.mapreduce.TestImportTsv
org.apache.hadoop.hbase.mapred.TestTableMapReduce
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
org.apache.hadoop.hbase.master.TestSplitLogManager
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1279//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1279//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1279//console
This message is automatically generated.
> Make compaction code standalone
> -------------------------------
>
> Key: HBASE-5616
> URL: https://issues.apache.org/jira/browse/HBASE-5616
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: stack
> Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, 5616v7.txt, standalone.txt
>
>
> This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5616) Make compaction code standalone
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236835#comment-13236835 ]
jiraposter@reviews.apache.org commented on HBASE-5616:
------------------------------------------------------
bq. On 2012-03-23 17:32:04, Lars Hofhansl wrote:
bq. > CompactionTool is a cool idea.
bq. > It's early, so I don't follow completely why this required moving the compaction code into a separate class, especially because you need an instance of a store to run its methods.
Store is 2200 lines long.
Need to make a start somewhere breaking up these Store and HRegion, etc., behemoths. The tendency otherwise is that they get bigger with time.
Also, arguing its all a ball of wax so we should shut down any attempt at untangling is probably not what you were reasoning (smile).
My main motive breaking out CompactionTool made is so I could run compactions in a profiler w/o a bunch of other moving parts in the way. I did some of that yesterday. Nothing obviously dumb going on at first glance. Requires more study figuring optimizations.
Thanks for the review.
bq. On 2012-03-23 17:32:04, Lars Hofhansl wrote:
bq. > src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java, line 45
bq. > <https://reviews.apache.org/r/4469/diff/1/?file=95053#file95053line45>
bq. >
bq. > If you need to pass in an instance of Store, what is the advantage?
bq. > Could also make it a public static helper method.
I could do that. If I went this way, it'd be harder to evolve though (public, static passing in all args).
This review comment makes me want to shutdown access to Compactor even more -- go via CompactionTool if you need it. I should add @InterfaceAudience.Private and remove the 'public' from the Class. If you are good w/ this, I'll stick up another patch w/ that on it.
bq. On 2012-03-23 17:32:04, Lars Hofhansl wrote:
bq. > src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java, line 135
bq. > <https://reviews.apache.org/r/4469/diff/1/?file=95059#file95059line135>
bq. >
bq. > That would have been possible before separating the compaction code out, right?
Yes. Would have made no sense since only the one client.
- Michael
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4469/#review6293
-----------------------------------------------------------
On 2012-03-23 17:02:01, Michael Stack wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4469/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-23 17:02:01)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Introduces a standalone CompactionTool under src/test. Can call it from its main
bq. and have it run compactions on arbitrary storefiles.
bq.
bq. Compaction code is moved out of Store into a new Compactor class.
bq.
bq. CompactionTool needs a Store and a mocked up Region to run because
bq. compacting uses a StoreScanner (A StoreScanner needs a Store. A
bq. Store needs an HRegion). Rather than expect to be passed a coherent
bq. HRegion pointer, instead, we fake up one using Mockito.
bq.
bq. I tried to break out of HRegion a "Region" Interface. This Region Interface
bq. would have a basic subset of HRegion functionality and we'd pass this
bq. instead of HRegion to SplitTransaction, Store, CompactionRequest, etc.
bq. but the change would be massive. Everything expects to be able to do
bq. anything on an HRegion. This is work we need to do but I"m not doing
bq. it as part of this patch.
bq.
bq. M src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
bq. Use new CompactionTool instead of asking Store directly to compact.
bq. M src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java
bq. Formatting.
bq. M src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
bq. Format the log message so properly spaced.
bq. A src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java
bq. New compactiontool. Runs all compactions.
bq. M src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
bq. Make getOpenAndCloseThreadPool static.
bq. M src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
bq. Move compaction code out to CompactionTool.
bq. Refactor so can override ttl and Store home directory so Store
bq. is more mockable; can now stand up a Store on its own w/o real
bq. HRegion context.
bq. M src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
bq. Formatting.
bq.
bq.
bq. This addresses bug hbase-5616.
bq. https://issues.apache.org/jira/browse/hbase-5616
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java b52e5d3
bq. src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 4efdc6b
bq. src/main/java/org/apache/hadoop/hbase/regionserver/Store.java dcede5a
bq. src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionProgress.java 553eee0
bq. src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java d2329e1
bq. src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 4bfd42f
bq. src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java 91ac652
bq.
bq. Diff: https://reviews.apache.org/r/4469/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Michael
bq.
bq.
> Make compaction code standalone
> -------------------------------
>
> Key: HBASE-5616
> URL: https://issues.apache.org/jira/browse/HBASE-5616
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: stack
> Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, standalone.txt
>
>
> This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5616) Make compaction code standalone
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5616:
-------------------------
Fix Version/s: 0.94.4
Committed to 0.94 as part of HBASE-7253
> Make compaction code standalone
> -------------------------------
>
> Key: HBASE-5616
> URL: https://issues.apache.org/jira/browse/HBASE-5616
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: stack
> Fix For: 0.96.0, 0.94.4
>
> Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, 5616v7.txt, 5616v7.txt, addlicense.txt, standalone.txt
>
>
> This is part of hbase-2462. Make the compaction code standalone so can run it independent of hbase. Will make it easier to profile and try stuff out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira