You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Mikhail Bautin (Created) (JIRA)" <ji...@apache.org> on 2011/11/30 23:49:40 UTC
[jira] [Created] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
HBase cluster test tool (port from 0.89-fb)
-------------------------------------------
Key: HBASE-4908
URL: https://issues.apache.org/jira/browse/HBASE-4908
Project: HBase
Issue Type: Test
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Bautin updated HBASE-4908:
----------------------------------
Status: Open (was: Patch Available)
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: 0001-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13162041#comment-13162041 ]
Phabricator commented on HBASE-4908:
------------------------------------
mbautin has commented on the revision "[jira] [HBASE-4908] HBase cluster test tool (port from 0.89-fb)".
Some replies inline. I will remove HdfsAppender test as well -- it seems unrelated to the main purpose of the load tester tool.
INLINE COMMENTS
src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:17 Moved to org.apache.hadoop.hbase.util.
src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:92 Done.
src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:103 Made 20 a constant and removed separate logic for the (endKey - startKey) <= 1000 case.
src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:204 Mentioned this in the class comment.
src/test/java/org/apache/hadoop/hbase/manual/RestartMetaTest.java:36 Added.
src/test/java/org/apache/hadoop/hbase/manual/RestartMetaTest.java:152 There is not a lot of overlap of functionality, but perhaps some overlap in structure, e.g. command-line argument processing. I'd like to avoid doing a lot of refactoring until the next iteration. Ideally it should be possible to run every integration test both as a JUnit test and as a command-line tool.
REVISION DETAIL
https://reviews.facebook.net/D549
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13160495#comment-13160495 ]
Phabricator commented on HBASE-4908:
------------------------------------
stack has commented on the revision "[jira] [HBASE-4908] HBase cluster test tool (port from 0.89-fb)".
@Nicolas Sure on the distinction but PE can do threaded or MR'd workloads and has a means of plugging in different loading profiles (subclass Client and then implement run). I'd think it would be better able to put up a sustained loading when loading is coming out of MR (in spite of its ugly name -- name comes from BT paper).
I'm fine w/ this tool going in as is (after some cleanup). Can work then separately on unifying the two if it makes sense as a different project.
REVISION DETAIL
https://reviews.facebook.net/D549
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D549.1.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171453#comment-13171453 ]
Hudson commented on HBASE-4908:
-------------------------------
Integrated in HBase-TRUNK #2553 (See [https://builds.apache.org/job/HBase-TRUNK/2553/])
HBASE-4935 hbase 0.92.0 doesn't work going against 0.20.205.0, its packaged hadoop -- revert from TRUNK though this change was added as part of HBASE-4908
stack :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Fix For: 0.94.0
>
> Attachments: 0001-HBase-cluster-test-tool.patch, 0002-HBase-cluster-test-tool.patch, 0003-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch, D549.8.patch, D549.9.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-4908:
-------------------------------
Attachment: D549.9.patch
mbautin updated the revision "[jira] [HBASE-4908] HBase cluster test tool (port from 0.89-fb)".
Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA
Fixing how reader and writer are linked together in the mixed workload. Verified that LoadTestTool works in both reader-only, writer-only, and writer/reader modes on a dev cluster. Unit test failures with and without the patch are identical. I think this patch is good to be committed.
REVISION DETAIL
https://reviews.facebook.net/D549
AFFECTED FILES
pom.xml
src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java
src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java
src/main/java/org/apache/hadoop/hbase/HConstants.java
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
src/main/java/org/apache/hadoop/hbase/util/AbstractHBaseTool.java
src/main/java/org/apache/hadoop/hbase/util/Bytes.java
src/main/java/org/apache/hadoop/hbase/util/HMerge.java
src/main/java/org/apache/hadoop/hbase/util/Keying.java
src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java
src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
src/test/java/org/apache/hadoop/hbase/util/TestBytes.java
src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java
src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java
src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java
src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: 0001-HBase-cluster-test-tool.patch, 0002-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch, D549.8.patch, D549.9.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-4908:
-------------------------------
Attachment: D549.4.patch
mbautin updated the revision "[jira] [HBASE-4908] HBase cluster test tool (port from 0.89-fb)".
Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA
Addressed some more of Stack's comments. Got rid of the kill test -- it is a large area by itself and is best left out of the scope of this diff. Refactored the common part of RestartMetaTest and HBaseClusterTest into IntegrationTestTool. I think eventually we need to be able to run integration tests both from command line and through the JUnit framework. Also, we can probably unify the mini-cluster framework with the process-based HBase cluster stuff later, and have a subset of integration tests run in both modes.
I also added two unit tests: TestMiniClusterLoad{Sequential,Parallel}. The former runs a write workload followed by a separate read workload, and the latter one runs a read workload concurrently with the writes (which is noticeably slower -- maybe that's something to investigate).
A thorough round of testing and cleanup is still necessary before this diff can be finalized.
REVISION DETAIL
https://reviews.facebook.net/D549
AFFECTED FILES
pom.xml
src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java
src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java
src/main/java/org/apache/hadoop/hbase/HConstants.java
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
src/main/java/org/apache/hadoop/hbase/util/HMerge.java
src/main/java/org/apache/hadoop/hbase/util/Keying.java
src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
src/test/java/org/apache/hadoop/hbase/util/ClusterTestUtils.java
src/test/java/org/apache/hadoop/hbase/util/HBaseClusterTest.java
src/test/java/org/apache/hadoop/hbase/util/HdfsAppender.java
src/test/java/org/apache/hadoop/hbase/util/IntegrationTestTool.java
src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java
src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java
src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java
src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13162610#comment-13162610 ]
Phabricator commented on HBASE-4908:
------------------------------------
mbautin has commented on the revision "[jira] [HBASE-4908] HBase cluster test tool (port from 0.89-fb)".
@stack: see responses to your comments inline. A new version of the diff will follow.
INLINE COMMENTS
src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:47 Added class documentation.
src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:52 Removed trailing underscores.
src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:275 I added some class comment anyway, and removed kill test and append test from this tool (we can port them over separately).
src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:38 Moved these tool methods to Keying and HBaseTestingUtility and got rid of this class.
src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:53 Got rid of this method entirely and started using the HTable constructor directly.
src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:54 See above (got rid of this method).
src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:59 Moved this to HBaseTestingUtility.
src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:92 Changed to "getMetaRSPort" (because we only need the port number in this use case) and modified to use getRegionLocation.
src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:96 See the above comment.
src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:99 Got rid of this method.
src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:111 Moved to Keying. Not sure where HBaseSplitter is (could not find that type or file).
src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:124 Moved to Keying.
src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:139 Moved to Bytes.
src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:151 Moved to Bytes.
src/test/java/org/apache/hadoop/hbase/manual/utils/HdfsAppender.java:36 Removed HdfsAppender from this diff.
src/test/java/org/apache/hadoop/hbase/manual/utils/KillProcessesAndVerify.java:33 Removed the kill test from this diff.
src/test/java/org/apache/hadoop/hbase/manual/utils/KillProcessesAndVerify.java:119 Removed this entire class from the diff for now.
src/test/java/org/apache/hadoop/hbase/manual/utils/KillProcessesAndVerify.java:152 Removed this entire class (kill test) for now.
src/test/java/org/apache/hadoop/hbase/manual/utils/MultiThreadedAction.java:31 Added.
src/test/java/org/apache/hadoop/hbase/manual/utils/MultiThreadedActionMBean.java:19 Removed this class. Its only purpose was to collect stats from the distributed load test through JMX.
src/test/java/org/apache/hadoop/hbase/manual/utils/ProcessBasedLocalHBaseCluster.java:36 Added a class comment. Unlike MiniHBaseCluster, this starts each daemon as a separate process and allows to do real kill testing.
src/test/java/org/apache/hadoop/hbase/manual/utils/ProcessBasedLocalHBaseCluster.java:256 Refactored this to generate configuration from a map. Does that address your concern, or are you objecting to the whole idea of auto-generating configuration for daemon processes in a process-based local HBase cluster?
REVISION DETAIL
https://reviews.facebook.net/D549
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164922#comment-13164922 ]
Hadoop QA commented on HBASE-4908:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12506552/0003-HBase-cluster-test-tool.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 74 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated -160 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 74 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/471//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/471//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/471//console
This message is automatically generated.
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: 0001-HBase-cluster-test-tool.patch, 0002-HBase-cluster-test-tool.patch, 0003-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch, D549.8.patch, D549.9.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Bautin updated HBASE-4908:
----------------------------------
Status: Patch Available (was: Open)
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: 0001-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163179#comment-13163179 ]
Phabricator commented on HBASE-4908:
------------------------------------
nspiegelberg has commented on the revision "[jira] [HBASE-4908] HBase cluster test tool (port from 0.89-fb)".
INLINE COMMENTS
src/main/java/org/apache/hadoop/hbase/util/Bytes.java:1658-1680 this seems util-specific. this should probably be in some more specialized class rather than Bytes.java
src/main/java/org/apache/hadoop/hbase/util/Keying.java:41-43 all the changes in Keying.java are already found in RegionSplitter.java under HexStringSplit. I would reuse that class
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java:1820-1821 better to use
splits = RegionSplitter.HexStringSplit.split(totalNumberOfRegions)
REVISION DETAIL
https://reviews.facebook.net/D549
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: 0001-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Nicolas Spiegelberg (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nicolas Spiegelberg updated HBASE-4908:
---------------------------------------
Issue Type: Sub-task (was: Test)
Parent: HBASE-4907
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D549.1.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164079#comment-13164079 ]
Phabricator commented on HBASE-4908:
------------------------------------
mbautin has commented on the revision "[jira] [HBASE-4908] HBase cluster test tool (port from 0.89-fb)".
Thanks for reviews, Nicolas and Stack! See responses below. A new version of the code will follow. Still have to re-run the unit tests (I've been having some trouble with those recently -- http://pastebin.com/1G1ZcPeV) and sanity-check the command-line load tester. On a side note, here are some stats from my recent load test run on a 5-node cluster:
11/12/06 18:30:25 INFO util.MultiThreadedAction: [W:21] Keys=17180542, cols=819.4m, time=27:49:20 Overall: [keys/s= 171, latency=116 ms] Current: [keys/s=219, latency=90 ms], insertedUpTo=17180495, insertedQSize=26
11/12/06 18:30:25 INFO util.MultiThreadedAction: [R:10] Keys=250690067, cols=11.8g, time=27:49:20 Overall: [keys/s= 2502, latency=3 ms] Current: [keys/s=261, latency=38 ms], verified=250690067
(The number of writer's threads is reported as 21 because there is an "inserted keys tracker" thread that keeps track of the most recent contiguous key written by all writers.)
INLINE COMMENTS
src/test/java/org/apache/hadoop/hbase/util/IntegrationTestTool.java:129 Changed this to 80. There is no standard way to get terminal width in Java. http://stackoverflow.com/questions/1286461/can-i-find-the-console-width-with-java
src/test/java/org/apache/hadoop/hbase/util/LoadTest.java:40 This is the command-line part of LoadTest, so it should not run as part of the test suite. The same multithreaded writer/verifier code is reused in a couple of "large" unit tests, namely TestMiniClusterLoad{Parallel,Sequential}.
I added a line about the differences between this load tester and PerformanceEvaluation.
Renamed this to LoadTestTool.
src/test/java/org/apache/hadoop/hbase/util/LoadTest.java:318 We first wait for all writers to finish and then wait for all readers to finish, so we should exit when all of those threads stop.
src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java:37 Yes, this can be converted to an IntegrationTest. I think making this a unit test was proposed in the past but the controversy was that it spawns a bunch of child processes and extra care should be taken to shut down all of them. We can leave this as a command-line tool for now and convert to a unit test when it is more stable.
src/main/java/org/apache/hadoop/hbase/util/Bytes.java:1658-1680 Removed these methods and reused the existing RegionSetting.HexStringSplit class.
src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java:28 I moved this to from the test jar to the main jar because of the waitForBaseZNode method that I added to ZKUtil. We need this kind of watcher for quick ZK checks when we don't want to use a callback. Passing a null instead of a watcher does not work.
src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java:176 Not sure what changes you are talking about. I made a few conf options that I needed to use constants in HConstants.
src/main/java/org/apache/hadoop/hbase/util/Keying.java:41-43 Removed this.
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java:1820-1821 Done. Thanks!
src/test/java/org/apache/hadoop/hbase/util/IntegrationTestTool.java:39 Renamed this to AbstractHBaseTool and moved to the main jar, since this is general-purpose command line processing functionality. This also avoids the confusion with the IntegrationTest notation (thanks for the link!) I think we will be creating a lot of integration tests very soon.
REVISION DETAIL
https://reviews.facebook.net/D549
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: 0001-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13165041#comment-13165041 ]
Hudson commented on HBASE-4908:
-------------------------------
Integrated in HBase-TRUNK-security #25 (See [https://builds.apache.org/job/HBase-TRUNK-security/25/])
[jira] [HBASE-4908] HBase cluster test tool (port from 0.89-fb)
Summary:
Porting one of our HBase cluster test tools (a single-process multi-threaded
load generator and verifier) from 0.89-fb to trunk.
I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that
it has some features that I have not tried yet (some kind of a kill test, and
some way to run HBase as multiple processes on one machine).
The main utility of this piece of code for us has been the HBaseClusterTest
command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a
load test in our five-node dev cluster testing, e.g.:
hbase org.apache.hadoop.hbase.util.LoadTestTool -write 50:100:20 -tn loadtest4
-read 100:10 -zk <zk_quorum_node> -bloom ROWCOL -compression LZO -key_window 5
-max_read_errors 10000 -num_keys 10000000000 -start_key 0
Test Plan:
Run this on a dev cluster. Run all unit tests.
Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA
Reviewed By: nspiegelberg
CC: stack, nspiegelberg, mbautin, Karthik
Differential Revision: 549
nspiegelberg :
Files :
* /hbase/trunk/pom.xml
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/AbstractHBaseTool.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/Bytes.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/HMerge.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/Keying.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestBytes.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Fix For: 0.94.0
>
> Attachments: 0001-HBase-cluster-test-tool.patch, 0002-HBase-cluster-test-tool.patch, 0003-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch, D549.8.patch, D549.9.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Nicolas Spiegelberg (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nicolas Spiegelberg updated HBASE-4908:
---------------------------------------
Resolution: Fixed
Fix Version/s: 0.94.0
Status: Resolved (was: Patch Available)
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Fix For: 0.94.0
>
> Attachments: 0001-HBase-cluster-test-tool.patch, 0002-HBase-cluster-test-tool.patch, 0003-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch, D549.8.patch, D549.9.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171336#comment-13171336 ]
stack commented on HBASE-4908:
------------------------------
Backed out SequenceFileLogReader.java change -- will address in a way that doesn't break us going against 0.22 and 0.23 in HBASE-4935
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Fix For: 0.94.0
>
> Attachments: 0001-HBase-cluster-test-tool.patch, 0002-HBase-cluster-test-tool.patch, 0003-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch, D549.8.patch, D549.9.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Bautin updated HBASE-4908:
----------------------------------
Attachment: 0002-HBase-cluster-test-tool.patch
Uploading a patch for Jenkins testing.
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: 0001-HBase-cluster-test-tool.patch, 0002-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch, D549.8.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163401#comment-13163401 ]
Phabricator commented on HBASE-4908:
------------------------------------
Karthik has commented on the revision "[jira] [HBASE-4908] HBase cluster test tool (port from 0.89-fb)".
INLINE COMMENTS
src/test/java/org/apache/hadoop/hbase/util/LoadTest.java:318 One question - when does this integration test exit? Both for the write and read paths?
REVISION DETAIL
https://reviews.facebook.net/D549
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: 0001-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171486#comment-13171486 ]
Hudson commented on HBASE-4908:
-------------------------------
Integrated in HBase-TRUNK-security #35 (See [https://builds.apache.org/job/HBase-TRUNK-security/35/])
HBASE-4935 hbase 0.92.0 doesn't work going against 0.20.205.0, its packaged hadoop -- revert from TRUNK though this change was added as part of HBASE-4908
stack :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Fix For: 0.94.0
>
> Attachments: 0001-HBase-cluster-test-tool.patch, 0002-HBase-cluster-test-tool.patch, 0003-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch, D549.8.patch, D549.9.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-4908:
-------------------------------
Attachment: D549.5.patch
mbautin updated the revision "[jira] [HBASE-4908] HBase cluster test tool (port from 0.89-fb)".
Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA
This passes unit tests (I will confirm this again -- please do not commit yet), and is successfully running a load test in a 5-node cluster.
The reader verifies keys sequentially in the current version, and if it catches up with the reader it sleeps until more keys are written. This is different from the original version of HBaseTest in 0.89-fb (renamed LoadTest in this version), and I think I will modify the reader to read some previous keys that are known to have been written when the "current key" of the reader catches up with the last key written by the writer.
I also made the number of read errors to tolerate configurable (the --max_read_errors option).
I think it will take me a couple more iterations on this diff before it can be committed. Sorry for a lot of updates and emails.
REVISION DETAIL
https://reviews.facebook.net/D549
AFFECTED FILES
pom.xml
src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java
src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java
src/main/java/org/apache/hadoop/hbase/HConstants.java
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
src/main/java/org/apache/hadoop/hbase/util/Bytes.java
src/main/java/org/apache/hadoop/hbase/util/HMerge.java
src/main/java/org/apache/hadoop/hbase/util/Keying.java
src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
src/test/java/org/apache/hadoop/hbase/util/IntegrationTestTool.java
src/test/java/org/apache/hadoop/hbase/util/LoadTest.java
src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
src/test/java/org/apache/hadoop/hbase/util/TestBytes.java
src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java
src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java
src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java
src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13160476#comment-13160476 ]
Phabricator commented on HBASE-4908:
------------------------------------
stack has commented on the revision "[jira] [HBASE-4908] HBase cluster test tool (port from 0.89-fb)".
Kinda close to PE but I suppose different enough. This can give constant update on rates of read and write. Could add that to PE. Also, can do load+read at same time. Would have to do that in different threads in PE. Has more variable load too.
INLINE COMMENTS
src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:17 is this the right package M? util?
src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:47 A bit of class doc on what it is might help.
Might also just point them at usage it seems since that looks like it might be pretty good.
src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:52 No biggie but these data member names with a trailing underscore a bit silly (smile)
src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:58 I suppose its fine this is hard-coded for now.
src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:92 No biggie: check cols is at least 3 in size so we can return 'nicer' message if less than 3
src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:103 This hard coding should be explained somewhere? Maybe in class comment? The 20/1. Or in usage.
src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:204 Yeah, I think these 'features' need to bubble it up to the class comment so folks can see whats in here w/o having to read code.
src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:275 Hmm... maybe this is good enough when it prints out for figuring whats in here?
src/test/java/org/apache/hadoop/hbase/manual/RestartMetaTest.java:36 Class comment
src/test/java/org/apache/hadoop/hbase/manual/RestartMetaTest.java:17 ditto on package. Should this be in under src/test or under src/main? Maybe package should be tool rather than manual?
src/test/java/org/apache/hadoop/hbase/manual/RestartMetaTest.java:152 There is a lot of overlap w/ previous test here, the HBaseClusterTest tool?
src/test/java/org/apache/hadoop/hbase/manual/utils/DataGenerator.java:24 comment on what this does?
src/test/java/org/apache/hadoop/hbase/manual/utils/DisplayFormatUtils.java:24 This class not needed? Use http://hadoop.apache.org/common/docs/r0.20.1/api/org/apache/hadoop/util/StringUtils.html#humanReadableInt(long)
src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:38 Class comment. HBaseUtils is a pretty generic name; says nothing about content.
src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:53 Remove
src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:54 Why not just let this out? Should at least say in javadoc that it can return a null?
src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:59 This method does not look like it belongs in a generic hbaseutils class. It looks like it belongs to the testing tool that makes up the bulk of this patch. Doesn't seem like method name describes properly what it does either. Why not let out exceptions rather than catch and log.
src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:92 HSA is deprecated.
src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:96 Use MetaReader? e.g:
public static Pair<HRegionInfo, ServerName> getRegion(
CatalogTracker catalogTracker, byte [] regionName)
src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:99 Odd name for this method and it also looks a bit dodgy returning an HBC but it only has zk location set into it.
src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:111 This looks like it belongs in Keying class under util? Or in HBaseSplitter?
src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:124 ditto
src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:139 This should be in Bytes (is it not already?)
src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:151 ditto
src/test/java/org/apache/hadoop/hbase/manual/utils/HdfsAppender.java:36 Does this belong in an hbase test class?
src/test/java/org/apache/hadoop/hbase/manual/utils/KillProcessesAndVerify.java:33 class comment.
src/test/java/org/apache/hadoop/hbase/manual/utils/KillProcessesAndVerify.java:119 Let these out rather than just print them?
src/test/java/org/apache/hadoop/hbase/manual/utils/KillProcessesAndVerify.java:152 What keys are being passed here?
src/test/java/org/apache/hadoop/hbase/manual/utils/MultiThreadedAction.java:31 Class comment
src/test/java/org/apache/hadoop/hbase/manual/utils/MultiThreadedReader.java:156 Remove
src/test/java/org/apache/hadoop/hbase/manual/utils/ProcessBasedLocalHBaseCluster.java:36 Class comment. Is this doing what minihbasecluster does?
src/test/java/org/apache/hadoop/hbase/manual/utils/ProcessBasedLocalHBaseCluster.java:256 Not sure this a good idea.
REVISION DETAIL
https://reviews.facebook.net/D549
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D549.1.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-4908:
-------------------------------
Attachment: D549.1.patch
mbautin requested code review of "[jira] [HBASE-4908] HBase cluster test tool (port from 0.89-fb)".
Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA
Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
TEST PLAN
Run this on a dev cluster. Explore various modes of the tool (I've only tried the load test mode before). Probably make a unit test that runs a small local load test.
REVISION DETAIL
https://reviews.facebook.net/D549
AFFECTED FILES
src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java
src/test/java/org/apache/hadoop/hbase/manual/RestartMetaTest.java
src/test/java/org/apache/hadoop/hbase/manual/utils/DataGenerator.java
src/test/java/org/apache/hadoop/hbase/manual/utils/DisplayFormatUtils.java
src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java
src/test/java/org/apache/hadoop/hbase/manual/utils/HdfsAppender.java
src/test/java/org/apache/hadoop/hbase/manual/utils/KillProcessesAndVerify.java
src/test/java/org/apache/hadoop/hbase/manual/utils/MultiThreadedAction.java
src/test/java/org/apache/hadoop/hbase/manual/utils/MultiThreadedActionMBean.java
src/test/java/org/apache/hadoop/hbase/manual/utils/MultiThreadedReader.java
src/test/java/org/apache/hadoop/hbase/manual/utils/MultiThreadedWriter.java
src/test/java/org/apache/hadoop/hbase/manual/utils/ProcessBasedLocalHBaseCluster.java
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/1227/
Tip: use the X-Herald-Rules header to filter Herald messages in your client.
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Test
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D549.1.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164938#comment-13164938 ]
Phabricator commented on HBASE-4908:
------------------------------------
nspiegelberg has accepted the revision "[jira] [HBASE-4908] HBase cluster test tool (port from 0.89-fb)".
lgtm
REVISION DETAIL
https://reviews.facebook.net/D549
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: 0001-HBase-cluster-test-tool.patch, 0002-HBase-cluster-test-tool.patch, 0003-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch, D549.8.patch, D549.9.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Bautin updated HBASE-4908:
----------------------------------
Status: Open (was: Patch Available)
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: 0001-HBase-cluster-test-tool.patch, 0002-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch, D549.8.patch, D549.9.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Bautin updated HBASE-4908:
----------------------------------
Status: Patch Available (was: Open)
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: 0001-HBase-cluster-test-tool.patch, 0002-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch, D549.8.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-4908:
-------------------------------
Attachment: D549.2.patch
mbautin updated the revision "[jira] [HBASE-4908] HBase cluster test tool (port from 0.89-fb)".
Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA
Partially addressing Stack's comments (I will address the rest of them in the next version of this diff). The load test seems to work fine on a dev cluster. I also tried out RestartMetaTest. The idea is to start a local HBase cluster with multiple regionservers on different ports, load some data, kill a regionserver, restart it, and do some checks. I like the of a "process-based local HBase cluster" because it opens up a possibility for a whole new range of unit tests (integration tests?) where we are allowed to kill processes for real, but cannot inspect the internals of those processes. This also allows us to test cluster management scripts (the hbase script, hbase-daemon.sh, etc.) from Java itself. Of course, we need to make sure we don't leave those processes hanging, but that can be addressed in the shutdown hook of the unit test (or @After handler).
RestartMetaTest still needs some work to make it robust and usable, and we probably want to make it a unit test.
REVISION DETAIL
https://reviews.facebook.net/D549
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/HConstants.java
src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
src/main/java/org/apache/hadoop/hbase/util/HMerge.java
src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
src/test/java/org/apache/hadoop/hbase/util/ClusterTestUtils.java
src/test/java/org/apache/hadoop/hbase/util/DataGenerator.java
src/test/java/org/apache/hadoop/hbase/util/HBaseClusterTest.java
src/test/java/org/apache/hadoop/hbase/util/HdfsAppender.java
src/test/java/org/apache/hadoop/hbase/util/KillProcessesAndVerify.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D549.1.patch, D549.2.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180717#comment-13180717 ]
Zhihong Yu commented on HBASE-4908:
-----------------------------------
We have OPT_ZK_QUORUM.
I think adding a config parameter for zookeeper client port would make using this tool more easily.
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Fix For: 0.94.0
>
> Attachments: 0001-HBase-cluster-test-tool.patch, 0002-HBase-cluster-test-tool.patch, 0003-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch, D549.8.patch, D549.9.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164926#comment-13164926 ]
Phabricator commented on HBASE-4908:
------------------------------------
mbautin has commented on the revision "[jira] [HBASE-4908] HBase cluster test tool (port from 0.89-fb)".
Some more unit tests results: 1189 test methods, 252 test classes succeeded. TestLogRolling failed when running on the cluster using mrunit, but succeeded locally.
mvn -Dtest=TestLogRolling#testLogRollOnDatanodeDeath -P localTests test
This patch should be OK to commit.
REVISION DETAIL
https://reviews.facebook.net/D549
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: 0001-HBase-cluster-test-tool.patch, 0002-HBase-cluster-test-tool.patch, 0003-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch, D549.8.patch, D549.9.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163201#comment-13163201 ]
Hadoop QA commented on HBASE-4908:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12506189/0001-HBase-cluster-test-tool.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 77 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated -160 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 72 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.util.TestRegionSplitter
org.apache.hadoop.hbase.client.TestMultiParallel
org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler
org.apache.hadoop.hbase.regionserver.wal.TestHLogBench
org.apache.hadoop.hbase.rest.TestGzipFilter
org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD
org.apache.hadoop.hbase.regionserver.TestAtomicOperation
org.apache.hadoop.hbase.rest.TestScannersWithFilters
org.apache.hadoop.hbase.TestInfoServers
org.apache.hadoop.hbase.regionserver.TestParallelPut
org.apache.hadoop.hbase.coprocessor.TestClassLoading
org.apache.hadoop.hbase.filter.TestColumnRangeFilter
org.apache.hadoop.hbase.client.TestHCM
org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary
org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildHole
org.apache.hadoop.hbase.coprocessor.TestMasterObserver
org.apache.hadoop.hbase.rest.TestStatusResource
org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort
org.apache.hadoop.hbase.rest.TestVersionResource
org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed
org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol
org.apache.hadoop.hbase.rest.TestRowResource
org.apache.hadoop.hbase.rest.TestScannerResource
org.apache.hadoop.hbase.ipc.TestDelayedRpc
org.apache.hadoop.hbase.rest.client.TestRemoteAdmin
org.apache.hadoop.hbase.util.TestFSUtils
org.apache.hadoop.hbase.rest.TestTableResource
org.apache.hadoop.hbase.regionserver.wal.TestWALReplay
org.apache.hadoop.hbase.util.TestIdLock
org.apache.hadoop.hbase.rest.TestTransform
org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint
org.apache.hadoop.hbase.regionserver.TestHRegion
org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort
org.apache.hadoop.hbase.catalog.TestMetaReaderEditor
org.apache.hadoop.hbase.client.TestMetaScanner
org.apache.hadoop.hbase.io.hfile.TestHFileBlock
org.apache.hadoop.hbase.client.TestTimestampsFilter
org.apache.hadoop.hbase.master.TestMaster
org.apache.hadoop.hbase.TestDrainingServer
org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass
org.apache.hadoop.hbase.rest.TestSchemaResource
org.apache.hadoop.hbase.TestAcidGuarantees
org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion
org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildBase
org.apache.hadoop.hbase.avro.TestAvroServer
org.apache.hadoop.hbase.rest.client.TestRemoteTable
org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol
org.apache.hadoop.hbase.util.TestHBaseFsck
org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove
org.apache.hadoop.hbase.client.TestHTableUtil
org.apache.hadoop.hbase.thrift.TestThriftServer
org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface
org.apache.hadoop.hbase.regionserver.TestRegionServerMetrics
org.apache.hadoop.hbase.util.TestMergeTable
org.apache.hadoop.hbase.mapreduce.TestImportTsv
org.apache.hadoop.hbase.rest.TestMultiRowResource
org.apache.hadoop.hbase.TestMultiVersions
org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildOverlap
org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove
org.apache.hadoop.hbase.coprocessor.TestWALObserver
org.apache.hadoop.hbase.master.TestOpenedRegionHandler
org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/450//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/450//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/450//console
This message is automatically generated.
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: 0001-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164950#comment-13164950 ]
Phabricator commented on HBASE-4908:
------------------------------------
mbautin has committed the revision "[jira] [HBASE-4908] HBase cluster test tool (port from 0.89-fb)".
REVISION DETAIL
https://reviews.facebook.net/D549
COMMIT
https://reviews.facebook.net/rHBASE1211746
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Fix For: 0.94.0
>
> Attachments: 0001-HBase-cluster-test-tool.patch, 0002-HBase-cluster-test-tool.patch, 0003-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch, D549.8.patch, D549.9.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Bautin updated HBASE-4908:
----------------------------------
Status: Patch Available (was: Open)
One more round of Jenkins testing for the patch.
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: 0001-HBase-cluster-test-tool.patch, 0002-HBase-cluster-test-tool.patch, 0003-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch, D549.8.patch, D549.9.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Bautin updated HBASE-4908:
----------------------------------
Attachment: 0003-HBase-cluster-test-tool.patch
Attaching the most recent patch.
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: 0001-HBase-cluster-test-tool.patch, 0002-HBase-cluster-test-tool.patch, 0003-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch, D549.8.patch, D549.9.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Bautin updated HBASE-4908:
----------------------------------
Status: Open (was: Patch Available)
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: 0001-HBase-cluster-test-tool.patch, 0002-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch, D549.8.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13165091#comment-13165091 ]
Hudson commented on HBASE-4908:
-------------------------------
Integrated in HBase-TRUNK #2526 (See [https://builds.apache.org/job/HBase-TRUNK/2526/])
[jira] [HBASE-4908] HBase cluster test tool (port from 0.89-fb)
Summary:
Porting one of our HBase cluster test tools (a single-process multi-threaded
load generator and verifier) from 0.89-fb to trunk.
I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that
it has some features that I have not tried yet (some kind of a kill test, and
some way to run HBase as multiple processes on one machine).
The main utility of this piece of code for us has been the HBaseClusterTest
command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a
load test in our five-node dev cluster testing, e.g.:
hbase org.apache.hadoop.hbase.util.LoadTestTool -write 50:100:20 -tn loadtest4
-read 100:10 -zk <zk_quorum_node> -bloom ROWCOL -compression LZO -key_window 5
-max_read_errors 10000 -num_keys 10000000000 -start_key 0
Test Plan:
Run this on a dev cluster. Run all unit tests.
Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA
Reviewed By: nspiegelberg
CC: stack, nspiegelberg, mbautin, Karthik
Differential Revision: 549
nspiegelberg :
Files :
* /hbase/trunk/pom.xml
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/AbstractHBaseTool.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/Bytes.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/HMerge.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/Keying.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestBytes.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Fix For: 0.94.0
>
> Attachments: 0001-HBase-cluster-test-tool.patch, 0002-HBase-cluster-test-tool.patch, 0003-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch, D549.8.patch, D549.9.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-4908:
-------------------------------
Attachment: D549.7.patch
mbautin updated the revision "[jira] [HBASE-4908] HBase cluster test tool (port from 0.89-fb)".
Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA
Rebased on changes from trunk.
REVISION DETAIL
https://reviews.facebook.net/D549
AFFECTED FILES
pom.xml
src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java
src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java
src/main/java/org/apache/hadoop/hbase/HConstants.java
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
src/main/java/org/apache/hadoop/hbase/util/Bytes.java
src/main/java/org/apache/hadoop/hbase/util/HMerge.java
src/main/java/org/apache/hadoop/hbase/util/Keying.java
src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
src/test/java/org/apache/hadoop/hbase/util/IntegrationTestTool.java
src/test/java/org/apache/hadoop/hbase/util/LoadTest.java
src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
src/test/java/org/apache/hadoop/hbase/util/TestBytes.java
src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java
src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java
src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java
src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164087#comment-13164087 ]
Hadoop QA commented on HBASE-4908:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12506381/D549.8.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 52 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/455//console
This message is automatically generated.
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: 0001-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch, D549.8.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-4908:
-------------------------------
Attachment: D549.3.patch
mbautin updated the revision "[jira] [HBASE-4908] HBase cluster test tool (port from 0.89-fb)".
Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA
Debugged and significantly improved RestartMetaTest and its dependency ProcessBasedLocalHBaseCluster to the point of actually being able to run a local multi-process test of an HBase cluster that kills the region server with the .META. region and splits the logs. Discovered that our log splitting does not work correctly with RawLocalFileSystem or checksum-enabled local filesystem, and added a fix to the hack in SequenceFileLogReader to only apply reflection logic to DFSInputStream.
Still need to clean up the code, re-run unit tests, and re-test HBaseClusterTest on a live cluster.
REVISION DETAIL
https://reviews.facebook.net/D549
AFFECTED FILES
pom.xml
src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java
src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java
src/main/java/org/apache/hadoop/hbase/HConstants.java
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
src/main/java/org/apache/hadoop/hbase/util/HMerge.java
src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
src/test/java/org/apache/hadoop/hbase/util/ClusterTestUtils.java
src/test/java/org/apache/hadoop/hbase/util/DataGenerator.java
src/test/java/org/apache/hadoop/hbase/util/HBaseClusterTest.java
src/test/java/org/apache/hadoop/hbase/util/HdfsAppender.java
src/test/java/org/apache/hadoop/hbase/util/KillProcessesAndVerify.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D549.1.patch, D549.2.patch, D549.3.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-4908:
-------------------------------
Attachment: D549.8.patch
mbautin updated the revision "[jira] [HBASE-4908] HBase cluster test tool (port from 0.89-fb)".
Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA
Addressing Stack's and Nicolas's comments. Still need to re-run the unit tests.
REVISION DETAIL
https://reviews.facebook.net/D549
AFFECTED FILES
pom.xml
src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java
src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java
src/main/java/org/apache/hadoop/hbase/HConstants.java
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
src/main/java/org/apache/hadoop/hbase/util/AbstractHBaseTool.java
src/main/java/org/apache/hadoop/hbase/util/Bytes.java
src/main/java/org/apache/hadoop/hbase/util/HMerge.java
src/main/java/org/apache/hadoop/hbase/util/Keying.java
src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java
src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
src/test/java/org/apache/hadoop/hbase/util/TestBytes.java
src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java
src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java
src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java
src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: 0001-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch, D549.8.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Bautin updated HBASE-4908:
----------------------------------
Status: Patch Available (was: Open)
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: 0001-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163185#comment-13163185 ]
Phabricator commented on HBASE-4908:
------------------------------------
nspiegelberg has commented on the revision "[jira] [HBASE-4908] HBase cluster test tool (port from 0.89-fb)".
INLINE COMMENTS
src/test/java/org/apache/hadoop/hbase/util/IntegrationTestTool.java:129 isn't the standard width 80? Is there a system setting we can use?
REVISION DETAIL
https://reviews.facebook.net/D549
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: 0001-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Bautin updated HBASE-4908:
----------------------------------
Attachment: 0001-HBase-cluster-test-tool.patch
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: 0001-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163946#comment-13163946 ]
Phabricator commented on HBASE-4908:
------------------------------------
stack has commented on the revision "[jira] [HBASE-4908] HBase cluster test tool (port from 0.89-fb)".
This is looking great Mikhail.
INLINE COMMENTS
src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java:28 When would I want one of these?
src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java:176 Is this from another issue Mikhail? (No matter if it is... we can take care of it on commit).
src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java:37 Thanks for doing this.
src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java:111 Thanks for fixing this.
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java:1798 Good
src/test/java/org/apache/hadoop/hbase/util/IntegrationTestTool.java:39 Will this be run as an IntegrationTest because it has the IntegrationTest prefix; (see the Integration Test section on this page, http://hbase.apache.org/book/hbase.tests.html, if you don't have cluse what I'm on about)?
I like the idea of this class. We need it. Nice how you subclass tool.
src/test/java/org/apache/hadoop/hbase/util/LoadTest.java:40 Should this class have the IntegrationTest prefix or you think this is just a tool not part of IntegrationTests?
Since its sitting beside PerformanceEvaluation tool, should you say something on how it differs from it?
src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java:37 Should this be an IntegrationTest? (We can do the convertion in another issue).
REVISION DETAIL
https://reviews.facebook.net/D549
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: 0001-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-4908:
-------------------------------
Attachment: D549.6.patch
mbautin updated the revision "[jira] [HBASE-4908] HBase cluster test tool (port from 0.89-fb)".
Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA
The unit tests run fine (except TestReplication.queueFailover, which is flaky), and cluster testing works well. I rewrote the writer/reader synchronization mechanism after talking to Kannan, so that the reader gets the precise knowledge of what is the last key in the contiguous sequence of keys starting from startKey that the writer has written. This required implementing a producer/consumer pattern with a special InsertedKeyTracker thread sorting inserted keys and advancing the "last contiguous key" marker. Also clarified and improved a bunch of command-line stuff in LoadTest.
REVISION DETAIL
https://reviews.facebook.net/D549
AFFECTED FILES
pom.xml
src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java
src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java
src/main/java/org/apache/hadoop/hbase/HConstants.java
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
src/main/java/org/apache/hadoop/hbase/util/Bytes.java
src/main/java/org/apache/hadoop/hbase/util/HMerge.java
src/main/java/org/apache/hadoop/hbase/util/Keying.java
src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
src/test/java/org/apache/hadoop/hbase/util/IntegrationTestTool.java
src/test/java/org/apache/hadoop/hbase/util/LoadTest.java
src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
src/test/java/org/apache/hadoop/hbase/util/TestBytes.java
src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java
src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java
src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java
src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164120#comment-13164120 ]
Hadoop QA commented on HBASE-4908:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12506382/0002-HBase-cluster-test-tool.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 74 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated -160 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 74 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildHole
org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildOverlap
org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildBase
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/456//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/456//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/456//console
This message is automatically generated.
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: 0001-HBase-cluster-test-tool.patch, 0002-HBase-cluster-test-tool.patch, D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, D549.5.patch, D549.6.patch, D549.7.patch, D549.8.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4908) HBase cluster test tool (port from
0.89-fb)
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13160490#comment-13160490 ]
Phabricator commented on HBASE-4908:
------------------------------------
nspiegelberg has commented on the revision "[jira] [HBASE-4908] HBase cluster test tool (port from 0.89-fb)".
@stack : We should have a distinction:
PerformanceEvaluation - meant to get an idea of the performance of your system setup. Able to find performance regressions or understand theoretical performance gains of datapath changes
Load Tester - meant to provide deterministic, long-running load on a system. Allows for large scale verification to catch race conditions resulting from logic error or process failure (in a distributed system setting).
REVISION DETAIL
https://reviews.facebook.net/D549
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D549.1.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira