You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Mikhail Bautin (Created) (JIRA)" <ji...@apache.org> on 2012/03/31 00:13:25 UTC
[jira] [Created] (HBASE-5684) Make ProcessBasedLocalHBaseCluster
run HDFS and make it more robust
Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
-------------------------------------------------------------------
Key: HBASE-5684
URL: https://issues.apache.org/jira/browse/HBASE-5684
Project: HBase
Issue Type: Improvement
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5684) Make ProcessBasedLocalHBaseCluster
run HDFS and make it more robust
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253056#comment-13253056 ]
Phabricator commented on HBASE-5684:
------------------------------------
mbautin has commented on the revision "[jira] [HBASE-5684] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".
INLINE COMMENTS
src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java:93 This is actually a stability fix. We have seen the load balancer shedding regions from most loaded regionservers in some tests.
REVISION DETAIL
https://reviews.facebook.net/D2757
BRANCH
make_processbasedlocalhbasecluster_run_HBASE-5684_v11
> Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
> -------------------------------------------------------------------
>
> Key: HBASE-5684
> URL: https://issues.apache.org/jira/browse/HBASE-5684
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch, D2709.4.patch, D2757.1.patch
>
>
> Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5684) Make ProcessBasedLocalHBaseCluster
run HDFS and make it more robust
Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Bautin updated HBASE-5684:
----------------------------------
Attachment: jira-HBASE-5684-Make-ProcessBasedLocalHBaseCluster-r-2012-04-12_20_42_02.patch
> Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
> -------------------------------------------------------------------
>
> Key: HBASE-5684
> URL: https://issues.apache.org/jira/browse/HBASE-5684
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch, D2709.4.patch, D2757.1.patch, D2757.2.patch, jira-HBASE-5684-Make-ProcessBasedLocalHBaseCluster-r-2012-04-12_20_42_02.patch
>
>
> Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5684) Make ProcessBasedLocalHBaseCluster
run HDFS and make it more robust
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13251860#comment-13251860 ]
Phabricator commented on HBASE-5684:
------------------------------------
Liyin has commented on the revision "[jira] [HBASE-5684] [89-fb] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".
Mikhail, nice work:) LGTM !
REVISION DETAIL
https://reviews.facebook.net/D2709
BRANCH
make_processbasedlocalhbasecluster_run_HBASE-5684_v10
> Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
> -------------------------------------------------------------------
>
> Key: HBASE-5684
> URL: https://issues.apache.org/jira/browse/HBASE-5684
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch, D2709.4.patch
>
>
> Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5684) Make ProcessBasedLocalHBaseCluster
run HDFS and make it more robust
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5684:
-------------------------------
Attachment: D2709.1.patch
mbautin requested code review of "[jira] [HBASE-5684] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".
Reviewers: Kannan, Liyin, khemani, Karthik, madhuvaidya, amirshim, JIRA
Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
Also all logs of local HBase cluster daemons are now tailed to the primary log of the unit test with the appropriate prefixes to make debugging easier.
TEST PLAN
Run the new unit test multiple times (10x or 50x).
Run all unit tests.
REVISION DETAIL
https://reviews.facebook.net/D2709
AFFECTED FILES
bin/hbase-config.sh
bin/test/kill_process_based_cluster.sh
src/main/java/org/apache/hadoop/hbase/HConstants.java
src/main/java/org/apache/hadoop/hbase/master/HMaster.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
src/test/java/org/apache/hadoop/hbase/HTestConst.java
src/test/java/org/apache/hadoop/hbase/TestInfoServers.java
src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
src/test/java/org/apache/hadoop/hbase/master/TestZKBasedCloseRegion.java
src/test/java/org/apache/hadoop/hbase/master/TestZKBasedReopenRegion.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java
src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java
src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
src/test/java/org/apache/hadoop/hbase/util/TestHBaseHomePath.java
src/test/java/org/apache/hadoop/hbase/util/TestProcessBasedCluster.java
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/6195/
Tip: use the X-Herald-Rules header to filter Herald messages in your client.
> Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
> -------------------------------------------------------------------
>
> Key: HBASE-5684
> URL: https://issues.apache.org/jira/browse/HBASE-5684
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D2709.1.patch
>
>
> Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5684) Make ProcessBasedLocalHBaseCluster
run HDFS and make it more robust
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5684:
-------------------------------
Attachment: D2757.4.patch
mbautin updated the revision "[jira] [HBASE-5684] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".
Reviewers: tedyu, stack, lhofhansl, nspiegelberg, amirshim, JIRA
Fixing a couple more minor things.
REVISION DETAIL
https://reviews.facebook.net/D2757
AFFECTED FILES
bin/test/process_based_cluster.sh
src/main/java/org/apache/hadoop/hbase/HConstants.java
src/main/java/org/apache/hadoop/hbase/master/HMaster.java
src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
src/test/java/org/apache/hadoop/hbase/HTestConst.java
src/test/java/org/apache/hadoop/hbase/TestInfoServers.java
src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java
src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java
src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
src/test/java/org/apache/hadoop/hbase/util/TestProcessBasedCluster.java
> Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
> -------------------------------------------------------------------
>
> Key: HBASE-5684
> URL: https://issues.apache.org/jira/browse/HBASE-5684
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch, D2709.4.patch, D2757.1.patch, D2757.2.patch, D2757.3.patch, D2757.4.patch, jira-HBASE-5684-Make-ProcessBasedLocalHBaseCluster-r-2012-04-12_20_42_02.patch
>
>
> Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5684) Make ProcessBasedLocalHBaseCluster
run HDFS and make it more robust
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13251157#comment-13251157 ]
Phabricator commented on HBASE-5684:
------------------------------------
tedyu has accepted the revision "[jira] [HBASE-5684] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".
I still see 'This diff has Unit Test Problems.'
REVISION DETAIL
https://reviews.facebook.net/D2709
BRANCH
make_processbasedlocalhbasecluster_run_HBASE-5684_v9
> Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
> -------------------------------------------------------------------
>
> Key: HBASE-5684
> URL: https://issues.apache.org/jira/browse/HBASE-5684
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch
>
>
> Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5684) Make ProcessBasedLocalHBaseCluster
run HDFS and make it more robust
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5684:
-------------------------------
Attachment: D2757.1.patch
mbautin requested code review of "[jira] [HBASE-5684] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".
Reviewers: tedyu, stack, lhofhansl, nspiegelberg, amirshim, JIRA
Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
Also all logs of local HBase cluster daemons are now tailed to the primary log of the unit test with the appropriate prefixes to make debugging easier.
This is a trunk diff. The 89-fb version is D2709.
TEST PLAN
Run the new unit test multiple times (10x or 50x).
Run all unit tests.
REVISION DETAIL
https://reviews.facebook.net/D2757
AFFECTED FILES
bin/test/process_based_cluster.sh
src/main/java/org/apache/hadoop/hbase/HConstants.java
src/main/java/org/apache/hadoop/hbase/master/HMaster.java
src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
src/test/java/org/apache/hadoop/hbase/HTestConst.java
src/test/java/org/apache/hadoop/hbase/TestInfoServers.java
src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java
src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java
src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
src/test/java/org/apache/hadoop/hbase/util/TestProcessBasedCluster.java
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/6285/
Tip: use the X-Herald-Rules header to filter Herald messages in your client.
> Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
> -------------------------------------------------------------------
>
> Key: HBASE-5684
> URL: https://issues.apache.org/jira/browse/HBASE-5684
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch, D2709.4.patch, D2757.1.patch
>
>
> Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5684) Make ProcessBasedLocalHBaseCluster
run HDFS and make it more robust
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13251162#comment-13251162 ]
Zhihong Yu commented on HBASE-5684:
-----------------------------------
bq. do you mean "Unit Tests Skipped"
Yes.
> Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
> -------------------------------------------------------------------
>
> Key: HBASE-5684
> URL: https://issues.apache.org/jira/browse/HBASE-5684
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch
>
>
> Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5684) Make ProcessBasedLocalHBaseCluster
run HDFS and make it more robust
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253944#comment-13253944 ]
Hudson commented on HBASE-5684:
-------------------------------
Integrated in HBase-TRUNK #2759 (See [https://builds.apache.org/job/HBase-TRUNK/2759/])
[jira] [HBASE-5684] Make ProcessBasedLocalHBaseCluster run HDFS and make it more
robust
Summary:
Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We
need it to start a process-based HDFS cluster as well. We also need to make the
whole thing more stable so we can use it in unit tests.
Also all logs of local HBase cluster daemons are now tailed to the primary log
of the unit test with the appropriate prefixes to make debugging easier.
This is a trunk diff. The 89-fb version is D2709.
Test Plan:
Run the new unit test multiple times (10x or 50x).
Run all unit tests.
Reviewers: tedyu, stack, lhofhansl, nspiegelberg, amirshim, JIRA
Reviewed By: tedyu
Differential Revision: https://reviews.facebook.net/D2757 (Revision 1326036)
Result = FAILURE
mbautin :
Files :
* /hbase/trunk/bin/test
* /hbase/trunk/bin/test/process_based_cluster.sh
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HTestConst.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestInfoServers.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestProcessBasedCluster.java
> Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
> -------------------------------------------------------------------
>
> Key: HBASE-5684
> URL: https://issues.apache.org/jira/browse/HBASE-5684
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch, D2709.4.patch, D2757.1.patch, D2757.2.patch, D2757.3.patch, D2757.4.patch, jira-HBASE-5684-Make-ProcessBasedLocalHBaseCluster-r-2012-04-12_20_42_02.patch
>
>
> Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5684) Make ProcessBasedLocalHBaseCluster
run HDFS and make it more robust
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250415#comment-13250415 ]
Phabricator commented on HBASE-5684:
------------------------------------
tedyu has commented on the revision "[jira] [HBASE-5684] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".
Patch looks good.
INLINE COMMENTS
src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java:44 Should read 'to look up'
src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java:114 Why is sorting needed ?
src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java:250 I think removing 'to kill' would make the sentence clearer.
src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java:395 Should we add mapping for 'fs.defaultFS' as well ?
REVISION DETAIL
https://reviews.facebook.net/D2709
> Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
> -------------------------------------------------------------------
>
> Key: HBASE-5684
> URL: https://issues.apache.org/jira/browse/HBASE-5684
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D2709.1.patch
>
>
> Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5684) Make ProcessBasedLocalHBaseCluster
run HDFS and make it more robust
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253999#comment-13253999 ]
Hudson commented on HBASE-5684:
-------------------------------
Integrated in HBase-TRUNK-security #171 (See [https://builds.apache.org/job/HBase-TRUNK-security/171/])
[jira] [HBASE-5684] Make ProcessBasedLocalHBaseCluster run HDFS and make it more
robust
Summary:
Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We
need it to start a process-based HDFS cluster as well. We also need to make the
whole thing more stable so we can use it in unit tests.
Also all logs of local HBase cluster daemons are now tailed to the primary log
of the unit test with the appropriate prefixes to make debugging easier.
This is a trunk diff. The 89-fb version is D2709.
Test Plan:
Run the new unit test multiple times (10x or 50x).
Run all unit tests.
Reviewers: tedyu, stack, lhofhansl, nspiegelberg, amirshim, JIRA
Reviewed By: tedyu
Differential Revision: https://reviews.facebook.net/D2757 (Revision 1326036)
Result = FAILURE
mbautin :
Files :
* /hbase/trunk/bin/test
* /hbase/trunk/bin/test/process_based_cluster.sh
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HTestConst.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestInfoServers.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestProcessBasedCluster.java
> Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
> -------------------------------------------------------------------
>
> Key: HBASE-5684
> URL: https://issues.apache.org/jira/browse/HBASE-5684
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch, D2709.4.patch, D2757.1.patch, D2757.2.patch, D2757.3.patch, D2757.4.patch, jira-HBASE-5684-Make-ProcessBasedLocalHBaseCluster-r-2012-04-12_20_42_02.patch
>
>
> Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5684) Make ProcessBasedLocalHBaseCluster
run HDFS and make it more robust
Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Bautin updated HBASE-5684:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to trunk.
> Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
> -------------------------------------------------------------------
>
> Key: HBASE-5684
> URL: https://issues.apache.org/jira/browse/HBASE-5684
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch, D2709.4.patch, D2757.1.patch, D2757.2.patch, D2757.3.patch, D2757.4.patch, jira-HBASE-5684-Make-ProcessBasedLocalHBaseCluster-r-2012-04-12_20_42_02.patch
>
>
> Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5684) Make ProcessBasedLocalHBaseCluster
run HDFS and make it more robust
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13251160#comment-13251160 ]
Phabricator commented on HBASE-5684:
------------------------------------
mbautin has commented on the revision "[jira] [HBASE-5684] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".
Ted: do you mean "Unit Tests Skipped" at the top of the diff? We currently don't have integration between HBase unit tests and Arcanist/Phabricator.
REVISION DETAIL
https://reviews.facebook.net/D2709
BRANCH
make_processbasedlocalhbasecluster_run_HBASE-5684_v9
> Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
> -------------------------------------------------------------------
>
> Key: HBASE-5684
> URL: https://issues.apache.org/jira/browse/HBASE-5684
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch
>
>
> Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5684) Make ProcessBasedLocalHBaseCluster
run HDFS and make it more robust
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5684:
-------------------------------
Attachment: D2757.3.patch
mbautin updated the revision "[jira] [HBASE-5684] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".
Reviewers: tedyu, stack, lhofhansl, nspiegelberg, amirshim, JIRA
Rebasing on recent changes in trunk.
REVISION DETAIL
https://reviews.facebook.net/D2757
AFFECTED FILES
bin/test/process_based_cluster.sh
src/main/java/org/apache/hadoop/hbase/HConstants.java
src/main/java/org/apache/hadoop/hbase/master/HMaster.java
src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
src/test/java/org/apache/hadoop/hbase/HTestConst.java
src/test/java/org/apache/hadoop/hbase/TestInfoServers.java
src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java
src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java
src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
src/test/java/org/apache/hadoop/hbase/util/TestProcessBasedCluster.java
> Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
> -------------------------------------------------------------------
>
> Key: HBASE-5684
> URL: https://issues.apache.org/jira/browse/HBASE-5684
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch, D2709.4.patch, D2757.1.patch, D2757.2.patch, D2757.3.patch, jira-HBASE-5684-Make-ProcessBasedLocalHBaseCluster-r-2012-04-12_20_42_02.patch
>
>
> Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5684) Make ProcessBasedLocalHBaseCluster
run HDFS and make it more robust
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253105#comment-13253105 ]
Hadoop QA commented on HBASE-5684:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12522525/jira-HBASE-5684-Make-ProcessBasedLocalHBaseCluster-r-2012-04-12_20_42_02.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 40 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplication
org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit
org.apache.hadoop.hbase.replication.TestMultiSlaveReplication
org.apache.hadoop.hbase.regionserver.wal.TestHLog
org.apache.hadoop.hbase.replication.TestMasterReplication
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1506//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1506//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1506//console
This message is automatically generated.
> Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
> -------------------------------------------------------------------
>
> Key: HBASE-5684
> URL: https://issues.apache.org/jira/browse/HBASE-5684
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch, D2709.4.patch, D2757.1.patch, D2757.2.patch, jira-HBASE-5684-Make-ProcessBasedLocalHBaseCluster-r-2012-04-12_20_42_02.patch
>
>
> Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5684) Make ProcessBasedLocalHBaseCluster
run HDFS and make it more robust
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253926#comment-13253926 ]
Phabricator commented on HBASE-5684:
------------------------------------
mbautin has committed the revision "[jira] [HBASE-5684] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".
REVISION DETAIL
https://reviews.facebook.net/D2757
COMMIT
https://reviews.facebook.net/rHBASE1326036
> Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
> -------------------------------------------------------------------
>
> Key: HBASE-5684
> URL: https://issues.apache.org/jira/browse/HBASE-5684
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch, D2709.4.patch, D2757.1.patch, D2757.2.patch, D2757.3.patch, D2757.4.patch, jira-HBASE-5684-Make-ProcessBasedLocalHBaseCluster-r-2012-04-12_20_42_02.patch
>
>
> Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5684) Make ProcessBasedLocalHBaseCluster
run HDFS and make it more robust
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252152#comment-13252152 ]
Phabricator commented on HBASE-5684:
------------------------------------
tedyu has accepted the revision "[jira] [HBASE-5684] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".
INLINE COMMENTS
src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java:93 Is this restriction needed ?
REVISION DETAIL
https://reviews.facebook.net/D2757
BRANCH
make_processbasedlocalhbasecluster_run_HBASE-5684_v11
> Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
> -------------------------------------------------------------------
>
> Key: HBASE-5684
> URL: https://issues.apache.org/jira/browse/HBASE-5684
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch, D2709.4.patch, D2757.1.patch
>
>
> Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5684) Make ProcessBasedLocalHBaseCluster
run HDFS and make it more robust
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250888#comment-13250888 ]
Phabricator commented on HBASE-5684:
------------------------------------
mbautin has commented on the revision "[jira] [HBASE-5684] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".
Addressing Ted's comments. I will post a new version shortly.
INLINE COMMENTS
src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java:114 Even though ports for same-type servers in the process-based cluster are selected randomly, it would still make sense to call the lowest-port server #0, the next-lowest #1, etc.
src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java:250 Done.
src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java:395 Done.
src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java:44 Done.
REVISION DETAIL
https://reviews.facebook.net/D2709
> Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
> -------------------------------------------------------------------
>
> Key: HBASE-5684
> URL: https://issues.apache.org/jira/browse/HBASE-5684
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D2709.1.patch
>
>
> Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5684) Make ProcessBasedLocalHBaseCluster
run HDFS and make it more robust
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252109#comment-13252109 ]
Hadoop QA commented on HBASE-5684:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12522355/D2757.1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 30 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1489//console
This message is automatically generated.
> Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
> -------------------------------------------------------------------
>
> Key: HBASE-5684
> URL: https://issues.apache.org/jira/browse/HBASE-5684
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch, D2709.4.patch, D2757.1.patch
>
>
> Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5684) Make ProcessBasedLocalHBaseCluster
run HDFS and make it more robust
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5684:
-------------------------------
Attachment: D2757.2.patch
mbautin updated the revision "[jira] [HBASE-5684] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".
Reviewers: tedyu, stack, lhofhansl, nspiegelberg, amirshim, JIRA
Setting ZooKeeper data directory and adding test category.
REVISION DETAIL
https://reviews.facebook.net/D2757
AFFECTED FILES
bin/test/process_based_cluster.sh
src/main/java/org/apache/hadoop/hbase/HConstants.java
src/main/java/org/apache/hadoop/hbase/master/HMaster.java
src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
src/test/java/org/apache/hadoop/hbase/HTestConst.java
src/test/java/org/apache/hadoop/hbase/TestInfoServers.java
src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java
src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java
src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
src/test/java/org/apache/hadoop/hbase/util/TestProcessBasedCluster.java
> Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
> -------------------------------------------------------------------
>
> Key: HBASE-5684
> URL: https://issues.apache.org/jira/browse/HBASE-5684
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch, D2709.4.patch, D2757.1.patch, D2757.2.patch
>
>
> Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5684) Make ProcessBasedLocalHBaseCluster
run HDFS and make it more robust
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253902#comment-13253902 ]
Hadoop QA commented on HBASE-5684:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12522636/D2757.3.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 29 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplication
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1519//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1519//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1519//console
This message is automatically generated.
> Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
> -------------------------------------------------------------------
>
> Key: HBASE-5684
> URL: https://issues.apache.org/jira/browse/HBASE-5684
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch, D2709.4.patch, D2757.1.patch, D2757.2.patch, D2757.3.patch, jira-HBASE-5684-Make-ProcessBasedLocalHBaseCluster-r-2012-04-12_20_42_02.patch
>
>
> Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5684) Make ProcessBasedLocalHBaseCluster
run HDFS and make it more robust
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13251867#comment-13251867 ]
Phabricator commented on HBASE-5684:
------------------------------------
amirshim has commented on the revision "[jira] [HBASE-5684] [89-fb] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".
Looks good.
As we talked, maybe we should use [[http://docs.oracle.com/javase/7/docs/api/java/nio/file/WatchService.html | WatchService]] for log tailer once we decide to move to Java 7.
And we might want to split HBaseTestingUtility.java at some point since it's over 1500 lines.
REVISION DETAIL
https://reviews.facebook.net/D2709
BRANCH
make_processbasedlocalhbasecluster_run_HBASE-5684_v10
> Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
> -------------------------------------------------------------------
>
> Key: HBASE-5684
> URL: https://issues.apache.org/jira/browse/HBASE-5684
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch, D2709.4.patch
>
>
> Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5684) Make ProcessBasedLocalHBaseCluster
run HDFS and make it more robust
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5684:
-------------------------------
Attachment: D2709.2.patch
mbautin updated the revision "[jira] [HBASE-5684] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".
Reviewers: Kannan, Liyin, khemani, Karthik, madhuvaidya, amirshim, JIRA
Addressing Ted's comments. Also improving the utility script to be able to both show and kill local cluster daemons, now using the pid files stored in target/local_cluster.
REVISION DETAIL
https://reviews.facebook.net/D2709
AFFECTED FILES
bin/hbase-config.sh
bin/test/process_based_cluster.sh
src/main/java/org/apache/hadoop/hbase/HConstants.java
src/main/java/org/apache/hadoop/hbase/master/HMaster.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
src/test/java/org/apache/hadoop/hbase/HTestConst.java
src/test/java/org/apache/hadoop/hbase/TestInfoServers.java
src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
src/test/java/org/apache/hadoop/hbase/master/TestZKBasedCloseRegion.java
src/test/java/org/apache/hadoop/hbase/master/TestZKBasedReopenRegion.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java
src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java
src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
src/test/java/org/apache/hadoop/hbase/util/TestHBaseHomePath.java
src/test/java/org/apache/hadoop/hbase/util/TestProcessBasedCluster.java
> Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
> -------------------------------------------------------------------
>
> Key: HBASE-5684
> URL: https://issues.apache.org/jira/browse/HBASE-5684
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D2709.1.patch, D2709.2.patch
>
>
> Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5684) Make ProcessBasedLocalHBaseCluster
run HDFS and make it more robust
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253957#comment-13253957 ]
Phabricator commented on HBASE-5684:
------------------------------------
mbautin has committed the revision "[jira] [HBASE-5684] [89-fb] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".
REVISION DETAIL
https://reviews.facebook.net/D2709
COMMIT
https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1326045
> Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
> -------------------------------------------------------------------
>
> Key: HBASE-5684
> URL: https://issues.apache.org/jira/browse/HBASE-5684
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch, D2709.4.patch, D2757.1.patch, D2757.2.patch, D2757.3.patch, D2757.4.patch, jira-HBASE-5684-Make-ProcessBasedLocalHBaseCluster-r-2012-04-12_20_42_02.patch
>
>
> Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5684) Make ProcessBasedLocalHBaseCluster
run HDFS and make it more robust
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253081#comment-13253081 ]
Hadoop QA commented on HBASE-5684:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12522524/D2757.2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 30 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1505//console
This message is automatically generated.
> Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
> -------------------------------------------------------------------
>
> Key: HBASE-5684
> URL: https://issues.apache.org/jira/browse/HBASE-5684
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch, D2709.4.patch, D2757.1.patch, D2757.2.patch
>
>
> Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5684) Make ProcessBasedLocalHBaseCluster
run HDFS and make it more robust
Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Bautin updated HBASE-5684:
----------------------------------
Status: Patch Available (was: Open)
> Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
> -------------------------------------------------------------------
>
> Key: HBASE-5684
> URL: https://issues.apache.org/jira/browse/HBASE-5684
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch, D2709.4.patch, D2757.1.patch
>
>
> Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5684) Make ProcessBasedLocalHBaseCluster
run HDFS and make it more robust
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5684:
-------------------------------
Attachment: D2709.4.patch
mbautin updated the revision "[jira] [HBASE-5684] [89-fb] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".
Reviewers: Kannan, Liyin, khemani, Karthik, madhuvaidya, amirshim, JIRA, tedyu
Setting test timeout.
REVISION DETAIL
https://reviews.facebook.net/D2709
AFFECTED FILES
bin/hbase-config.sh
bin/test/process_based_cluster.sh
src/main/java/org/apache/hadoop/hbase/HConstants.java
src/main/java/org/apache/hadoop/hbase/master/HMaster.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
src/test/java/org/apache/hadoop/hbase/HTestConst.java
src/test/java/org/apache/hadoop/hbase/TestInfoServers.java
src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
src/test/java/org/apache/hadoop/hbase/master/TestZKBasedCloseRegion.java
src/test/java/org/apache/hadoop/hbase/master/TestZKBasedReopenRegion.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java
src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java
src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
src/test/java/org/apache/hadoop/hbase/util/TestProcessBasedCluster.java
> Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
> -------------------------------------------------------------------
>
> Key: HBASE-5684
> URL: https://issues.apache.org/jira/browse/HBASE-5684
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch, D2709.4.patch
>
>
> Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5684) Make ProcessBasedLocalHBaseCluster
run HDFS and make it more robust
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5684:
-------------------------------
Attachment: D2709.3.patch
mbautin updated the revision "[jira] [HBASE-5684] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".
Reviewers: Kannan, Liyin, khemani, Karthik, madhuvaidya, amirshim, JIRA
Ted: I have moved testHomePath to the other unit test.
REVISION DETAIL
https://reviews.facebook.net/D2709
AFFECTED FILES
bin/hbase-config.sh
bin/test/process_based_cluster.sh
src/main/java/org/apache/hadoop/hbase/HConstants.java
src/main/java/org/apache/hadoop/hbase/master/HMaster.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
src/test/java/org/apache/hadoop/hbase/HTestConst.java
src/test/java/org/apache/hadoop/hbase/TestInfoServers.java
src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
src/test/java/org/apache/hadoop/hbase/master/TestZKBasedCloseRegion.java
src/test/java/org/apache/hadoop/hbase/master/TestZKBasedReopenRegion.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java
src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java
src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
src/test/java/org/apache/hadoop/hbase/util/TestProcessBasedCluster.java
> Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
> -------------------------------------------------------------------
>
> Key: HBASE-5684
> URL: https://issues.apache.org/jira/browse/HBASE-5684
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch
>
>
> Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5684) Make ProcessBasedLocalHBaseCluster
run HDFS and make it more robust
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250952#comment-13250952 ]
Phabricator commented on HBASE-5684:
------------------------------------
tedyu has commented on the revision "[jira] [HBASE-5684] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".
Patch looks good.
I was warned that there was unit test problem when I tried to accept.
INLINE COMMENTS
src/test/java/org/apache/hadoop/hbase/util/TestHBaseHomePath.java:24 This test doesn't have assertion.
Is it possible to merge it into other test classes ?
REVISION DETAIL
https://reviews.facebook.net/D2709
> Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
> -------------------------------------------------------------------
>
> Key: HBASE-5684
> URL: https://issues.apache.org/jira/browse/HBASE-5684
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D2709.1.patch, D2709.2.patch
>
>
> Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira