You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Devaraj Das (JIRA)" <ji...@apache.org> on 2012/05/18 23:11:12 UTC
[jira] [Created] (HBASE-6053) Enhance TestRegionRebalancing test to
be a system test
Devaraj Das created HBASE-6053:
----------------------------------
Summary: Enhance TestRegionRebalancing test to be a system test
Key: HBASE-6053
URL: https://issues.apache.org/jira/browse/HBASE-6053
Project: HBase
Issue Type: Bug
Components: test
Reporter: Devaraj Das
Assignee: Devaraj Das
Priority: Minor
TestRegionRebalancing can be converted to be a system test
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6053) Enhance TestRegionRebalancing test to
be a system test
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HBASE-6053:
-------------------------------
Attachment: 6053-1.patch
Attached is a revised patch that attempts to abstract some of the MiniHBaseCluster APIs to a common place, and implemented by RealHBaseCluster and MiniHBaseCluster. Far from complete, I think, but serves as a start. Haven't yet updated the system test (TestRegionRebalancing) to use the new APIs. Review feedback welcome.
> Enhance TestRegionRebalancing test to be a system test
> ------------------------------------------------------
>
> Key: HBASE-6053
> URL: https://issues.apache.org/jira/browse/HBASE-6053
> Project: HBase
> Issue Type: Bug
> Components: test
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Priority: Minor
> Attachments: 6053-1.patch, regionRebalancingSystemTest.txt
>
>
> TestRegionRebalancing can be converted to be a system test
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6053) Enhance TestRegionRebalancing test
to be a system test
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13294075#comment-13294075 ]
Devaraj Das commented on HBASE-6053:
------------------------------------
@Enis, yes the RealHBaseCluster class upon initialization kills all the known servers. Ok, I'll remove the RandomKiller, and have it in a follow up.
> Enhance TestRegionRebalancing test to be a system test
> ------------------------------------------------------
>
> Key: HBASE-6053
> URL: https://issues.apache.org/jira/browse/HBASE-6053
> Project: HBase
> Issue Type: Sub-task
> Components: test
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Priority: Minor
> Attachments: 6053-1.patch, regionRebalancingSystemTest.txt
>
>
> TestRegionRebalancing can be converted to be a system test
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6053) Enhance TestRegionRebalancing test
to be a system test
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289822#comment-13289822 ]
Devaraj Das commented on HBASE-6053:
------------------------------------
@Stack, sorry for getting back so late on this one...
On the questions:
bq. We already have a fundamental class named ServerManager. See in master package. Naming your new class ServerManager will cause confusion.
Yes, I missed this somehow..
bq. Why we need ServerManager at all? What does it add that is not in our wrapper scripts starting and stopping servers? (Why do control up in java rather than in shell scripts w/ easier access to ssh and the shell commands this java script depends on?
The Java interfaces provides ways to control servers from tests in an easier way. It basically wraps the scripts. But one could provide alternate implementations of the interfaces.. Look at the changes in TestRegionRebalancing where the ServerManager methods are used.
bq. Whats going on here? Are you going to hunt all system properties for those named w/ a proc_ prefix?
Actually, the proc_ was added so that the ServerManager implementation methods could do a more precise grep for the hbase processes (sometimes the cmdline is long and ps doesn't display the Java class name for RegionServer or Master). BTW the proc_* are already there for the hadoop daemons (look at bin/hadoop shell script).
> Enhance TestRegionRebalancing test to be a system test
> ------------------------------------------------------
>
> Key: HBASE-6053
> URL: https://issues.apache.org/jira/browse/HBASE-6053
> Project: HBase
> Issue Type: Bug
> Components: test
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Priority: Minor
> Attachments: regionRebalancingSystemTest.txt
>
>
> TestRegionRebalancing can be converted to be a system test
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6053) Enhance TestRegionRebalancing test
to be a system test
Posted by "Enis Soztutar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13294010#comment-13294010 ]
Enis Soztutar commented on HBASE-6053:
--------------------------------------
TestRegionRebalancing assumes that there are 1 RS available, and adds other RSs afterwards. What happens when we run this on 10/100 node cluster. We can have more RS, than initial regions. Should we also generalize the testing condition? Or the test will shut down every RS, except for 1, and restart them afterwards?
We can remove RandomKiller, not used for now.
> Enhance TestRegionRebalancing test to be a system test
> ------------------------------------------------------
>
> Key: HBASE-6053
> URL: https://issues.apache.org/jira/browse/HBASE-6053
> Project: HBase
> Issue Type: Sub-task
> Components: test
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Priority: Minor
> Attachments: 6053-1.patch, regionRebalancingSystemTest.txt
>
>
> TestRegionRebalancing can be converted to be a system test
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6053) Enhance TestRegionRebalancing test
to be a system test
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289941#comment-13289941 ]
stack commented on HBASE-6053:
------------------------------
bq. But one could provide alternate implementations of the interfaces..
Such as? What you think?
bq. Look at the changes in TestRegionRebalancing where the ServerManager methods are used.
Yeah. Would be nice if this realcluster flag stuff and the calls to ServerManager were all encapsulated up inside HBaseTestingUtility? You'd pass a flag to HTU to say whether real cluster or not and then test would run the same -- you'd start and stop servers using the HTU semantics as they are now only under it it would be doing this switching you have going on here.
How hard would that be to do?
Could we evolve to that?
If you think so, then maybe review this w/ that in mind, rename the SM class (and move it under src/test?) and then we could commit.
On the proc thing, I hadn't noticed. Probably a good idea but maybe we should come up w/ something else over in hbase land? hbase_ or something?
> Enhance TestRegionRebalancing test to be a system test
> ------------------------------------------------------
>
> Key: HBASE-6053
> URL: https://issues.apache.org/jira/browse/HBASE-6053
> Project: HBase
> Issue Type: Bug
> Components: test
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Priority: Minor
> Attachments: regionRebalancingSystemTest.txt
>
>
> TestRegionRebalancing can be converted to be a system test
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6053) Enhance TestRegionRebalancing test
to be a system test
Posted by "Enis Soztutar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397187#comment-13397187 ]
Enis Soztutar commented on HBASE-6053:
--------------------------------------
After some discussions, we realized that the patch is too big to handle. I've opened HBASE-6241 for tracking the changes for the HBaseCluster/MiniHBaseCluster/RealHBaseCluster related changes. In this issue, we can track the TestRegionRebalancing-specific changes. Obviously this issue will depend on the new issue.
> Enhance TestRegionRebalancing test to be a system test
> ------------------------------------------------------
>
> Key: HBASE-6053
> URL: https://issues.apache.org/jira/browse/HBASE-6053
> Project: HBase
> Issue Type: Sub-task
> Components: test
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Priority: Minor
> Attachments: 6053-1.patch, regionRebalancingSystemTest.txt
>
>
> TestRegionRebalancing can be converted to be a system test
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6053) Enhance TestRegionRebalancing test
to be a system test
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286339#comment-13286339 ]
stack commented on HBASE-6053:
------------------------------
We already have a fundamental class named ServerManager. See in master package. Naming your new class ServerManager will cause confusion.
Why we need ServerManager at all? What does it add that is not in our wrapper scripts starting and stopping servers? (Why do control up in java rather than in shell scripts w/ easier access to ssh and the shell commands this java script depends on?
The change to TestRegionRebalancing looks good making it a Tool
Whats going on here?
{code}
Index: bin/hbase
===================================================================
--- bin/hbase (revision 1339959)
+++ bin/hbase (working copy)
@@ -305,7 +305,7 @@
# Exec unless HBASE_NOEXEC is set.
if [ "${HBASE_NOEXEC}" != "" ]; then
- "$JAVA" -XX:OnOutOfMemoryError="kill -9 %p" $JAVA_HEAP_MAX $HBASE_OPTS -classpath "$CLASSPATH" $CLASS "$@"
+ "$JAVA" -Dproc_$COMMAND {-XX:OnOutOfMemoryError="kill -9 %p" $JAVA_HEAP_MAX $HBASE_OPTS -classpath "$CLASSPATH" $CLASS "$@"
else
- exec "$JAVA" -XX:OnOutOfMemoryError="kill -9 %p" $JAVA_HEAP_MAX $HBASE_OPTS -classpath "$CLASSPATH" $CLASS "$@"
+ exec "$JAVA" -Dproc_$COMMAND -XX:OnOutOfMemoryError="kill -9 %p" $JAVA_HEAP_MAX $HBASE_OPTS -classpath "$CLASSPATH" $CLASS "$@"
fi
{code}
Are you going to hunt all system properties for those named w/ a proc_ prefix?
There seems to be a stray { in the line you add above.
> Enhance TestRegionRebalancing test to be a system test
> ------------------------------------------------------
>
> Key: HBASE-6053
> URL: https://issues.apache.org/jira/browse/HBASE-6053
> Project: HBase
> Issue Type: Bug
> Components: test
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Priority: Minor
> Attachments: regionRebalancingSystemTest.txt
>
>
> TestRegionRebalancing can be converted to be a system test
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6053) Enhance TestRegionRebalancing test
to be a system test
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13290426#comment-13290426 ]
Devaraj Das commented on HBASE-6053:
------------------------------------
bq. But one could provide alternate implementations of the interfaces..
bq. Such as? What you think?
I guess I should not have said "But one could provide alternate implementations of the interfaces..", and instead should have said that one could provide other implementations to control other servers...
For now, the methods in the abstract class are somewhat tied to the wrapper shell scripts. In the patch, I provide an implementation for the RegionServer manager, but its not hard to imagine a similar implementation can be provided for other daemons in the Hadoop ecosystem (like DataNode, HMaster, etc.).
Tweaks will be needed for sure but these could be a starting point..
bq. Yeah. Would be nice if this realcluster flag stuff and the calls to ServerManager were all encapsulated up inside HBaseTestingUtility?
IIRC I had thought about this option as well, but felt that that approach won't work as well. I don't remember what the problem was. I'll retry that approach..
bq. On the proc thing, I hadn't noticed. Probably a good idea but maybe we should come up w/ something else over in hbase land? hbase_ or something?
To be consistent with the Hadoop counterparts, I'd suggest we go with proc_server (which is a shortname for process_server). But yeah I am willing to change it to hbase_ if that's a pressing concern..
> Enhance TestRegionRebalancing test to be a system test
> ------------------------------------------------------
>
> Key: HBASE-6053
> URL: https://issues.apache.org/jira/browse/HBASE-6053
> Project: HBase
> Issue Type: Bug
> Components: test
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Priority: Minor
> Attachments: regionRebalancingSystemTest.txt
>
>
> TestRegionRebalancing can be converted to be a system test
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6053) Enhance TestRegionRebalancing test to
be a system test
Posted by "Enis Soztutar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Enis Soztutar updated HBASE-6053:
---------------------------------
Issue Type: Sub-task (was: Bug)
Parent: HBASE-6201
> Enhance TestRegionRebalancing test to be a system test
> ------------------------------------------------------
>
> Key: HBASE-6053
> URL: https://issues.apache.org/jira/browse/HBASE-6053
> Project: HBase
> Issue Type: Sub-task
> Components: test
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Priority: Minor
> Attachments: 6053-1.patch, regionRebalancingSystemTest.txt
>
>
> TestRegionRebalancing can be converted to be a system test
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6053) Enhance TestRegionRebalancing test to
be a system test
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HBASE-6053:
-------------------------------
Attachment: regionRebalancingSystemTest.txt
Here is a patch. In this patch, I also added a simple utility that can start/stop/resume/kill servers (including random start/stop). There is quite a lot of improvement opportunity on that one but seems like a start.
> Enhance TestRegionRebalancing test to be a system test
> ------------------------------------------------------
>
> Key: HBASE-6053
> URL: https://issues.apache.org/jira/browse/HBASE-6053
> Project: HBase
> Issue Type: Bug
> Components: test
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Priority: Minor
> Attachments: regionRebalancingSystemTest.txt
>
>
> TestRegionRebalancing can be converted to be a system test
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira