[jira] [Commented] (BIGTOP-635) Implement a cluster-abstraction, discovery and manipulation framework for iTest

["Matteo Bertozzi (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BIGTOP-635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426825#comment-13426825 ] 

Matteo Bertozzi commented on BIGTOP-635:
----------------------------------------

I think that we can share some code between this and HBASE-6241.

For example i like the ClusterManager class, but I will personally rename it in Service, breaking down each pice like
 * abstract Service
 ** name/user/conf
 ** metrics
 ** start/stop/restart/signal/kill/suspend/resume/
 * HBaseRegionServer extend Service
 * HBaseMaster extend Service
 * HDFSNamenode extend Service
 * HDFSDataNode extend Service
 * MRJobTracker extend Service
 * MRTaskTracker extend Service

On top of it you can create your "Cluster" classes like
 * abstract Cluster
 * HDFSCluster extend Cluster
 ** add/remove/get DataNode
 ** add/remove/get NameNode
 * HBaseCluster extend Cluster
 ** add/remove/get Master
 ** add/remove/get Region Servers
 * MRCluster
 ** add/remove/get JobTracker
 ** add/remove/get TaskTracker

At this point you can implement each service to do the operation like
 /etc/init.d/service start|stop|...

And you've just one implementation to add/remove/get nodes for each cluster type,
and one Cluster manager that can group all together. add/remove HBaseCluster, add/remove HDFSCluster...

In the current patch HADOOP_HOME uses env while users are hardcoded (like "hdfs") an idea can be having the setUser in the service and use the hardcoded/env has default.

Another nice thing to add is exposing service metrics, I think that just a simple Map is enough that allows to make some performance tests like trying to manually call split region when number of files or region size is X, or the latency is to high.. or other nice stuff.

Also you can try to grep in the hbase code for how the mini clusters are used. Most of the tests are using the mini cluster to simulate crashes or GC collect by suspending the process and so on.
                
> Implement a cluster-abstraction, discovery and manipulation framework for iTest
> -------------------------------------------------------------------------------
>
>                 Key: BIGTOP-635
>                 URL: https://issues.apache.org/jira/browse/BIGTOP-635
>             Project: Bigtop
>          Issue Type: New Feature
>          Components: Tests
>    Affects Versions: 0.4.0
>            Reporter: Roman Shaposhnik
>            Assignee: Sujay Rau
>             Fix For: 0.5.0
>
>         Attachments: BigtopClusterManager.zip, BigtopClusterManagerv2.zip, ClusterManagerAPI.pdf, bigtop-635.patch, bigtop-635.patch
>
>
> We've come to a point where our tests need to have a uniform way of interfacing with the cluster under test. It is no longer ok to assume that the test can be executed on a particular node (and thus have access to services running on it). It is also less than ideal for tests to assume a particular type of interaction with the services since it tends to break in different deployment scenarios. 
> A framework that needs to be put in place has to be capable of (regardless of where a test using it is executed on):
>   # representing the abstract configuration of the cluster
>   # representing the abstract topology of the entire cluster (services running on a cluster, nodes hosting the daemons, racks, etc).
>   # giving tests an ability to query this topology
>   # giving tests an ability to affect the nodes in that topology in a particular way (refreshing configuration, restarting services, etc.)
> Of course, the ideal solution here would be to give Bigtop tests a programmatic access to a Hadoop cluster management framework such as Cloudera's CM or Apache Ambari. 
> As with any ideal solutions I don't think it is realistic though. Hence we have to cook something up. At this point I'm really focused on getting the API right and I'm totally fine with an implementation of that API to be something as silly as a bunch of ssh-based scripts or something.
> This JIRA is primarily focused on coming up with such an API. Anybody who's willing to help is welcome to.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira