[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing

["Dima Spivak (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15311607#comment-15311607 ] 

Dima Spivak commented on HBASE-12721:
-------------------------------------

Please do! Here are some instructions to get started while I continue to hammer out a proper patch:
# Install Docker 1.11 on a machine. I did the development and testing of this under an Ubuntu 14.04 AWS instance with a 3.13.0-85 kernel (earlier Ubuntu kernels might have problems), so something similar would be the surest way not to run into issues. If you have a cloud instance at your disposal, you can simplify setup by taking advantage of a framework script that I've temporarily uploaded to Gist (I ran it through bit.ly because of the long raw URL, but I won't be offended if you read through the script before running it). If using the script, spin up your single cloud instance and then run {{curl -sL http://bit.ly/20UntA8}}, which will install Docker, update the host's root user's SSH keys to let you log into the containers we're going to start up, and generally make your life easier. *Note that this script copies the SSH keys from the Docker images we're creating into the {{~root/.ssh folder}} on your host, so don't use this script on a machine with an existing SSH keypair in {{~root/.ssh}}.*
# If you ran my setup script, run {{source /usr/local/bin/clusterdock.sh}} on the host machine to enable use of {{clusterdock_run}}, a small helper function that makes the framework super easy to use. _Instead of dealing with virtualenvs for this Python project, we run it out of a Docker container_. If you already had Docker installed and just want to source the script from Gist, run
{noformat}
curl -sL http://bit.ly/1ROTmE3 | source /dev/stdin
{noformat}
# Stand up an HBase cluster. For the purpose of a quick demo, I uploaded images I built with this framework to my personal Docker Hub acount. To get a running 4-node HBase cluster with Oracle Java 8u91, Hadoop 2.7.2, and the most recent build of HBase {{master}} as of when I built it an hour ago, just type:
{noformat}
clusterdock_run ./bin/start_cluster apache_hbase --primary-node=node-1 --secondary-nodes='node-{2..4}' --hbase-version=master --start-services
{noformat}

That's it. Running that will pull down the necessary HBase image (also built with this framework), setup some configs (i.e. you could have created a 5-node cluster by changing the Bash-expandable argument to {{\-\-secondary-nodes}}), and then start up services. Standard out will give information on what's happening and how to access various web UIs, but you should also be able to SSH to nodes by name (assuming you have the key-pairs setup) by using the hostnames shown during startup. Other things to try:
- Stop and remove all running containers:
{noformat}
clusterdock_run ./bin/housekeeping nuke
{noformat}
- Build your own HBase cluster images:
{noformat}
clusterdock_run ./bin/build_cluster apache_hbase --hbase-version=master --hbase-git-commit=master
{noformat}
- Learn more about options a particular script provides:
{noformat}
clusterdock_run ./bin/start_cluster -h
{noformat}

Let me know if you run into any issues!

> Create Docker container cluster infrastructure to enable better testing
> -----------------------------------------------------------------------
>
>                 Key: HBASE-12721
>                 URL: https://issues.apache.org/jira/browse/HBASE-12721
>             Project: HBase
>          Issue Type: New Feature
>            Reporter: Dima Spivak
>            Assignee: Dima Spivak
>
> Some simple work on using HBase with Docker was committed into /dev-support as "hbase_docker;" all this did was stand up a standalone cluster from source and start a shell. Now seems like a good time to extend this to be useful for applications that could actual benefit the community, especially around testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up distributed HBase clusters on a single host machine in a couple minutes and run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up an HBase cluster can be done once and then different client source/binary permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any upstream automation on build.apache.org, in part because it's a pain to set up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own top-level module ("hbase-docker" would conveniently fit the existing schema :-)), strives to create a simple framework for deploying "distributed," multi-container Apache HBase clusters.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)