[jira] [Comment Edited] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing

["Andrew Purtell (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352122#comment-15352122 ] 

Andrew Purtell edited comment on HBASE-12721 at 6/28/16 12:31 AM:
------------------------------------------------------------------

[~dimaspivak], what are your thoughts on bundling local SSD storage aka 'ephemeral store' volumes into a single logical volume and mounting it on /var/lib/docker/volumes ? This would happen underneath your clusterdock stuff whenever the instance is booted.

This would mean that any docker volumes created while launching clusters would not be persisted across instance stop and restart, but on the other hand volumes can be used to back cluster data like namenode and datanode storage. This is important for getting reasonable I/O latency and variance.

I know Docker data volumes are meant to persist independent of the container’s life cycle. We'd still have that as long as the EC2 instance is up, but not if it is put into 'stopped' state and later started again. 


was (Author: apurtell):
[~dimaspivak], what are your thoughts on bundling local SSD storage aka 'ephemeral store' volumes into a single logical volume and mounting it on /var/lib/docker/volumes ? This would mean that any docker volumes created while launching clusters would not be persisted across instance stop and restart, but on the other hand volumes can be used to back cluster data like namenode and datanode storage. This is important for getting reasonable I/O latency and variance.

I know Docker data volumes are meant to persist independent of the container’s life cycle. We'd still have that as long as the EC2 instance is up, or simply rebooted, but not if it is put into 'stopped' state and later started again. 

> Create Docker container cluster infrastructure to enable better testing
> -----------------------------------------------------------------------
>
>                 Key: HBASE-12721
>                 URL: https://issues.apache.org/jira/browse/HBASE-12721
>             Project: HBase
>          Issue Type: New Feature
>          Components: build, community, documentation, test
>            Reporter: Dima Spivak
>            Assignee: Dima Spivak
>
> Some simple work on using HBase with Docker was committed into /dev-support as "hbase_docker;" all this did was stand up a standalone cluster from source and start a shell. Now seems like a good time to extend this to be useful for applications that could actual benefit the community, especially around testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up distributed HBase clusters on a single host machine in a couple minutes and run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up an HBase cluster can be done once and then different client source/binary permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any upstream automation on build.apache.org, in part because it's a pain to set up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own top-level module ("hbase-docker" would conveniently fit the existing schema :-)), strives to create a simple framework for deploying "distributed," multi-container Apache HBase clusters.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)