[jira] [Updated] (HADOOP-16063) Docker based pseudo-cluster definitions and test scripts for Hdfs/Yarn

["Elek, Marton (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-16063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Elek, Marton updated HADOOP-16063:
----------------------------------
    Labels:   (was: newbie)

> Docker based pseudo-cluster definitions and test scripts for Hdfs/Yarn
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-16063
>                 URL: https://issues.apache.org/jira/browse/HADOOP-16063
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Elek, Marton
>            Priority: Major
>
> During the recent releases of Apache Hadoop Ozone we had multiple experiments using docker/docker-compose to support the development of ozone.
> As of now the hadoop-ozone distribution contains two directories in additional the regular hadoop directories (bin, share/lib, etc
> h3. compose
> The ./compose directory of the distribution contains different type of pseudo-cluster definitions. To start an ozone cluster is as easy as "cd compose/ozone && docker-compose up-d"
> The clusters also could be scaled up and down (docker-compose scale datanode=3)
> There are multiple cluster definitions for different use cases (for example ozone+s3 or hdfs+ozone).
> The docker-compose files are based on apache/hadoop-runner image which is an "empty" image. It doesnt' contain any hadoop distribution. Instead the current hadoop is used (the ../.. is mapped as a volume at /opt/hadoop)
> With this approach it's very easy to 1) start a cluster from the distribution 2) test any patch from the dev tree, as after any build a new cluster can be started easily (with multiple nodes and datanodes)
> h3. smoketest
> We also started to use a simple robotframework based test suite. (see ./smoketest directory). It's a high level test definition very similar to the smoketests which are executed manually by the contributors during a release vote.
> But it's a formal definition to start cluster from different docker-compose definitions and execute simple shell scripts (and compare the output).
>  
> I believe that both approaches helped a lot during the development of ozone and I propose to do the same improvements on the main hadoop distribution.
> I propose to provide docker-compose based example cluster definitions for yarn/hdfs and for different use cases (simple hdfs, router based federation, etc.)
> It can help to understand the different configuration and try out new features with predefined config set.
> Long term we can also add robottests to help the release votes (basic wordcount/mr tests could be scripted)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org