You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@bigtop.apache.org by ev...@apache.org on 2017/07/01 13:36:52 UTC
[08/50] [abbrv] bigtop git commit: BIGTOP-2772. Add --dryrun mode
into helper script and document
BIGTOP-2772. Add --dryrun mode into helper script and document
Project: http://git-wip-us.apache.org/repos/asf/bigtop/repo
Commit: http://git-wip-us.apache.org/repos/asf/bigtop/commit/5874ba53
Tree: http://git-wip-us.apache.org/repos/asf/bigtop/tree/5874ba53
Diff: http://git-wip-us.apache.org/repos/asf/bigtop/diff/5874ba53
Branch: refs/heads/branch-1.2
Commit: 5874ba53dba27454cb100b5637ba426430ce6d1d
Parents: ef20c6c
Author: Evans Ye <ev...@apache.org>
Authored: Tue May 23 22:36:30 2017 +0800
Committer: Evans Ye <ev...@apache.org>
Committed: Sat Jul 1 12:52:54 2017 +0000
----------------------------------------------------------------------
docker/sandbox/README.md | 42 +++++++++++---------
docker/sandbox/build.sh | 1 +
.../sandbox/site.yaml.template.centos-6_hadoop | 2 +-
.../sandbox/site.yaml.template.debian-8_hadoop | 2 +-
4 files changed, 26 insertions(+), 21 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/bigtop/blob/5874ba53/docker/sandbox/README.md
----------------------------------------------------------------------
diff --git a/docker/sandbox/README.md b/docker/sandbox/README.md
index 711903c..d7a6105 100644
--- a/docker/sandbox/README.md
+++ b/docker/sandbox/README.md
@@ -19,52 +19,56 @@ A tool to build and run big data pseudo cluster using Docker.
## How to run
-* Make sure you have Docker installed. We've tested this using [Docker for Mac](https://docs.docker.com/docker-for-mac/)
+Make sure you have Docker installed. We've tested this using [Docker for Mac](https://docs.docker.com/docker-for-mac/)
-* Currently supported OS list:
+Currently supported OS list:
* centos-6
* debian-8
- * ubuntu-14.04
+ * ubuntu-16.04
-* Running Hadoop
+### Run Hadoop HDFS
```
-docker run -ti --privileged -p 50070:50070 bigtop/sandbox:centos-6_hadoop
+docker run -d -p 50070:50070 bigtop/sandbox:centos-6_hdfs
```
-* Running Spark (Standalone mode)
+### Run Spark on YARN
```
-docker run -ti --privileged -p 8080:8080 bigtop/sandbox:debian-8_spark
+docker run -d -p 50070:50070 -p 8088:8088 -p 8080:8080 bigtop/sandbox:debian-8_hdfs_yarn_spark
```
-* Running Hadoop + HBase
+### Run Hadoop HDFS + HBase
```
-docker run -ti --privileged -p 50070:50070 -p 60010:60010 bigtop/sandbox:ubuntu-14.04_hbase
+docker run -d -p 50070:50070 -p 60010:60010 bigtop/sandbox:ubuntu-16.04_hdfs_hbase
```
## How to build
-### Examples
+### Build a Hadoop HDFS sandbox image
-* Build sandbox image that has Hadoop provisioned
+```
+./build.sh -a bigtop -o centos-6 -c hdfs
+```
+
+### Build a Hadoop HDFS, Hadoop YARN, and Spark on YARN sandbox image
```
-./build.sh -a bigtop -o debian-8 -c hadoop
+./build.sh -a bigtop -o debian-8 -c "hdfs, yarn, spark"
```
-* Build sandbox image that has Hadoop and Spark provisioned
+### Build a Hadoop HDFS and HBase sandbox image
```
-./build.sh -a bigtop -o debian-8 -c "hadoop, spark"
+./build.sh -a bigtop -o ubuntu-16.04 -c "hdfs, hbase"
```
-* Build sandbox image that has Hadoop and HBase provisioned
+### Use --dryrun to skip the build and get Dockerfile and configuration
```
-./build.sh -a bigtop -o debian-8 -c "hadoop, yarn, hbase"
+./build.sh -a bigtop -o ubuntu-16.04 -c "hdfs, hbase" --dryrun
```
### Change the repository of packages
@@ -73,10 +77,10 @@ docker run -ti --privileged -p 50070:50070 -p 60010:60010 bigtop/sandbox:ubuntu-
```
export REPO=http://ci.bigtop.apache.org:8080/job/Bigtop-trunk-repos/BUILD_ENVIRONMENTS=centos-6%2Clabel=docker-slave-06//ws/output
-./build.sh -a bigtop -o centos-6 -c "hadoop, spark, ignite"
+./build.sh -a bigtop -o centos-6 -c "hdfs, yarn, spark, ignite"
```
-### Customize your Hadoop stack
+### Customize your Big Data Stack
* Edit *site.yaml.template.centos-6_hadoop* to create your own prefered stack
@@ -89,7 +93,7 @@ vim site.yaml.template.centos-6_hadoop_ignite
```
...
-hadoop_cluster_node::cluster_components: [hadoop, yarn, ignite]
+hadoop_cluster_node::cluster_components: [hdfs, yarn, ignite]
...
```
http://git-wip-us.apache.org/repos/asf/bigtop/blob/5874ba53/docker/sandbox/build.sh
----------------------------------------------------------------------
diff --git a/docker/sandbox/build.sh b/docker/sandbox/build.sh
index 4dfe2d3..58a76e7 100755
--- a/docker/sandbox/build.sh
+++ b/docker/sandbox/build.sh
@@ -145,6 +145,7 @@ usage() {
echo " -c, --components Specify components to build."
echo " You need to specify a comma separated, quoted string."
echo " For example: --components \"hadoop, yarn\""
+ echo " -d, --dryrun Generate the Dockerfile and configuration and skip the build."
echo " -f, --file Specify a written site.yaml config file."
echo " -o, --operating-system Specify an OS from Bigtop supported OS list."
echo " RPM base: ${RPMS[*]}"
http://git-wip-us.apache.org/repos/asf/bigtop/blob/5874ba53/docker/sandbox/site.yaml.template.centos-6_hadoop
----------------------------------------------------------------------
diff --git a/docker/sandbox/site.yaml.template.centos-6_hadoop b/docker/sandbox/site.yaml.template.centos-6_hadoop
index 4fd7497..51fd682 100644
--- a/docker/sandbox/site.yaml.template.centos-6_hadoop
+++ b/docker/sandbox/site.yaml.template.centos-6_hadoop
@@ -16,5 +16,5 @@
bigtop::hadoop_head_node: "head.node.fqdn"
hadoop::hadoop_storage_dirs: [/data/1, /data/2]
bigtop::bigtop_repo_uri: http://bigtop-repos.s3.amazonaws.com/releases/1.2.0/centos/6/x86_64
-hadoop_cluster_node::cluster_components: [hdfs, yarn, mapreduce]
+hadoop_cluster_node::cluster_components: [hdfs, yarn, mapred-app]
bigtop::jdk_package_name: java-1.8.0-openjdk-devel.x86_64
http://git-wip-us.apache.org/repos/asf/bigtop/blob/5874ba53/docker/sandbox/site.yaml.template.debian-8_hadoop
----------------------------------------------------------------------
diff --git a/docker/sandbox/site.yaml.template.debian-8_hadoop b/docker/sandbox/site.yaml.template.debian-8_hadoop
index d87146b..21c5d9d 100644
--- a/docker/sandbox/site.yaml.template.debian-8_hadoop
+++ b/docker/sandbox/site.yaml.template.debian-8_hadoop
@@ -16,5 +16,5 @@
bigtop::hadoop_head_node: "head.node.fqdn"
hadoop::hadoop_storage_dirs: [/data/1, /data/2]
bigtop::bigtop_repo_uri: http://bigtop-repos.s3.amazonaws.com/releases/1.2.0/debian/8/x86_64
-hadoop_cluster_node::cluster_components: [hdfs, yarn, mapreduce]
+hadoop_cluster_node::cluster_components: [hdfs, yarn, mapred-app]
bigtop::jdk_package_name: openjdk-8-jdk