You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@bigtop.apache.org by ev...@apache.org on 2015/01/27 14:56:37 UTC
bigtop git commit: BIGTOP-1603. Rename docker-puppet and
vagrant-puppet to better indicate its function and difference
Repository: bigtop
Updated Branches:
refs/heads/master f7fc0fe78 -> d8a506577
BIGTOP-1603. Rename docker-puppet and vagrant-puppet to better indicate its function and difference
Project: http://git-wip-us.apache.org/repos/asf/bigtop/repo
Commit: http://git-wip-us.apache.org/repos/asf/bigtop/commit/d8a50657
Tree: http://git-wip-us.apache.org/repos/asf/bigtop/tree/d8a50657
Diff: http://git-wip-us.apache.org/repos/asf/bigtop/diff/d8a50657
Branch: refs/heads/master
Commit: d8a50657734dd087472f72296d5152f28bb4f899
Parents: f7fc0fe
Author: Evans Ye <ev...@apache.org>
Authored: Tue Jan 20 13:44:28 2015 +0000
Committer: Evans Ye <ev...@apache.org>
Committed: Tue Jan 27 13:55:20 2015 +0000
----------------------------------------------------------------------
README.md | 2 +-
bigtop-deploy/puppet/README.md | 2 +-
bigtop-deploy/vm/docker-puppet/.gitignore | 3 -
bigtop-deploy/vm/docker-puppet/Dockerfile | 33 ----
bigtop-deploy/vm/docker-puppet/README.md | 179 -------------------
bigtop-deploy/vm/docker-puppet/Vagrantfile | 74 --------
.../vm/docker-puppet/boot2docker/Vagrantfile | 24 ---
bigtop-deploy/vm/docker-puppet/docker-hadoop.sh | 117 ------------
bigtop-deploy/vm/docker-puppet/provision.sh | 36 ----
.../vm/docker-puppet/vagrantconfig.yaml | 13 --
.../vm/vagrant-puppet-docker/.gitignore | 3 +
.../vm/vagrant-puppet-docker/Dockerfile | 33 ++++
.../vm/vagrant-puppet-docker/README.md | 179 +++++++++++++++++++
.../vm/vagrant-puppet-docker/Vagrantfile | 74 ++++++++
.../boot2docker/Vagrantfile | 24 +++
.../vm/vagrant-puppet-docker/docker-hadoop.sh | 117 ++++++++++++
.../vm/vagrant-puppet-docker/provision.sh | 36 ++++
.../vm/vagrant-puppet-docker/vagrantconfig.yaml | 13 ++
bigtop-deploy/vm/vagrant-puppet-vm/.gitignore | 1 +
bigtop-deploy/vm/vagrant-puppet-vm/README.md | 92 ++++++++++
bigtop-deploy/vm/vagrant-puppet-vm/Vagrantfile | 123 +++++++++++++
.../vm/vagrant-puppet-vm/vagrantconfig.yaml | 8 +
bigtop-deploy/vm/vagrant-puppet/.gitignore | 1 -
bigtop-deploy/vm/vagrant-puppet/README.md | 92 ----------
bigtop-deploy/vm/vagrant-puppet/Vagrantfile | 123 -------------
.../vm/vagrant-puppet/vagrantconfig.yaml | 8 -
26 files changed, 705 insertions(+), 705 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index de51707..cdf4c4c 100644
--- a/README.md
+++ b/README.md
@@ -24,7 +24,7 @@ The primary goal of Apache Bigtop is to build a community around the packaging a
Immediately Get Started with Deployment and Smoke Testing of BigTop
===================================================================
-The simplest way to get a feel for how bigtop works, is to just cd into `bigtop-deploy/vm` and try out the recipes under vagrant-puppet, docker-puppet, and so on. Each one rapidly spins up, and runs the bigtop smoke tests on, a local bigtop based big data distribution. Once you get the gist, you can hack around with the recipes to learn how the puppet/rpm/smoke-tests all work together, going deeper into the components you are interested in as described below.
+The simplest way to get a feel for how bigtop works, is to just cd into `bigtop-deploy/vm` and try out the recipes under vagrant-puppet-vm, vagrant-puppet-docker, and so on. Each one rapidly spins up, and runs the bigtop smoke tests on, a local bigtop based big data distribution. Once you get the gist, you can hack around with the recipes to learn how the puppet/rpm/smoke-tests all work together, going deeper into the components you are interested in as described below.
Quick overview of source code directories
=========================================
http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/puppet/README.md
----------------------------------------------------------------------
diff --git a/bigtop-deploy/puppet/README.md b/bigtop-deploy/puppet/README.md
index c76185a..b2a8b6a 100644
--- a/bigtop-deploy/puppet/README.md
+++ b/bigtop-deploy/puppet/README.md
@@ -30,7 +30,7 @@ If you have site.csv in /etc/puppet/config,
Then you should use --confdir=/etc/puppet , and puppet finds the config dir underneath.
-As an end to end example, you can follow the vagrant-puppet recipes to see how to set up
+As an end to end example, you can follow the vagrant-puppet-vm recipes to see how to set up
a puppet managed bigtop hadoop installation. Those examples are gauranteed to work and
http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/docker-puppet/.gitignore
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/docker-puppet/.gitignore b/bigtop-deploy/vm/docker-puppet/.gitignore
deleted file mode 100644
index 2cd389d..0000000
--- a/bigtop-deploy/vm/docker-puppet/.gitignore
+++ /dev/null
@@ -1,3 +0,0 @@
-config.rb
-config/
-hosts
http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/docker-puppet/Dockerfile
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/docker-puppet/Dockerfile b/bigtop-deploy/vm/docker-puppet/Dockerfile
deleted file mode 100644
index 8e65126..0000000
--- a/bigtop-deploy/vm/docker-puppet/Dockerfile
+++ /dev/null
@@ -1,33 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-FROM bigtop/seed:centos-6.4
-
-# enable ssh
-RUN yum -y install openssh-server openssh-clients sudo
-RUN sed -i.bak s/UsePAM\ yes/UsePAM\ no/ /etc/ssh/sshd_config
-RUN ssh-keygen -q -N "" -t dsa -f /etc/ssh/ssh_host_dsa_key
-RUN ssh-keygen -q -N "" -t rsa -f /etc/ssh/ssh_host_rsa_key
-
-# requiretty off
-RUN sed -i.bak 's/requiretty/!requiretty/' /etc/sudoers
-
-# setup vagrant account
-RUN mkdir /root/.ssh/
-RUN chmod 0755 /root/.ssh
-RUN wget http://github.com/mitchellh/vagrant/raw/master/keys/vagrant.pub --no-check-certificate -O /root/.ssh/authorized_keys
-RUN chmod 0644 /root/.ssh/authorized_keys
-
-CMD /usr/sbin/sshd -D
http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/docker-puppet/README.md
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/docker-puppet/README.md b/bigtop-deploy/vm/docker-puppet/README.md
deleted file mode 100644
index 7848571..0000000
--- a/bigtop-deploy/vm/docker-puppet/README.md
+++ /dev/null
@@ -1,179 +0,0 @@
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
-
-------------------------------------------------------------------------------------------------------------------------------------------------------
-
-#BigTop Docker provisioner
-
-## Overview
-
-The Vagrantfile definition and wrapper script that creates Bigtop virtual Hadoop cluster on top of Docker containers for you, by pulling from existing publishing bigtop repositories.
-This cluster can be used:
-
-- to test bigtop smoke tests
-- to test bigtop puppet recipes
-
-These containers start sshd daemons, which vagrant uses to provision and install the hadoop cluster.
-
-This has been verified on docker client 1.2.0, with api version 1.15, and vagrant 1.6.5 on Fedora 20 as well as Centos 6.
-
-## Prerequisites
-
-### OS X and Windows
-
-* Install [VirtualBox](https://www.virtualbox.org/wiki/Downloads)
-
-* Install [Vagrant](http://www.vagrantup.com/downloads.html). Need version 1.6.5 or higher.
-
-### Linux
-
-* [Kernel Requirements](http://docker.readthedocs.org/en/v0.5.3/installation/kernel/)
-
-* Install [Docker](https://docs.docker.com/installation/)
-
-* Install [Vagrant](http://www.vagrantup.com/downloads.html)
-
-## Getting Started
-
-* Create a 3 node Bigtop Hadoop cluster from scratch
-
-NOTE : SELinux can PREVENT you from ssh'ing into your docker container.
-As a brute force way to disable it - remove it from vi /etc/sysconfig/docker arguments
-(fedora and centos may by default launch docker daemon with the --selinux-enabled option)!
-In the future, lets update this README with the RIGHT way to allow selinux without breaking
-ssh into a docker container!
-
-```
-service docker restart
-docker pull bigtop/seed:centos-6.4
-```
-
-Now, you can start your cluster:
-
-```
-cd bigtop/bigtop-deploy/vm/docker-puppet
-./docker-hadoop.sh --build-image --create 3
-```
-In case of errors you can attempt running as root, or else, ping the mailing list.
-
-## USAGE
-
-1) Build up the base Docker image that supports Vagrant.
-
-```
-./docker-hadoop.sh --build-image
-```
-
-2) Create a Bigtop Hadoop cluster by given # of node. (will place a file called config.rb)
-
-```
-./docker-hadoop.sh --create 3
-```
-
-3) Destroy the cluster.
-
-```
-./docker-hadoop.sh --destroy
-```
-
-4) Update your cluster after doing configuration changes. (re-run puppet apply)
-
-```
-./docker-hadoop.sh --provision
-```
-
-5) Run Bigtop smoke tests
-
-```
-./docker-hadoop.sh --smoke-tests
-```
-
-6) Chain your operations with-in one command.
-
-```
-./docker-hadoop.sh --build-image --create 5 --smoke-tests --destroy
-```
-
-Commands will be executed by following order:
-
-```
-build-image => create 5 node cluster => destroy the cluster
-```
-
-7) Run hbase-test.sh to evaluate the deployment.
-
-```
-../utils/hbase-test.sh
-```
-
-8) See helper message:
-
-```
-./docker-hadoop.sh -h
-usage: docker-hadoop.sh [options]
- -b, --build-image Build base Docker image for Bigtop Hadoop
- (must be exectued at least once before creating cluster)
- -c NUM_INSTANCES, --create=NUM_INSTANCES Create a docker based Bigtop Hadoop cluster
- -p, --provision Deploy configuration changes
- -s, --smoke-tests Run Bigtop smoke tests
- -d, --destroy Destroy the cluster
- -h, --help
-
-```
-
-##Configurations
-
-* There are several parameters can be configured in the vagrantconfig.yaml:
-
-1) Modify memory limit for Docker containers
-
-```
-docker:
- memory_size: "2048"
-
-```
-
-2) If you're running Docker provisioner on OS X or Windows, you can customize the boot2docker VM settings
-
-```
-boot2docker:
- memory_size: "4196"
- number_cpus: "2"
-```
-
-3) Use different host ports mapping for web UIs
-
-```
-namenode_ui_port: "50070"
-yarn_ui_port: "8088"
-hbase_ui_port: "60010"
-
-```
-Note: If running on OS X or Windows, the boot2docker VM should be reloaded after ports changed
-
-
-##Configure Apache Hadoop ecosystem components
-* Choose the ecosystem you want to be deployed by modifying components in vagrantconfig.yaml
-
-```
-components: "hadoop,hbase,yarn,..."
-```
-
-By default, Apache Hadoop, YARN, and Apache HBase will be installed.
-See `bigtop-deploy/puppet/config/site.csv.example` for more details.
-
-##Notes
-
-* Users currently using vagrant 1.6+ is strongly recommanded to upgrade to 1.6.4+, otherwise you will encounter the [issue](https://github.com/mitchellh/vagrant/issues/3769) when installing plguins
http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/docker-puppet/Vagrantfile
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/docker-puppet/Vagrantfile b/bigtop-deploy/vm/docker-puppet/Vagrantfile
deleted file mode 100755
index 3ec023b..0000000
--- a/bigtop-deploy/vm/docker-puppet/Vagrantfile
+++ /dev/null
@@ -1,74 +0,0 @@
-# -*- mode: ruby -*-
-# vi: set ft=ruby :
-
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-require 'fileutils'
-
-# number of instances
-$num_instances = 3
-
-# load configuration from config.rb
-CONFIG = File.join(File.dirname(__FILE__), "config.rb")
-if File.exist?(CONFIG)
- require CONFIG
-end
-
-require "yaml"
-
-_config = YAML.load(File.open(File.join(File.dirname(__FILE__), "vagrantconfig.yaml"), File::RDONLY).read)
-CONF = _config
-
-ENV['VAGRANT_DEFAULT_PROVIDER'] ||= 'docker'
-
-Vagrant.require_version ">= 1.6.0"
-
-VAGRANTFILE_API_VERSION = "2"
-Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
-
- # image definition
- config.vm.define "image", autostart: false do |image|
- image.vm.provider "docker" do |d|
- d.build_dir = "."
- d.build_args = ["-t",CONF["docker"]["image"]]
- d.vagrant_machine = "boot2docker"
- d.vagrant_vagrantfile = "./boot2docker/Vagrantfile"
- end
- image.ssh.username = "root"
- image.ssh.port = 22
- end
-
- # nodes definition
- (1..$num_instances).each do |i|
- config.vm.define "bigtop#{i}" do |bigtop|
- # docker container settings
- bigtop.vm.provider "docker" do |d|
- d.image = CONF["docker"]["image"]
- d.create_args = ["--privileged=true", "-m", CONF["docker"]['memory_size'] + "m"]
- if "#{i}" == "1"
- d.ports = [CONF["namenode_ui_port"] + ":50070", CONF["yarn_ui_port"] + ":8088", CONF["hbase_ui_port"] + ":60010"]
- end
- d.vagrant_machine = "boot2docker"
- d.vagrant_vagrantfile = "./boot2docker/Vagrantfile"
- end
- bigtop.ssh.username = "root"
- bigtop.ssh.port = 22
- bigtop.vm.hostname = "bigtop#{i}.docker"
- bigtop.vm.synced_folder "../../../", "/bigtop-home"
- end
- end
-
-end
http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/docker-puppet/boot2docker/Vagrantfile
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/docker-puppet/boot2docker/Vagrantfile b/bigtop-deploy/vm/docker-puppet/boot2docker/Vagrantfile
deleted file mode 100644
index 5913fec..0000000
--- a/bigtop-deploy/vm/docker-puppet/boot2docker/Vagrantfile
+++ /dev/null
@@ -1,24 +0,0 @@
-# -*- mode: ruby -*-
-# vi: set ft=ruby :
-
-require "yaml"
-
-_config = YAML.load(File.open(File.join(File.dirname(__FILE__), "../vagrantconfig.yaml"), File::RDONLY).read)
-CONF = _config
-
-VAGRANTFILE_API_VERSION = "2"
-
-Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
-
- config.vm.define "boot2docker" do |node|
- node.vm.box = "yungsang/boot2docker"
- node.vm.provider "virtualbox" do |vb|
- vb.customize ["modifyvm", :id, "--memory", CONF["boot2docker"]["memory_size"]]
- vb.customize ['modifyvm', :id, '--cpus', CONF["boot2docker"]['number_cpus']]
- end
- node.vm.network "forwarded_port", host: CONF["namenode_ui_port"], guest: 50070
- node.vm.network "forwarded_port", host: CONF["yarn_ui_port"], guest: 8088
- node.vm.network "forwarded_port", host: CONF["hbase_ui_port"], guest: 60010
- end
-
-end
http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/docker-puppet/docker-hadoop.sh
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/docker-puppet/docker-hadoop.sh b/bigtop-deploy/vm/docker-puppet/docker-hadoop.sh
deleted file mode 100755
index 4b504b3..0000000
--- a/bigtop-deploy/vm/docker-puppet/docker-hadoop.sh
+++ /dev/null
@@ -1,117 +0,0 @@
-#!/bin/bash
-
-usage() {
- echo "usage: $PROG [options]"
- echo " -b, --build-image Build base Docker image for Bigtop Hadoop"
- echo " (must be exectued at least once before creating cluster)"
- echo " -c NUM_INSTANCES, --create=NUM_INSTANCES Create a Docker based Bigtop Hadoop cluster"
- echo " -p, --provision Deploy configuration changes"
- echo " -s, --smoke-tests Run Bigtop smoke tests"
- echo " -d, --destroy Destroy the cluster"
- echo " -h, --help"
- exit 1
-}
-
-build-image() {
- vagrant up image --provider docker
- {
- echo "echo -e '\nBUILD IMAGE SUCCESS.\n'" |vagrant ssh image
- } || {
- >&2 echo -e "\nBUILD IMAGE FAILED!\n"
- exit 2
- }
-}
-
-create() {
- echo "\$num_instances = $1" > config.rb
- vagrant up --no-parallel
- nodes=(`vagrant status |grep running |grep -v image |awk '{print $1}'`)
- hadoop_head_node=(`echo "hostname -f" |vagrant ssh ${nodes[0]} |tail -n 1`)
- repo=$(get-yaml-config repo)
- components=$(get-yaml-config components)
- echo "/bigtop-home/bigtop-deploy/vm/utils/setup-env.sh" |vagrant ssh ${nodes[0]}
- echo "/vagrant/provision.sh $hadoop_head_node $repo $components" |vagrant ssh ${nodes[0]}
- bigtop-puppet ${nodes[0]}
- for ((i=1 ; i<${#nodes[*]} ; i++)); do
- (
- echo "/bigtop-home/bigtop-deploy/vm/utils/setup-env.sh" |vagrant ssh ${nodes[$i]}
- echo "/vagrant/provision.sh $hadoop_head_node $repo $components" |vagrant ssh ${nodes[$i]}
- bigtop-puppet ${nodes[$i]}
- ) &
- done
- wait
-}
-
-provision() {
- nodes=(`vagrant status |grep running |grep -v image |awk '{print $1}'`)
- for node in $nodes; do
- bigtop-puppet $node &
- done
- wait
-}
-
-smoke-tests() {
- nodes=(`vagrant status |grep running |grep -v image |awk '{print $1}'`)
- echo "/bigtop-home/bigtop-deploy/vm/utils/smoke-tests.sh" |vagrant ssh ${nodes[0]}
-}
-
-
-destroy() {
- rm -rf ./hosts ./config ./config.rb
- nodes=(`vagrant status |grep running |grep -v image |awk '{print $1}'`)
- for node in $nodes; do
- vagrant destroy -f $node
- done
- wait
-}
-
-bigtop-puppet() {
- echo "puppet apply -d --confdir=/vagrant --modulepath=/bigtop-home/bigtop-deploy/puppet/modules:/etc/puppet/modules /bigtop-home/bigtop-deploy/puppet/manifests/site.pp" |vagrant ssh $1
-}
-
-get-yaml-config() {
- RUBY_EXE=ruby
- which ruby > /dev/null 2>&1
- if [ $? -ne 0 ]; then
- # use vagrant embedded ruby on Windows
- RUBY_EXE=$(dirname $(which vagrant))/../embedded/bin/ruby
- fi
- RUBY_SCRIPT="data = YAML::load(STDIN.read); puts data['$1'];"
- cat vagrantconfig.yaml | $RUBY_EXE -ryaml -e "$RUBY_SCRIPT" | tr -d '\r'
-}
-
-PROG=`basename $0`
-
-if [ $# -eq 0 ]; then
- usage
-fi
-
-while [ $# -gt 0 ]; do
- case "$1" in
- -b|--build-image)
- build-image
- shift;;
- -c|--create)
- if [ $# -lt 2 ]; then
- echo "Create requires a number" 1>&2
- usage
- fi
- create $2
- shift 2;;
- -p|--provision)
- provision
- shift;;
- -s|--smoke-tests)
- smoke-tests
- shift;;
- -d|--destroy)
- destroy
- shift;;
- -h|--help)
- usage
- shift;;
- *)
- echo "Unknown argument: '$1'" 1>&2
- usage;;
- esac
-done
http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/docker-puppet/provision.sh
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/docker-puppet/provision.sh b/bigtop-deploy/vm/docker-puppet/provision.sh
deleted file mode 100755
index da087c6..0000000
--- a/bigtop-deploy/vm/docker-puppet/provision.sh
+++ /dev/null
@@ -1,36 +0,0 @@
-#!/bin/bash
-
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-sysctl kernel.hostname=`hostname -f`
-
-# Unmount device /etc/hosts and replace it by a shared hosts file
-echo -e "`hostname -i`\t`hostname -f`" >> /vagrant/hosts
-umount /etc/hosts
-mv /etc/hosts /etc/hosts.bak
-ln -s /vagrant/hosts /etc/hosts
-
-# Prepare puppet configuration file
-cd /etc/puppet/modules && puppet module install puppetlabs/stdlib
-
-mkdir /vagrant/config
-cat > /vagrant/config/site.csv << EOF
-hadoop_head_node,$1
-hadoop_storage_dirs,/data/1,/data/2
-bigtop_yumrepo_uri,$2
-jdk_package_name,java-1.7.0-openjdk-devel.x86_64
-components,$3
-EOF
http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/docker-puppet/vagrantconfig.yaml
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/docker-puppet/vagrantconfig.yaml b/bigtop-deploy/vm/docker-puppet/vagrantconfig.yaml
deleted file mode 100644
index 6a53d02..0000000
--- a/bigtop-deploy/vm/docker-puppet/vagrantconfig.yaml
+++ /dev/null
@@ -1,13 +0,0 @@
-docker:
- memory_size: "2048"
- image: "bigtop/ssh:centos-6.4"
-
-boot2docker:
- memory_size: "4196"
- number_cpus: "2"
-
-repo: "http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-0.8.0/label=centos6/6/artifact/output/"
-components: "hadoop,yarn"
-namenode_ui_port: "50070"
-yarn_ui_port: "8088"
-hbase_ui_port: "60010"
http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet-docker/.gitignore
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/vagrant-puppet-docker/.gitignore b/bigtop-deploy/vm/vagrant-puppet-docker/.gitignore
new file mode 100644
index 0000000..2cd389d
--- /dev/null
+++ b/bigtop-deploy/vm/vagrant-puppet-docker/.gitignore
@@ -0,0 +1,3 @@
+config.rb
+config/
+hosts
http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet-docker/Dockerfile
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/vagrant-puppet-docker/Dockerfile b/bigtop-deploy/vm/vagrant-puppet-docker/Dockerfile
new file mode 100644
index 0000000..8e65126
--- /dev/null
+++ b/bigtop-deploy/vm/vagrant-puppet-docker/Dockerfile
@@ -0,0 +1,33 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM bigtop/seed:centos-6.4
+
+# enable ssh
+RUN yum -y install openssh-server openssh-clients sudo
+RUN sed -i.bak s/UsePAM\ yes/UsePAM\ no/ /etc/ssh/sshd_config
+RUN ssh-keygen -q -N "" -t dsa -f /etc/ssh/ssh_host_dsa_key
+RUN ssh-keygen -q -N "" -t rsa -f /etc/ssh/ssh_host_rsa_key
+
+# requiretty off
+RUN sed -i.bak 's/requiretty/!requiretty/' /etc/sudoers
+
+# setup vagrant account
+RUN mkdir /root/.ssh/
+RUN chmod 0755 /root/.ssh
+RUN wget http://github.com/mitchellh/vagrant/raw/master/keys/vagrant.pub --no-check-certificate -O /root/.ssh/authorized_keys
+RUN chmod 0644 /root/.ssh/authorized_keys
+
+CMD /usr/sbin/sshd -D
http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet-docker/README.md
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/vagrant-puppet-docker/README.md b/bigtop-deploy/vm/vagrant-puppet-docker/README.md
new file mode 100644
index 0000000..6d609b6
--- /dev/null
+++ b/bigtop-deploy/vm/vagrant-puppet-docker/README.md
@@ -0,0 +1,179 @@
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+
+------------------------------------------------------------------------------------------------------------------------------------------------------
+
+#BigTop Docker provisioner
+
+## Overview
+
+The Vagrantfile definition and wrapper script that creates Bigtop virtual Hadoop cluster on top of Docker containers for you, by pulling from existing publishing bigtop repositories.
+This cluster can be used:
+
+- to test bigtop smoke tests
+- to test bigtop puppet recipes
+
+These containers start sshd daemons, which vagrant uses to provision and install the hadoop cluster.
+
+This has been verified on docker client 1.2.0, with api version 1.15, and vagrant 1.6.5 on Fedora 20 as well as Centos 6.
+
+## Prerequisites
+
+### OS X and Windows
+
+* Install [VirtualBox](https://www.virtualbox.org/wiki/Downloads)
+
+* Install [Vagrant](http://www.vagrantup.com/downloads.html). Need version 1.6.5 or higher.
+
+### Linux
+
+* [Kernel Requirements](http://docker.readthedocs.org/en/v0.5.3/installation/kernel/)
+
+* Install [Docker](https://docs.docker.com/installation/)
+
+* Install [Vagrant](http://www.vagrantup.com/downloads.html)
+
+## Getting Started
+
+* Create a 3 node Bigtop Hadoop cluster from scratch
+
+NOTE : SELinux can PREVENT you from ssh'ing into your docker container.
+As a brute force way to disable it - remove it from vi /etc/sysconfig/docker arguments
+(fedora and centos may by default launch docker daemon with the --selinux-enabled option)!
+In the future, lets update this README with the RIGHT way to allow selinux without breaking
+ssh into a docker container!
+
+```
+service docker restart
+docker pull bigtop/seed:centos-6.4
+```
+
+Now, you can start your cluster:
+
+```
+cd bigtop/bigtop-deploy/vm/vagrant-puppet-docker
+./docker-hadoop.sh --build-image --create 3
+```
+In case of errors you can attempt running as root, or else, ping the mailing list.
+
+## USAGE
+
+1) Build up the base Docker image that supports Vagrant.
+
+```
+./docker-hadoop.sh --build-image
+```
+
+2) Create a Bigtop Hadoop cluster by given # of node. (will place a file called config.rb)
+
+```
+./docker-hadoop.sh --create 3
+```
+
+3) Destroy the cluster.
+
+```
+./docker-hadoop.sh --destroy
+```
+
+4) Update your cluster after doing configuration changes. (re-run puppet apply)
+
+```
+./docker-hadoop.sh --provision
+```
+
+5) Run Bigtop smoke tests
+
+```
+./docker-hadoop.sh --smoke-tests
+```
+
+6) Chain your operations with-in one command.
+
+```
+./docker-hadoop.sh --build-image --create 5 --smoke-tests --destroy
+```
+
+Commands will be executed by following order:
+
+```
+build-image => create 5 node cluster => destroy the cluster
+```
+
+7) Run hbase-test.sh to evaluate the deployment.
+
+```
+../utils/hbase-test.sh
+```
+
+8) See helper message:
+
+```
+./docker-hadoop.sh -h
+usage: docker-hadoop.sh [options]
+ -b, --build-image Build base Docker image for Bigtop Hadoop
+ (must be exectued at least once before creating cluster)
+ -c NUM_INSTANCES, --create=NUM_INSTANCES Create a docker based Bigtop Hadoop cluster
+ -p, --provision Deploy configuration changes
+ -s, --smoke-tests Run Bigtop smoke tests
+ -d, --destroy Destroy the cluster
+ -h, --help
+
+```
+
+##Configurations
+
+* There are several parameters can be configured in the vagrantconfig.yaml:
+
+1) Modify memory limit for Docker containers
+
+```
+docker:
+ memory_size: "2048"
+
+```
+
+2) If you're running Docker provisioner on OS X or Windows, you can customize the boot2docker VM settings
+
+```
+boot2docker:
+ memory_size: "4196"
+ number_cpus: "2"
+```
+
+3) Use different host ports mapping for web UIs
+
+```
+namenode_ui_port: "50070"
+yarn_ui_port: "8088"
+hbase_ui_port: "60010"
+
+```
+Note: If running on OS X or Windows, the boot2docker VM should be reloaded after ports changed
+
+
+##Configure Apache Hadoop ecosystem components
+* Choose the ecosystem you want to be deployed by modifying components in vagrantconfig.yaml
+
+```
+components: "hadoop,hbase,yarn,..."
+```
+
+By default, Apache Hadoop, YARN, and Apache HBase will be installed.
+See `bigtop-deploy/puppet/config/site.csv.example` for more details.
+
+##Notes
+
+* Users currently using vagrant 1.6+ is strongly recommanded to upgrade to 1.6.4+, otherwise you will encounter the [issue](https://github.com/mitchellh/vagrant/issues/3769) when installing plguins
http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet-docker/Vagrantfile
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/vagrant-puppet-docker/Vagrantfile b/bigtop-deploy/vm/vagrant-puppet-docker/Vagrantfile
new file mode 100755
index 0000000..3ec023b
--- /dev/null
+++ b/bigtop-deploy/vm/vagrant-puppet-docker/Vagrantfile
@@ -0,0 +1,74 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+require 'fileutils'
+
+# number of instances
+$num_instances = 3
+
+# load configuration from config.rb
+CONFIG = File.join(File.dirname(__FILE__), "config.rb")
+if File.exist?(CONFIG)
+ require CONFIG
+end
+
+require "yaml"
+
+_config = YAML.load(File.open(File.join(File.dirname(__FILE__), "vagrantconfig.yaml"), File::RDONLY).read)
+CONF = _config
+
+ENV['VAGRANT_DEFAULT_PROVIDER'] ||= 'docker'
+
+Vagrant.require_version ">= 1.6.0"
+
+VAGRANTFILE_API_VERSION = "2"
+Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
+
+ # image definition
+ config.vm.define "image", autostart: false do |image|
+ image.vm.provider "docker" do |d|
+ d.build_dir = "."
+ d.build_args = ["-t",CONF["docker"]["image"]]
+ d.vagrant_machine = "boot2docker"
+ d.vagrant_vagrantfile = "./boot2docker/Vagrantfile"
+ end
+ image.ssh.username = "root"
+ image.ssh.port = 22
+ end
+
+ # nodes definition
+ (1..$num_instances).each do |i|
+ config.vm.define "bigtop#{i}" do |bigtop|
+ # docker container settings
+ bigtop.vm.provider "docker" do |d|
+ d.image = CONF["docker"]["image"]
+ d.create_args = ["--privileged=true", "-m", CONF["docker"]['memory_size'] + "m"]
+ if "#{i}" == "1"
+ d.ports = [CONF["namenode_ui_port"] + ":50070", CONF["yarn_ui_port"] + ":8088", CONF["hbase_ui_port"] + ":60010"]
+ end
+ d.vagrant_machine = "boot2docker"
+ d.vagrant_vagrantfile = "./boot2docker/Vagrantfile"
+ end
+ bigtop.ssh.username = "root"
+ bigtop.ssh.port = 22
+ bigtop.vm.hostname = "bigtop#{i}.docker"
+ bigtop.vm.synced_folder "../../../", "/bigtop-home"
+ end
+ end
+
+end
http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet-docker/boot2docker/Vagrantfile
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/vagrant-puppet-docker/boot2docker/Vagrantfile b/bigtop-deploy/vm/vagrant-puppet-docker/boot2docker/Vagrantfile
new file mode 100644
index 0000000..5913fec
--- /dev/null
+++ b/bigtop-deploy/vm/vagrant-puppet-docker/boot2docker/Vagrantfile
@@ -0,0 +1,24 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+require "yaml"
+
+_config = YAML.load(File.open(File.join(File.dirname(__FILE__), "../vagrantconfig.yaml"), File::RDONLY).read)
+CONF = _config
+
+VAGRANTFILE_API_VERSION = "2"
+
+Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
+
+ config.vm.define "boot2docker" do |node|
+ node.vm.box = "yungsang/boot2docker"
+ node.vm.provider "virtualbox" do |vb|
+ vb.customize ["modifyvm", :id, "--memory", CONF["boot2docker"]["memory_size"]]
+ vb.customize ['modifyvm', :id, '--cpus', CONF["boot2docker"]['number_cpus']]
+ end
+ node.vm.network "forwarded_port", host: CONF["namenode_ui_port"], guest: 50070
+ node.vm.network "forwarded_port", host: CONF["yarn_ui_port"], guest: 8088
+ node.vm.network "forwarded_port", host: CONF["hbase_ui_port"], guest: 60010
+ end
+
+end
http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet-docker/docker-hadoop.sh
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/vagrant-puppet-docker/docker-hadoop.sh b/bigtop-deploy/vm/vagrant-puppet-docker/docker-hadoop.sh
new file mode 100755
index 0000000..4b504b3
--- /dev/null
+++ b/bigtop-deploy/vm/vagrant-puppet-docker/docker-hadoop.sh
@@ -0,0 +1,117 @@
+#!/bin/bash
+
+usage() {
+ echo "usage: $PROG [options]"
+ echo " -b, --build-image Build base Docker image for Bigtop Hadoop"
+ echo " (must be exectued at least once before creating cluster)"
+ echo " -c NUM_INSTANCES, --create=NUM_INSTANCES Create a Docker based Bigtop Hadoop cluster"
+ echo " -p, --provision Deploy configuration changes"
+ echo " -s, --smoke-tests Run Bigtop smoke tests"
+ echo " -d, --destroy Destroy the cluster"
+ echo " -h, --help"
+ exit 1
+}
+
+build-image() {
+ vagrant up image --provider docker
+ {
+ echo "echo -e '\nBUILD IMAGE SUCCESS.\n'" |vagrant ssh image
+ } || {
+ >&2 echo -e "\nBUILD IMAGE FAILED!\n"
+ exit 2
+ }
+}
+
+create() {
+ echo "\$num_instances = $1" > config.rb
+ vagrant up --no-parallel
+ nodes=(`vagrant status |grep running |grep -v image |awk '{print $1}'`)
+ hadoop_head_node=(`echo "hostname -f" |vagrant ssh ${nodes[0]} |tail -n 1`)
+ repo=$(get-yaml-config repo)
+ components=$(get-yaml-config components)
+ echo "/bigtop-home/bigtop-deploy/vm/utils/setup-env.sh" |vagrant ssh ${nodes[0]}
+ echo "/vagrant/provision.sh $hadoop_head_node $repo $components" |vagrant ssh ${nodes[0]}
+ bigtop-puppet ${nodes[0]}
+ for ((i=1 ; i<${#nodes[*]} ; i++)); do
+ (
+ echo "/bigtop-home/bigtop-deploy/vm/utils/setup-env.sh" |vagrant ssh ${nodes[$i]}
+ echo "/vagrant/provision.sh $hadoop_head_node $repo $components" |vagrant ssh ${nodes[$i]}
+ bigtop-puppet ${nodes[$i]}
+ ) &
+ done
+ wait
+}
+
+provision() {
+ nodes=(`vagrant status |grep running |grep -v image |awk '{print $1}'`)
+ for node in $nodes; do
+ bigtop-puppet $node &
+ done
+ wait
+}
+
+smoke-tests() {
+ nodes=(`vagrant status |grep running |grep -v image |awk '{print $1}'`)
+ echo "/bigtop-home/bigtop-deploy/vm/utils/smoke-tests.sh" |vagrant ssh ${nodes[0]}
+}
+
+
+destroy() {
+ rm -rf ./hosts ./config ./config.rb
+ nodes=(`vagrant status |grep running |grep -v image |awk '{print $1}'`)
+ for node in $nodes; do
+ vagrant destroy -f $node
+ done
+ wait
+}
+
+bigtop-puppet() {
+ echo "puppet apply -d --confdir=/vagrant --modulepath=/bigtop-home/bigtop-deploy/puppet/modules:/etc/puppet/modules /bigtop-home/bigtop-deploy/puppet/manifests/site.pp" |vagrant ssh $1
+}
+
+get-yaml-config() {
+ RUBY_EXE=ruby
+ which ruby > /dev/null 2>&1
+ if [ $? -ne 0 ]; then
+ # use vagrant embedded ruby on Windows
+ RUBY_EXE=$(dirname $(which vagrant))/../embedded/bin/ruby
+ fi
+ RUBY_SCRIPT="data = YAML::load(STDIN.read); puts data['$1'];"
+ cat vagrantconfig.yaml | $RUBY_EXE -ryaml -e "$RUBY_SCRIPT" | tr -d '\r'
+}
+
+PROG=`basename $0`
+
+if [ $# -eq 0 ]; then
+ usage
+fi
+
+while [ $# -gt 0 ]; do
+ case "$1" in
+ -b|--build-image)
+ build-image
+ shift;;
+ -c|--create)
+ if [ $# -lt 2 ]; then
+ echo "Create requires a number" 1>&2
+ usage
+ fi
+ create $2
+ shift 2;;
+ -p|--provision)
+ provision
+ shift;;
+ -s|--smoke-tests)
+ smoke-tests
+ shift;;
+ -d|--destroy)
+ destroy
+ shift;;
+ -h|--help)
+ usage
+ shift;;
+ *)
+ echo "Unknown argument: '$1'" 1>&2
+ usage;;
+ esac
+done
http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet-docker/provision.sh
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/vagrant-puppet-docker/provision.sh b/bigtop-deploy/vm/vagrant-puppet-docker/provision.sh
new file mode 100755
index 0000000..da087c6
--- /dev/null
+++ b/bigtop-deploy/vm/vagrant-puppet-docker/provision.sh
@@ -0,0 +1,36 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+sysctl kernel.hostname=`hostname -f`
+
+# Unmount device /etc/hosts and replace it by a shared hosts file
+echo -e "`hostname -i`\t`hostname -f`" >> /vagrant/hosts
+umount /etc/hosts
+mv /etc/hosts /etc/hosts.bak
+ln -s /vagrant/hosts /etc/hosts
+
+# Prepare puppet configuration file
+cd /etc/puppet/modules && puppet module install puppetlabs/stdlib
+
+mkdir /vagrant/config
+cat > /vagrant/config/site.csv << EOF
+hadoop_head_node,$1
+hadoop_storage_dirs,/data/1,/data/2
+bigtop_yumrepo_uri,$2
+jdk_package_name,java-1.7.0-openjdk-devel.x86_64
+components,$3
+EOF
http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet-docker/vagrantconfig.yaml
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/vagrant-puppet-docker/vagrantconfig.yaml b/bigtop-deploy/vm/vagrant-puppet-docker/vagrantconfig.yaml
new file mode 100644
index 0000000..6a53d02
--- /dev/null
+++ b/bigtop-deploy/vm/vagrant-puppet-docker/vagrantconfig.yaml
@@ -0,0 +1,13 @@
+docker:
+ memory_size: "2048"
+ image: "bigtop/ssh:centos-6.4"
+
+boot2docker:
+ memory_size: "4196"
+ number_cpus: "2"
+
+repo: "http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-0.8.0/label=centos6/6/artifact/output/"
+components: "hadoop,yarn"
+namenode_ui_port: "50070"
+yarn_ui_port: "8088"
+hbase_ui_port: "60010"
http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet-vm/.gitignore
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/vagrant-puppet-vm/.gitignore b/bigtop-deploy/vm/vagrant-puppet-vm/.gitignore
new file mode 100644
index 0000000..04204c7
--- /dev/null
+++ b/bigtop-deploy/vm/vagrant-puppet-vm/.gitignore
@@ -0,0 +1 @@
+config
http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet-vm/README.md
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/vagrant-puppet-vm/README.md b/bigtop-deploy/vm/vagrant-puppet-vm/README.md
new file mode 100644
index 0000000..e1e3d48
--- /dev/null
+++ b/bigtop-deploy/vm/vagrant-puppet-vm/README.md
@@ -0,0 +1,92 @@
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+
+------------------------------------------------------------------------------------------------------------------------------------------------------
+
+#BigTop VM provisioner
+
+## Overview
+
+The Vagrantfile definition creates a bigtop virtual hadoop cluster for you, by pulling from existing publishing bigtop repositories.
+This cluster can be used:
+
+- to test bigtop smoke tests
+- to test bigtop puppet recipes
+
+Eventually, we may also add ability to build AND provision bigtop in a vagrant recipe, which would essentially
+give full validation of the BigTop stack.
+
+## USAGE
+
+1) Install [vagrant-hostmanager plugin](https://github.com/smdahlen/vagrant-hostmanager) to better manage `/etc/hosts`
+
+```
+vagrant plugin install vagrant-hostmanager
+```
+
+2) Install [vagrant-cachier plugin](https://github.com/fgrehm/vagrant-cachier) to cache packages at local
+
+```
+vagrant plugin install vagrant-cachier
+```
+
+3) To provision a 3 node Apache Hadoop cluster on top of vagrant boxes
+
+```
+vagrant up
+```
+
+4) You can specify number of nodes you'd like to provision by modifying `num_instances` in vagrantconfig.yaml
+
+```
+num_instances: 5
+```
+
+5) Test on local built packages is available by:
+
+first, build up local yum repo
+
+```
+cd bigtop; ./gradlew tachyon-yum
+```
+
+and then enable local yum in vagrantconfig.yaml
+
+```
+enable_local_yum: true
+```
+
+finally, spin up the cluster
+```
+vagrant up
+```
+
+##Configure Apache Hadoop ecosystem components
+* Choose the ecosystem you want to be deployed by modifying components in vagrantconfig.yaml
+
+```
+components: "hadoop,hbase,yarn,..."
+```
+
+By default, Apache Hadoop, YARN, and Apache HBase will be installed.
+See `bigtop-deploy/puppet/config/site.csv.example` for more details.
+
+##Example:
+
+5) Run hbase-test.sh to evaluate the deployment.
+
+```
+../utils/hbase-test.sh
+```
http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet-vm/Vagrantfile
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/vagrant-puppet-vm/Vagrantfile b/bigtop-deploy/vm/vagrant-puppet-vm/Vagrantfile
new file mode 100755
index 0000000..2f31055
--- /dev/null
+++ b/bigtop-deploy/vm/vagrant-puppet-vm/Vagrantfile
@@ -0,0 +1,123 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+require "yaml"
+
+_config = YAML.load(File.open(File.join(File.dirname(__FILE__), "vagrantconfig.yaml"), File::RDONLY).read)
+CONF = _config
+
+# Repository
+# Example for testing a Release candidate.
+repo = CONF['repo']
+# repo = "http://bigtop.s3.amazonaws.com/releases/0.7.0/redhat/6/x86_64"
+
+# number of instances
+num_instances = CONF['num_instances']
+
+# hadoop ecosystem components
+components = CONF['components']
+
+# Whether to run smoke tests
+run_smoke_tests = CONF['run_smoke_tests']
+
+# master node hostname
+bigtop_master = "bigtop1.vagrant"
+
+$script = <<SCRIPT
+service iptables stop
+chkconfig iptables off
+cat /dev/null > /etc/hosts
+echo "Bigtop yum repo = #{repo}"
+# Prepare puppet configuration file
+mkdir /vagrant/config
+cat > /vagrant/config/site.csv << EOF
+hadoop_head_node,#{bigtop_master}
+hadoop_storage_dirs,/data/1,/data/2
+bigtop_yumrepo_uri,#{repo}
+jdk_package_name,java-1.7.0-openjdk-devel.x86_64
+components,#{components}
+EOF
+SCRIPT
+
+VAGRANTFILE_API_VERSION = "2"
+Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
+ # manage /etc/hosts by hostmanager plugin(https://github.com/smdahlen/vagrant-hostmanager)
+ # use vagrant-cachier to cache packages at local(https://github.com/fgrehm/vagrant-cachier)
+ config.hostmanager.enabled = true
+
+ # use vagrant-cachier to cache packages at local(https://github.com/fgrehm/vagrant-cachier)
+ if Vagrant.has_plugin?("vagrant-cachier")
+ config.cache.scope = :box
+ end
+
+ # nodes definition
+ (1..num_instances).each do |i|
+ config.vm.define "bigtop#{i}" do |bigtop|
+
+ bigtop.vm.box = CONF['box']
+ bigtop_hostname="bigtop#{i}.vagrant"
+ bigtop_ip="10.10.10.1#{i}"
+
+ bigtop.vm.provider :virtualbox do |vb|
+ vb.customize ["modifyvm", :id, "--memory", CONF['memory_size']]
+ vb.customize ['modifyvm', :id, '--cpus', CONF['number_cpus']]
+ end
+
+ bigtop.vm.network :private_network, ip: bigtop_ip
+ bigtop.vm.hostname = bigtop_hostname
+
+ # three levels up is the bigtop "home" directory.
+ # the current directory has puppet recipes which we need for provisioning.
+ bigtop.vm.synced_folder "../../../", "/bigtop-home"
+
+ # We also add the bigtop-home output/ dir, so that locally built rpms will be available.
+ puts "Adding rpms from /output/ for local testing."
+
+ # This is a update to allow dev packages
+ # Force success - not worried if this step fails, since we generally only use it for development.
+ if CONF['enable_local_yum'] then
+ bigtop.vm.provision "shell", inline: "sudo echo \"gpgcheck=0\" >> /etc/yum.conf"
+ bigtop.vm.provision "shell", inline: "sudo yum-config-manager --add-repo file:///bigtop-home/output ; true"
+ end
+
+ # carry on w/ installation
+ bigtop.vm.provision "shell", path: "../utils/setup-env.sh"
+ bigtop.vm.provision "shell", inline: $script
+ bigtop.vm.provision :hostmanager
+
+ # deploy Apache Hadoop and it's ecosystem
+ bigtop.vm.provision :puppet do |puppet|
+ puppet.module_path = "../../puppet/modules/"
+ puppet.manifests_path = "../../puppet/manifests/"
+ puppet.manifest_file = "site.pp"
+ puppet.options = "--debug --confdir=/vagrant"
+ end
+
+ if run_smoke_tests then
+ if i==num_instances then
+ puts "creating provisioner directive for running tests"
+ bigtop.vm.provision "shell", path: "../utils/smoke-tests.sh"
+ else
+ puts "Not creating provisioner directive for tests yet... only on vm #{i} of #{num_instances}"
+ end
+ end
+
+ end
+ end
+
+end
http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet-vm/vagrantconfig.yaml
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/vagrant-puppet-vm/vagrantconfig.yaml b/bigtop-deploy/vm/vagrant-puppet-vm/vagrantconfig.yaml
new file mode 100644
index 0000000..7f79acc
--- /dev/null
+++ b/bigtop-deploy/vm/vagrant-puppet-vm/vagrantconfig.yaml
@@ -0,0 +1,8 @@
+memory_size: 4096
+number_cpus: 1
+box: "puppetlabs/centos-6.5-64-nocm"
+repo: "http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-0.8.0/label=centos6/6/artifact/output/"
+num_instances: 1
+components: "hadoop,yarn"
+enable_local_yum: false
+run_smoke_tests: false
http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet/.gitignore
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/vagrant-puppet/.gitignore b/bigtop-deploy/vm/vagrant-puppet/.gitignore
deleted file mode 100644
index 04204c7..0000000
--- a/bigtop-deploy/vm/vagrant-puppet/.gitignore
+++ /dev/null
@@ -1 +0,0 @@
-config
http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet/README.md
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/vagrant-puppet/README.md b/bigtop-deploy/vm/vagrant-puppet/README.md
deleted file mode 100644
index e1e3d48..0000000
--- a/bigtop-deploy/vm/vagrant-puppet/README.md
+++ /dev/null
@@ -1,92 +0,0 @@
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
-
-------------------------------------------------------------------------------------------------------------------------------------------------------
-
-#BigTop VM provisioner
-
-## Overview
-
-The Vagrantfile definition creates a bigtop virtual hadoop cluster for you, by pulling from existing publishing bigtop repositories.
-This cluster can be used:
-
-- to test bigtop smoke tests
-- to test bigtop puppet recipes
-
-Eventually, we may also add ability to build AND provision bigtop in a vagrant recipe, which would essentially
-give full validation of the BigTop stack.
-
-## USAGE
-
-1) Install [vagrant-hostmanager plugin](https://github.com/smdahlen/vagrant-hostmanager) to better manage `/etc/hosts`
-
-```
-vagrant plugin install vagrant-hostmanager
-```
-
-2) Install [vagrant-cachier plugin](https://github.com/fgrehm/vagrant-cachier) to cache packages at local
-
-```
-vagrant plugin install vagrant-cachier
-```
-
-3) To provision a 3 node Apache Hadoop cluster on top of vagrant boxes
-
-```
-vagrant up
-```
-
-4) You can specify number of nodes you'd like to provision by modifying `num_instances` in vagrantconfig.yaml
-
-```
-num_instances: 5
-```
-
-5) Test on local built packages is available by:
-
-first, build up local yum repo
-
-```
-cd bigtop; ./gradlew tachyon-yum
-```
-
-and then enable local yum in vagrantconfig.yaml
-
-```
-enable_local_yum: true
-```
-
-finally, spin up the cluster
-```
-vagrant up
-```
-
-##Configure Apache Hadoop ecosystem components
-* Choose the ecosystem you want to be deployed by modifying components in vagrantconfig.yaml
-
-```
-components: "hadoop,hbase,yarn,..."
-```
-
-By default, Apache Hadoop, YARN, and Apache HBase will be installed.
-See `bigtop-deploy/puppet/config/site.csv.example` for more details.
-
-##Example:
-
-5) Run hbase-test.sh to evaluate the deployment.
-
-```
-../utils/hbase-test.sh
-```
http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet/Vagrantfile
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/vagrant-puppet/Vagrantfile b/bigtop-deploy/vm/vagrant-puppet/Vagrantfile
deleted file mode 100755
index 2f31055..0000000
--- a/bigtop-deploy/vm/vagrant-puppet/Vagrantfile
+++ /dev/null
@@ -1,123 +0,0 @@
-# -*- mode: ruby -*-
-# vi: set ft=ruby :
-
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-require "yaml"
-
-_config = YAML.load(File.open(File.join(File.dirname(__FILE__), "vagrantconfig.yaml"), File::RDONLY).read)
-CONF = _config
-
-# Repository
-# Example for testing a Release candidate.
-repo = CONF['repo']
-# repo = "http://bigtop.s3.amazonaws.com/releases/0.7.0/redhat/6/x86_64"
-
-# number of instances
-num_instances = CONF['num_instances']
-
-# hadoop ecosystem components
-components = CONF['components']
-
-# Whether to run smoke tests
-run_smoke_tests = CONF['run_smoke_tests']
-
-# master node hostname
-bigtop_master = "bigtop1.vagrant"
-
-$script = <<SCRIPT
-service iptables stop
-chkconfig iptables off
-cat /dev/null > /etc/hosts
-echo "Bigtop yum repo = #{repo}"
-# Prepare puppet configuration file
-mkdir /vagrant/config
-cat > /vagrant/config/site.csv << EOF
-hadoop_head_node,#{bigtop_master}
-hadoop_storage_dirs,/data/1,/data/2
-bigtop_yumrepo_uri,#{repo}
-jdk_package_name,java-1.7.0-openjdk-devel.x86_64
-components,#{components}
-EOF
-SCRIPT
-
-VAGRANTFILE_API_VERSION = "2"
-Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
- # manage /etc/hosts by hostmanager plugin(https://github.com/smdahlen/vagrant-hostmanager)
- # use vagrant-cachier to cache packages at local(https://github.com/fgrehm/vagrant-cachier)
- config.hostmanager.enabled = true
-
- # use vagrant-cachier to cache packages at local(https://github.com/fgrehm/vagrant-cachier)
- if Vagrant.has_plugin?("vagrant-cachier")
- config.cache.scope = :box
- end
-
- # nodes definition
- (1..num_instances).each do |i|
- config.vm.define "bigtop#{i}" do |bigtop|
-
- bigtop.vm.box = CONF['box']
- bigtop_hostname="bigtop#{i}.vagrant"
- bigtop_ip="10.10.10.1#{i}"
-
- bigtop.vm.provider :virtualbox do |vb|
- vb.customize ["modifyvm", :id, "--memory", CONF['memory_size']]
- vb.customize ['modifyvm', :id, '--cpus', CONF['number_cpus']]
- end
-
- bigtop.vm.network :private_network, ip: bigtop_ip
- bigtop.vm.hostname = bigtop_hostname
-
- # three levels up is the bigtop "home" directory.
- # the current directory has puppet recipes which we need for provisioning.
- bigtop.vm.synced_folder "../../../", "/bigtop-home"
-
- # We also add the bigtop-home output/ dir, so that locally built rpms will be available.
- puts "Adding rpms from /output/ for local testing."
-
- # This is a update to allow dev packages
- # Force success - not worried if this step fails, since we generally only use it for development.
- if CONF['enable_local_yum'] then
- bigtop.vm.provision "shell", inline: "sudo echo \"gpgcheck=0\" >> /etc/yum.conf"
- bigtop.vm.provision "shell", inline: "sudo yum-config-manager --add-repo file:///bigtop-home/output ; true"
- end
-
- # carry on w/ installation
- bigtop.vm.provision "shell", path: "../utils/setup-env.sh"
- bigtop.vm.provision "shell", inline: $script
- bigtop.vm.provision :hostmanager
-
- # deploy Apache Hadoop and it's ecosystem
- bigtop.vm.provision :puppet do |puppet|
- puppet.module_path = "../../puppet/modules/"
- puppet.manifests_path = "../../puppet/manifests/"
- puppet.manifest_file = "site.pp"
- puppet.options = "--debug --confdir=/vagrant"
- end
-
- if run_smoke_tests then
- if i==num_instances then
- puts "creating provisioner directive for running tests"
- bigtop.vm.provision "shell", path: "../utils/smoke-tests.sh"
- else
- puts "Not creating provisioner directive for tests yet... only on vm #{i} of #{num_instances}"
- end
- end
-
- end
- end
-
-end
http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet/vagrantconfig.yaml
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/vagrant-puppet/vagrantconfig.yaml b/bigtop-deploy/vm/vagrant-puppet/vagrantconfig.yaml
deleted file mode 100644
index 7f79acc..0000000
--- a/bigtop-deploy/vm/vagrant-puppet/vagrantconfig.yaml
+++ /dev/null
@@ -1,8 +0,0 @@
-memory_size: 4096
-number_cpus: 1
-box: "puppetlabs/centos-6.5-64-nocm"
-repo: "http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-0.8.0/label=centos6/6/artifact/output/"
-num_instances: 1
-components: "hadoop,yarn"
-enable_local_yum: false
-run_smoke_tests: false