You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@bigtop.apache.org by co...@apache.org on 2014/04/08 22:12:12 UTC
git commit: BIGTOP-1178. Clusterize the puppetized vagrant deployer.
Repository: bigtop
Updated Branches:
refs/heads/master 7c716c644 -> 3298063c6
BIGTOP-1178. Clusterize the puppetized vagrant deployer.
Project: http://git-wip-us.apache.org/repos/asf/bigtop/repo
Commit: http://git-wip-us.apache.org/repos/asf/bigtop/commit/3298063c
Tree: http://git-wip-us.apache.org/repos/asf/bigtop/tree/3298063c
Diff: http://git-wip-us.apache.org/repos/asf/bigtop/diff/3298063c
Branch: refs/heads/master
Commit: 3298063c68ffb89f829f23f8c88f3270110749ec
Parents: 7c716c6
Author: Evans Ye <innovansye.gmail.com>
Authored: Tue Apr 8 13:11:46 2014 -0700
Committer: Konstantin Boudnik <co...@apache.org>
Committed: Tue Apr 8 13:11:46 2014 -0700
----------------------------------------------------------------------
bigtop-deploy/vm/vagrant-puppet/README | 16 ----
bigtop-deploy/vm/vagrant-puppet/README.md | 52 ++++++++++
bigtop-deploy/vm/vagrant-puppet/Vagrantfile | 105 ++++++++++++++++++++-
bigtop-deploy/vm/vagrant-puppet/hbase-test.sh | 17 +++-
bigtop-deploy/vm/vagrant-puppet/provision.sh | 21 +++++
bigtop-deploy/vm/vagrant-puppet/startup.sh | 40 ++++++++
6 files changed, 229 insertions(+), 22 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/bigtop/blob/3298063c/bigtop-deploy/vm/vagrant-puppet/README
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/vagrant-puppet/README b/bigtop-deploy/vm/vagrant-puppet/README
deleted file mode 100644
index 819ce0b..0000000
--- a/bigtop-deploy/vm/vagrant-puppet/README
+++ /dev/null
@@ -1,16 +0,0 @@
-Set up a vagrant box with bigtop ecosystem deployed by puppet
-
-Usage:
-
-1) Choose the ecosystem you want to be deployed by modify the components in provision.sh:
-
- components,hadoop,hbase,...
-
- By default, hadoop and hbase will be installed.
- See bigtop-deploy/puppet/config/site.csv.example for more details.
-
-
-2) vagrant up
-
-
-3) Run hbase-test.sh to evaluate the deployment.
http://git-wip-us.apache.org/repos/asf/bigtop/blob/3298063c/bigtop-deploy/vm/vagrant-puppet/README.md
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/vagrant-puppet/README.md b/bigtop-deploy/vm/vagrant-puppet/README.md
new file mode 100644
index 0000000..ce7fc8e
--- /dev/null
+++ b/bigtop-deploy/vm/vagrant-puppet/README.md
@@ -0,0 +1,52 @@
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+
+------------------------------------------------------------------------------------------------------------------------------------------------------
+
+#Deploy Apache Hadoop VM(s) on top of vagrant boxes using puppet
+
+##Preparation
+* You need to install [vagrant-hostmanager plugin](https://github.com/smdahlen/vagrant-hostmanager) to better manage `/etc/hosts`
+<pre>
+$ vagrant plugin install vagrant-hostmanager
+</pre>
+
+##Usage
+
+* To provision a 3 node Apache Hadoop cluster on top of vagrant boxes
+<pre>
+$ ./startup.sh --cluster
+</pre>
+
+* See options with -h specified
+<pre>
+$ ./startup.sh -h
+usage: startup.sh [options]
+ -s, --standalone deploy a standalone hadoop vm
+ -c, --cluster deploy a 3 node hadoop cluster
+ -h, --help
+</pre>
+
+* Run hbase-test.sh to evaluate the deployment.
+
+##Configure Apache Hadoop ecosystem components
+* Choose the ecosystem you want to be deployed by modify components in provision.sh:
+
+<pre>
+ components,hadoop,hbase,...
+</pre>
+
+ By default, Apache Hadoop and Apache HBase will be installed.
+ See `bigtop-deploy/puppet/config/site.csv.example` for more details.
http://git-wip-us.apache.org/repos/asf/bigtop/blob/3298063c/bigtop-deploy/vm/vagrant-puppet/Vagrantfile
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/vagrant-puppet/Vagrantfile b/bigtop-deploy/vm/vagrant-puppet/Vagrantfile
index d852253..b19c2fa 100755
--- a/bigtop-deploy/vm/vagrant-puppet/Vagrantfile
+++ b/bigtop-deploy/vm/vagrant-puppet/Vagrantfile
@@ -1,6 +1,21 @@
# -*- mode: ruby -*-
# vi: set ft=ruby :
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
vmbox = "puppetlab-centos-64-nocm"
@@ -8,11 +23,21 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
# Virtual Box implementation
vmboxurl = "http://puppet-vagrant-boxes.puppetlabs.com/centos-64-x64-vbox4210-nocm.box"
- hostname = "vagrant.bigtop1"
+ # Network environment settings
+ bigtop1_hostname = "bigtop1.vagrant"
+ bigtop1_ip = "10.10.10.12"
+ bigtop2_hostname = "bigtop2.vagrant"
+ bigtop2_ip = "10.10.10.13"
+ bigtop3_hostname = "bigtop3.vagrant"
+ bigtop3_ip = "10.10.10.14"
# The bigtop puppet recipes
bigtop_puppet = "../../puppet"
+ # manage /etc/hosts by hostmanager plugin(https://github.com/smdahlen/vagrant-hostmanager)
+ config.hostmanager.enabled = true
+
+
# First node in the cluster
config.vm.define :bigtop1 do |bigtop1|
@@ -23,18 +48,20 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
vb.customize ["modifyvm", :id, "--memory", "1024"]
end
- bigtop1.vm.network :private_network, ip: "10.10.10.12"
- bigtop1.vm.hostname = hostname
+ bigtop1.vm.network :private_network, ip: bigtop1_ip
+ bigtop1.vm.hostname = bigtop1_hostname
bigtop1.vm.synced_folder bigtop_puppet, "/bigtop-puppet"
# Shell provisioner to setup basic vm environment
bigtop1.vm.provision :shell do |shell|
shell.path = "provision.sh"
- shell.args = hostname
+ shell.args = bigtop1_hostname + " " + bigtop1_ip + " " + bigtop1_hostname + " " + bigtop1_ip
end
- # Deploy hadoop and it's ecosystem
+ bigtop1.vm.provision :hostmanager
+
+ # Deploy Apache Hadoop and it's ecosystem
bigtop1.vm.provision :puppet do |puppet|
puppet.module_path = bigtop_puppet + "/modules"
puppet.manifests_path = bigtop_puppet + "/manifests"
@@ -44,4 +71,72 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
end
+
+ # Second node in the cluster
+ config.vm.define :bigtop2 do |bigtop2|
+
+ bigtop2.vm.box = vmbox
+ bigtop2.vm.box_url = vmboxurl
+
+ bigtop2.vm.provider :virtualbox do |vb|
+ vb.customize ["modifyvm", :id, "--memory", "1024"]
+ end
+
+ bigtop2.vm.network :private_network, ip: bigtop2_ip
+ bigtop2.vm.hostname = bigtop2_hostname
+
+ bigtop2.vm.synced_folder bigtop_puppet, "/bigtop-puppet"
+
+ # Shell provisioner to setup basic vm environment
+ bigtop2.vm.provision :shell do |shell|
+ shell.path = "provision.sh"
+ shell.args = bigtop1_hostname + " " + bigtop1_ip + " " + bigtop2_hostname + " " + bigtop2_ip
+ end
+
+ bigtop2.vm.provision :hostmanager
+
+ # Deploy Apache Hadoop and it's ecosystem
+ bigtop2.vm.provision :puppet do |puppet|
+ puppet.module_path = bigtop_puppet + "/modules"
+ puppet.manifests_path = bigtop_puppet + "/manifests"
+ puppet.manifest_file = "site.pp"
+ puppet.options = "--debug --confdir=/bigtop-puppet"
+ end
+
+ end
+
+
+ # Third node in the cluster
+ config.vm.define :bigtop3 do |bigtop3|
+
+ bigtop3.vm.box = vmbox
+ bigtop3.vm.box_url = vmboxurl
+
+ bigtop3.vm.provider :virtualbox do |vb|
+ vb.customize ["modifyvm", :id, "--memory", "1024"]
+ end
+
+ bigtop3.vm.network :private_network, ip: bigtop3_ip
+ bigtop3.vm.hostname = bigtop3_hostname
+
+ bigtop3.vm.synced_folder bigtop_puppet, "/bigtop-puppet"
+
+ # Shell provisioner to setup basic vm environment
+ bigtop3.vm.provision :shell do |shell|
+ shell.path = "provision.sh"
+ shell.args = bigtop1_hostname + " " + bigtop1_ip + " " + bigtop3_hostname + " " + bigtop3_ip
+ end
+
+ bigtop3.vm.provision :hostmanager
+
+ # Deploy Apache Hadoop and it's ecosystem
+ bigtop3.vm.provision :puppet do |puppet|
+ puppet.module_path = bigtop_puppet + "/modules"
+ puppet.manifests_path = bigtop_puppet + "/manifests"
+ puppet.manifest_file = "site.pp"
+ puppet.options = "--debug --confdir=/bigtop-puppet"
+ end
+
+ end
+
end
http://git-wip-us.apache.org/repos/asf/bigtop/blob/3298063c/bigtop-deploy/vm/vagrant-puppet/hbase-test.sh
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/vagrant-puppet/hbase-test.sh b/bigtop-deploy/vm/vagrant-puppet/hbase-test.sh
index c4820b8..89b2b76 100755
--- a/bigtop-deploy/vm/vagrant-puppet/hbase-test.sh
+++ b/bigtop-deploy/vm/vagrant-puppet/hbase-test.sh
@@ -1,6 +1,21 @@
#!/bin/bash
-vagrant ssh -c "hbase shell <<EOF
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+vagrant ssh bigtop1 -c "hbase shell <<EOF
create 't1','cf1'
put 't1', 'row1', 'cf1:q1', 'value1'
scan 't1'
http://git-wip-us.apache.org/repos/asf/bigtop/blob/3298063c/bigtop-deploy/vm/vagrant-puppet/provision.sh
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/vagrant-puppet/provision.sh b/bigtop-deploy/vm/vagrant-puppet/provision.sh
index 6219470..92c58f3 100755
--- a/bigtop-deploy/vm/vagrant-puppet/provision.sh
+++ b/bigtop-deploy/vm/vagrant-puppet/provision.sh
@@ -1,7 +1,28 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
# Install puppet agent
yum -y install http://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm
yum -y install puppet-2.7.23-1.el6.noarch
+service iptables stop
+chkconfig iptables off
+cat /dev/null > /etc/hosts
+
# Prepare puppet configuration file
cat > /bigtop-puppet/config/site.csv << EOF
hadoop_head_node,$1
http://git-wip-us.apache.org/repos/asf/bigtop/blob/3298063c/bigtop-deploy/vm/vagrant-puppet/startup.sh
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/vagrant-puppet/startup.sh b/bigtop-deploy/vm/vagrant-puppet/startup.sh
new file mode 100644
index 0000000..34b52ad
--- /dev/null
+++ b/bigtop-deploy/vm/vagrant-puppet/startup.sh
@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+usage() {
+ echo "usage: `basename $0` [options]"
+ echo " -s, --standalone deploy a standalone hadoop VM"
+ echo " -c, --cluster deploy a 3 node hadoop cluster"
+ echo " -h, --help"
+ exit 1
+}
+
+case "$1" in
+-s|--standalone)
+ vagrant up bigtop1
+ shift;;
+-c|--cluster)
+ vagrant up --provision-with shell && vagrant provision --provision-with hostmanager,puppet
+ shift;;
+-h|--help)
+ usage
+ shift;;
+"")
+ usage
+ shift
+ break;;
+esac