You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@bigtop.apache.org by gu...@apache.org on 2022/08/10 08:56:27 UTC
[bigtop] branch master updated: BIGTOP-3774: Drop Docker Sandbox and Vagrant Provisioner (#978)
This is an automated email from the ASF dual-hosted git repository.
guyuqi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/bigtop.git
The following commit(s) were added to refs/heads/master by this push:
new 659fa1c5 BIGTOP-3774: Drop Docker Sandbox and Vagrant Provisioner (#978)
659fa1c5 is described below
commit 659fa1c591d9f67a24617950cf9169dbf2863a7f
Author: Evans Ye <ev...@apache.org>
AuthorDate: Wed Aug 10 16:56:22 2022 +0800
BIGTOP-3774: Drop Docker Sandbox and Vagrant Provisioner (#978)
* BIGTOP-3774: Drop Docker Sandbox and Vagrant Provisioner
* BIGTOP-3774: address yuqi's comment
---
MAINTAINERS.txt | 1 -
README.md | 4 +-
bigtop-deploy/puppet/README.md | 2 +-
bigtop-packages/src/common/hadoop/init-hcfs.json | 1 -
bigtop_toolchain/README.md | 9 -
bigtop_toolchain/manifests/deployment_tools.pp | 18 --
bigtop_toolchain/manifests/vagrant.pp | 43 ----
docker/sandbox/README.md | 104 ----------
docker/sandbox/build.sh | 232 ----------------------
docker/sandbox/docker-compose.yml | 23 ---
docker/sandbox/sandbox-env.sh | 34 ----
docker/sandbox/site.yaml.template.centos-6_hadoop | 19 --
docker/sandbox/site.yaml.template.debian-8_hadoop | 19 --
docker/sandbox/startup.sh | 48 -----
provisioner/utils/smoke-tests.sh | 3 +-
provisioner/vagrant/.gitignore | 1 -
provisioner/vagrant/README.md | 114 -----------
provisioner/vagrant/Vagrantfile | 162 ---------------
provisioner/vagrant/config_hosts | 45 -----
provisioner/vagrant/create-sandboxes.sh | 32 ---
provisioner/vagrant/gen_hosts.sh | 44 ----
provisioner/vagrant/vagrantconfig.yaml | 25 ---
src/site/xdoc/index.xml | 3 +-
23 files changed, 5 insertions(+), 981 deletions(-)
diff --git a/MAINTAINERS.txt b/MAINTAINERS.txt
index 7924dff5..21156c5f 100644
--- a/MAINTAINERS.txt
+++ b/MAINTAINERS.txt
@@ -24,6 +24,5 @@ sqoop: sean mackrory, youngwoo kim
tajo: yeongeon kim
test-artifacts and test-execution repos:
tez: evans ye, oflebbe
-vagrant recipes: jay vyas, evans ye
ycsb: youngwoo kim
zookeeper: sean mackrory, rvs
diff --git a/README.md b/README.md
index 3234f240..d4781755 100644
--- a/README.md
+++ b/README.md
@@ -26,7 +26,7 @@ The primary goal of Apache Bigtop is to build a community around the packaging a
Immediately Get Started with Deployment and Smoke Testing of BigTop
===================================================================
-The simplest way to get a feel for how bigtop works, is to just cd into `provisioner` and try out the recipes under vagrant or docker. Each one rapidly spins up, and runs the bigtop smoke tests on, a local bigtop based big data distribution. Once you get the gist, you can hack around with the recipes to learn how the puppet/rpm/smoke-tests all work together, going deeper into the components you are interested in as described below.
+The simplest way to get a feel for how bigtop works, is to just cd into `provisioner` and try out the docker recipes. It rapidly spins up, and runs the bigtop smoke tests on, a local bigtop based big data distribution. Once you get the gist, you can hack around with the recipes to learn how the puppet/rpm/smoke-tests all work together, going deeper into the components you are interested in as described below.
Quick overview of source code directories
=========================================
@@ -38,7 +38,7 @@ Quick overview of source code directories
* __test-artifacts__ : source for tests.
* __test-execution__ : maven pom drivers for running the integration tests found in test-artifacts.
* __bigtop-toolchain__ : puppet scripts for setting up an instance which can build Apache Bigtop, sets up utils like jdk/maven/protobufs/...
-* __provisioner__ : Vagrant and Docker Provisioner that automatically spin up Hadoop environment with one click.
+* __provisioner__ : Docker Provisioner that automatically spin up Hadoop environment with one click.
* __docker__ : Dockerfiles and Docker Sandbox build scripts.
Also, there is a new project underway, Apache Bigtop blueprints, which aims to create templates/examples that demonstrate/compare various Apache Hadoop ecosystem components with one another.
diff --git a/bigtop-deploy/puppet/README.md b/bigtop-deploy/puppet/README.md
index d30ddca9..451e1454 100644
--- a/bigtop-deploy/puppet/README.md
+++ b/bigtop-deploy/puppet/README.md
@@ -22,7 +22,7 @@ When running puppet apply, note that puppet's confdir is *underneath* the `--con
For example: If you have `site.csv` in `/etc/puppet/config`,
Then you should use `--confdir=/etc/puppet` , and puppet finds the config dir underneath.
-As an end to end example, you can follow the `vagrant-puppet-vm` recipes to see how to set up
+As an end to end example, you can follow the `provisioner/docker` recipes to see how to set up
a puppet managed bigtop hadoop installation. Those examples are gauranteed to work and
serve as a pedagogical round trip to the way bigtop integrates packaging, deployment, and
testing all into one package.
diff --git a/bigtop-packages/src/common/hadoop/init-hcfs.json b/bigtop-packages/src/common/hadoop/init-hcfs.json
index 0ae35f1d..40df592f 100644
--- a/bigtop-packages/src/common/hadoop/init-hcfs.json
+++ b/bigtop-packages/src/common/hadoop/init-hcfs.json
@@ -69,7 +69,6 @@
["/user","755","HCFS_SUPER_USER",null],
["/user/history","755","mapred","mapred"],
["/user/jenkins","777","jenkins",null],
- ["/user/vagrant","777","vagrant",null],
["/user/hive","777",null,null],
["/user/root","777","root",null],
["/user/hue","777","hue","hue"],
diff --git a/bigtop_toolchain/README.md b/bigtop_toolchain/README.md
index f243f8ef..2299ffc5 100644
--- a/bigtop_toolchain/README.md
+++ b/bigtop_toolchain/README.md
@@ -69,15 +69,6 @@ standalone with a command such as:
where <path_to_bigtop> is the cloned git repo.
-## Installation of Tools for Bigtop Deployment
-
-This is a separated set of manifests that helps to setup tools for Bigtop deployment.
-The usage is as below:
-
- puppet apply --modulepath=<path_to_bigtop> -e "include bigtop_toolchain::deployment_tools"
-
-By applying the snippet, Vagrant will be installed(the Docker installation will be added soon).
-
## Optional development tools
This isn't a part of fundamental toolchain recipes as we are trying to contain the size of CI and dev-
diff --git a/bigtop_toolchain/manifests/deployment_tools.pp b/bigtop_toolchain/manifests/deployment_tools.pp
deleted file mode 100644
index 41a75228..00000000
--- a/bigtop_toolchain/manifests/deployment_tools.pp
+++ /dev/null
@@ -1,18 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-class bigtop_toolchain::deployment_tools {
- include bigtop_toolchain::vagrant
-}
diff --git a/bigtop_toolchain/manifests/vagrant.pp b/bigtop_toolchain/manifests/vagrant.pp
deleted file mode 100644
index 93fd03a3..00000000
--- a/bigtop_toolchain/manifests/vagrant.pp
+++ /dev/null
@@ -1,43 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-class bigtop_toolchain::vagrant {
-
- $vagrant_version = '1.7.1_x86_64'
- $vagrant = "vagrant_${vagrant_version}"
-
- case $operatingsystem{
- /Ubuntu|Debian/: {
- exec {'download-vagrant':
- command => "/usr/bin/wget https://dl.bintray.com/mitchellh/vagrant/${vagrant}.deb",
- cwd => '/usr/src',
- creates => "/usr/src/${vagrant}.deb"
- }
- package { 'vagrant':
- ensure => 'latest',
- source => "/usr/src/${vagrant}.deb",
- provider => 'dpkg',
- require => Exec['download-vagrant']
- }
- }
- default : {
- package { 'vagrant':
- ensure => 'latest',
- source => "https://dl.bintray.com/mitchellh/vagrant/${vagrant}.rpm",
- provider => 'rpm'
- }
- }
- }
-}
diff --git a/docker/sandbox/README.md b/docker/sandbox/README.md
deleted file mode 100644
index 0782b761..00000000
--- a/docker/sandbox/README.md
+++ /dev/null
@@ -1,104 +0,0 @@
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-
-# Bigtop Docker Sandbox
-
-A tool to build and run big data pseudo cluster using Docker.
-
-## How to run
-
-Make sure you have Docker installed. We've tested this using [Docker for Mac](https://docs.docker.com/docker-for-mac/)
-
-Currently supported OS list:
-
- * centos-6
- * debian-8
- * ubuntu-16.04
-
-### Run Hadoop HDFS
-
-```
-docker run -d -p 50070:50070 bigtop/sandbox:centos-6_hdfs
-```
-
-### Run Spark on YARN
-
-```
-docker run -d -p 50070:50070 -p 8088:8088 -p 8080:8080 bigtop/sandbox:debian-8_hdfs_yarn_spark
-```
-
-### Run Hadoop HDFS + HBase
-
-```
-docker run -d -p 50070:50070 -p 60010:60010 bigtop/sandbox:ubuntu-16.04_hdfs_hbase
-```
-
-## How to build
-
-### Build a Hadoop HDFS sandbox image
-
-```
-./build.sh -a bigtop -o centos-6 -c hdfs
-```
-
-### Build a Hadoop HDFS, Hadoop YARN, and Spark on YARN sandbox image
-
-```
-./build.sh -a bigtop -o debian-8 -c "hdfs, yarn, spark"
-```
-
-### Build a Hadoop HDFS and HBase sandbox image
-
-```
-./build.sh -a bigtop -o ubuntu-16.04 -c "hdfs, hbase"
-```
-
-### Use --dryrun to skip the build and get Dockerfile and configuration
-
-```
-./build.sh -a bigtop -o ubuntu-16.04 -c "hdfs, hbase" --dryrun
-```
-
-### Change the repository of packages
-
-* Change the repository to Bigtop's nightly centos-6 repo
-
-```
-export REPO=http://ci.bigtop.apache.org:8080/job/Bigtop-trunk-repos/BUILD_ENVIRONMENTS=centos-6%2Clabel=docker-slave-06//ws/output
-./build.sh -a bigtop -o centos-6 -c "hdfs, yarn, spark"
-```
-
-### Customize your Big Data Stack
-
-* Edit *site.yaml.template.centos-6_hadoop* to create your own prefered stack
-
-```
-cp site.yaml.template.centos-6_hadoop site.yaml.template.centos-6_hadoop_hive
-vim site.yaml.template.centos-6_hadoop_hive
-```
-
-* Build
-
-```
-./build.sh -a bigtop -o centos-6 -f site.yaml.template.centos-6_hadoop_hive -t my_hive_stack
-```
-
-## Known issues
-
-### Fail to start daemons using systemd
-
-Since systemd requires CAP_SYS_ADMIN, currently any OS using systemd can not successfully started up daemons during image build time.
-
-Daemons can be brought up only if --privileged specified using docker run command.
diff --git a/docker/sandbox/build.sh b/docker/sandbox/build.sh
deleted file mode 100755
index 2a5179d8..00000000
--- a/docker/sandbox/build.sh
+++ /dev/null
@@ -1,232 +0,0 @@
-#!/bin/bash
-
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-source sandbox-env.sh
-
-build() {
- # prepare puppet recipes
- cp -r ../../bigtop-deploy/puppet bigtop-puppet
-
- # docker build
- docker-compose build --force-rm --no-cache --pull
-
- if [ $? -eq 0 ]; then
- echo "-------------------------------------------------"
- echo "Image $ACCOUNT/sandbox:$TAG built"
- echo "-------------------------------------------------"
- fi
-}
-
-cleanup() {
- rm -rf bigtop-puppet site.yaml.template Dockerfile
-}
-
-generate_config() {
- cat > site.yaml.template << EOF
-bigtop::hadoop_head_node: ${HEAD_NODE:-"head.node.fqdn"}
-hadoop::hadoop_storage_dirs: [/data/1, /data/2]
-bigtop::bigtop_repo_uri: ${REPO}
-hadoop_cluster_node::cluster_components: [${COMPONENTS}]
-EOF
-}
-
-generate_dockerfile() {
- cat > Dockerfile << EOF
-FROM bigtop/puppet:${PREFIX}-${OS}${ARCH_SUFFIX}
-ADD startup.sh /startup.sh
-ADD bigtop-puppet /bigtop-puppet
-ADD bigtop-puppet/hiera.yaml /etc/puppet/hiera.yaml
-ADD bigtop-puppet/hieradata /etc/puppet/hieradata
-ADD site.yaml.template /etc/puppet/hieradata/site.yaml.template
-RUN /startup.sh --bootstrap
-CMD /startup.sh --foreground
-EOF
-}
-
-generate_tag() {
- if [ -z "$TAG" ]; then
- TAG="${PREFIX}-${OS}_`echo ${COMPONENTS//,/_} | tr -d ' '`"
- fi
-}
-
-detect_repo() {
- OS_SEP_BY_SLASH=${OS/-//}
- REPO=${REPO:-"http://repos.bigtop.apache.org/releases/${PREFIX}/${OS_SEP_BY_SLASH}/$ARCH_FOR_PKG"}
-}
-
-image_config_validator() {
- invalid=1
- if [ -z "$ACCOUNT" ]; then
- echo "account unset!"
- invalid=0
- fi
- if [ -z "$OS" ]; then
- echo "operating system unset!"
- invalid=0
- fi
- if [ -z "$PREFIX" ]; then
- echo "prefix unset!"
- invalid=0
- fi
- if [ -z "$TAG" ]; then
- echo "tag unset!"
- invalid=0
- fi
- if [ $invalid -eq 0 ]; then
- usage
- fi
-}
-
-deploy_config_validator() {
- invalid=1
- if [ -z "$REPO" ]; then
- echo "repository unset!"
- invalid=0
- fi
- if [ -z "$COMPONENTS" ]; then
- echo "components unset!"
- invalid=0
- fi
- if [ $invalid -eq 0 ]; then
- usage
- fi
-}
-
-show_image_configs() {
- echo "-------------------------------------------------"
- echo "IMAGE CONFIGS:"
- echo "ACCOUNT $ACCOUNT"
- echo "PREFIX $PREFIX"
- echo "OS $OS"
- echo "TAG $TAG"
- echo "IMAGE $ACCOUNT/sandbox:$TAG"
- echo "-------------------------------------------------"
-}
-
-show_deploy_configs() {
- echo "-------------------------------------------------"
- echo "DEPLOY CONFIGS:"
- echo "REPOSITORY $REPO"
- echo "COMPONENTS $COMPONENTS"
- echo "-------------------------------------------------"
-}
-
-log() {
- echo -e "\n[LOG] $1\n"
-}
-
-usage() {
- echo "usage: $PROG args"
- echo " -a, --account Specify account name for image."
- echo " -c, --components Specify components to build."
- echo " You need to specify a comma separated, quoted string."
- echo " For example: --components \"hadoop, yarn\""
- echo " -d, --dryrun Generate the Dockerfile and configuration and skip the build."
- echo " -f, --file Specify a written site.yaml config file."
- echo " -o, --operating-system Specify an OS from Bigtop supported OS list."
- echo " RPM base: ${RPMS[*]}"
- echo " DEB base: ${DEBS[*]}"
- echo " -p, --prefix Specify prefix for image."
- echo " Example: trunk|1.2.0|1.2.1|1.3.0|1.4.0|..."
- echo " -t, --tag Specify tag for image."
- echo " If not specified, this will be auto filled by specified OS and components."
- exit 1
-}
-
-while [ $# -gt 0 ]; do
- case "$1" in
- -a|--account)
- if [ $# -lt 2 ]; then
- usage
- fi
- ACCOUNT=$2
- shift 2;;
- -d|--dryrun)
- DRYRUN=true
- shift;;
- -c|--components)
- if [ $# -lt 2 ]; then
- usage
- fi
- COMPONENTS=$2
- shift 2;;
- -f|--file)
- if [ $# -lt 2 ]; then
- usage
- fi
- FILE=$2
- shift 2;;
- -o|--operating-system)
- if [ $# -lt 2 ]; then
- usage
- fi
- OS=$2
- running_arch=$(uname -m)
- if [ "x86_64" == "${running_arch}" ]; then
- ARCH_FOR_PKG="amd64"
- else
- ARCH_FOR_PKG="arm64"
- ARCH_SUFFIX="-${running_arch}"
- fi
- shift 2;;
- -p|--prefix)
- if [ $# -lt 2 ]; then
- usage
- fi
- PREFIX=$2
- shift 2;;
- -t|--tag)
- if [ $# -lt 2 ]; then
- usage
- fi
- TAG=$2
- shift 2;;
- *)
- echo "Unknown argument: '$1'" 1>&2
- usage;;
- esac
-done
-
-cleanup
-generate_tag
-image_config_validator
-show_image_configs
-
-if [ -z "$FILE" ]; then
- detect_repo
- deploy_config_validator
- generate_config
- show_deploy_configs
-else
- if [ -f "$FILE" ]; then
- cp -vfr $FILE site.yaml.template
- else
- log "$FILE not exist!"
- exit 1
- fi
-fi
-
-export ACCOUNT
-export TAG
-generate_dockerfile
-if [ "$DRYRUN" == true ]; then
- log "Generated Dockerfile:"
- cat Dockerfile
- exit 0
-fi
-build
-cleanup
diff --git a/docker/sandbox/docker-compose.yml b/docker/sandbox/docker-compose.yml
deleted file mode 100644
index 0a4ff0ad..00000000
--- a/docker/sandbox/docker-compose.yml
+++ /dev/null
@@ -1,23 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-version: '2'
-
-services:
- sandbox:
- build:
- context: .
- dockerfile: Dockerfile
- image: ${ACCOUNT}/sandbox:${TAG}
diff --git a/docker/sandbox/sandbox-env.sh b/docker/sandbox/sandbox-env.sh
deleted file mode 100644
index bbb2021e..00000000
--- a/docker/sandbox/sandbox-env.sh
+++ /dev/null
@@ -1,34 +0,0 @@
-#!/bin/bash
-
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-PREFIX=1.4.0
-# REPO="YOUR CUSTOM REPO"
-
-RPMS=( \
- centos-6 \
- centos-7 \
- centos-7-aarch64 \
- fedora-25 \
- fedora-25-aarch64 \
- opensuse-42.1 \
-)
-DEBS=( \
- debian-8 \
- debian-8-aarch64 \
- debian-9-aarch64 \
- ubuntu-16.04 \
-)
diff --git a/docker/sandbox/site.yaml.template.centos-6_hadoop b/docker/sandbox/site.yaml.template.centos-6_hadoop
deleted file mode 100644
index 6e8caf2a..00000000
--- a/docker/sandbox/site.yaml.template.centos-6_hadoop
+++ /dev/null
@@ -1,19 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-bigtop::hadoop_head_node: "head.node.fqdn"
-hadoop::hadoop_storage_dirs: [/data/1, /data/2]
-bigtop::bigtop_repo_uri: http://repos.bigtop.apache.org/releases/1.2.1/centos/6/x86_64
-hadoop_cluster_node::cluster_components: [hdfs, yarn, mapred-app]
diff --git a/docker/sandbox/site.yaml.template.debian-8_hadoop b/docker/sandbox/site.yaml.template.debian-8_hadoop
deleted file mode 100644
index 3c32a69f..00000000
--- a/docker/sandbox/site.yaml.template.debian-8_hadoop
+++ /dev/null
@@ -1,19 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-bigtop::hadoop_head_node: "head.node.fqdn"
-hadoop::hadoop_storage_dirs: [/data/1, /data/2]
-bigtop::bigtop_repo_uri: http://repos.bigtop.apache.org/releases/1.2.1/debian/8/x86_64
-hadoop_cluster_node::cluster_components: [hdfs, yarn, mapred-app]
diff --git a/docker/sandbox/startup.sh b/docker/sandbox/startup.sh
deleted file mode 100755
index e183a48f..00000000
--- a/docker/sandbox/startup.sh
+++ /dev/null
@@ -1,48 +0,0 @@
-#!/bin/bash
-
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-init() {
- echo "`facter ipaddress` `facter fqdn`" >> /etc/hosts
- cp /etc/puppet/hieradata/site.yaml.template /etc/puppet/hieradata/site.yaml
- sed -i -e "s/head.node.fqdn/`facter fqdn`/g" /etc/puppet/hieradata/site.yaml
- if puppet --version | grep ^3 >/dev/null ; then
- future="--parser future"
- fi
- puppet apply $future --modulepath=/bigtop-puppet/modules:/etc/puppet/modules:/usr/share/puppet/modules /bigtop-puppet/manifests
-}
-
-usage() {
- echo "usage: $PROG args"
- echo " -f, --foreground Running foreground."
- echo " -b, --build Bootstrap the stack."
- exit 1
-}
-
-while [ $# -gt 0 ]; do
- case "$1" in
- -f|--foreground)
- init
- sleep infinity
- shift;;
- -b|--bootstrap)
- init
- shift;;
- *)
- echo "Unknown argument: '$1'" 1>&2
- usage;;
- esac
-done
diff --git a/provisioner/utils/smoke-tests.sh b/provisioner/utils/smoke-tests.sh
index e761d0a9..d4160c65 100755
--- a/provisioner/utils/smoke-tests.sh
+++ b/provisioner/utils/smoke-tests.sh
@@ -57,8 +57,7 @@ export ZOOKEEPER_HOME=${ZOOKEEPER_HOME:-/usr/lib/zookeeper}
echo -e "\n===== START TO RUN SMOKE TESTS: $SMOKE_TESTS =====\n"
-su -s /bin/bash $HCFS_USER -c "hadoop fs -mkdir -p /user/vagrant /user/root /user/yarn"
-su -s /bin/bash $HCFS_USER -c "hadoop fs -chmod 777 /user/vagrant"
+su -s /bin/bash $HCFS_USER -c "hadoop fs -mkdir -p /user/root /user/yarn"
su -s /bin/bash $HCFS_USER -c "hadoop fs -chmod 777 /user/root"
su -s /bin/bash $HCFS_USER -c "hadoop fs -chown yarn:yarn /user/yarn"
diff --git a/provisioner/vagrant/.gitignore b/provisioner/vagrant/.gitignore
deleted file mode 100644
index 04204c7c..00000000
--- a/provisioner/vagrant/.gitignore
+++ /dev/null
@@ -1 +0,0 @@
-config
diff --git a/provisioner/vagrant/README.md b/provisioner/vagrant/README.md
deleted file mode 100644
index d4a1ec33..00000000
--- a/provisioner/vagrant/README.md
+++ /dev/null
@@ -1,114 +0,0 @@
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
-
-------------------------------------------------------------------------------------------------------------------------------------------------------
-
-# BigTop VM provisioner
-
-## Overview
-
-The Vagrantfile definition creates a bigtop virtual hadoop cluster for you, by pulling from existing publishing bigtop repositories.
-This cluster can be used:
-
-- to test bigtop smoke tests
-- to test bigtop puppet recipes
-
-Eventually, we may also add ability to build AND provision bigtop in a vagrant recipe, which would essentially
-give full validation of the BigTop stack.
-
-## USAGE
-
-1) Install [Vagrant](https://www.vagrantup.com/downloads.html)(DO NOT install 1.8.5 because of a critical [bug](https://github.com/mitchellh/vagrant/issues/7631))
-
-2) Install [vagrant-hostmanager plugin](https://github.com/smdahlen/vagrant-hostmanager) to manage `/etc/hosts`
-
-```
-vagrant plugin install vagrant-hostmanager
-```
-
-3) (Optional) Install [vagrant-cachier plugin](https://github.com/fgrehm/vagrant-cachier) to cache packages at local
-
-```
-vagrant plugin install vagrant-cachier
-```
-
-4) To provision a one node Apache Hadoop cluster on top of vagrant boxes
-
-```
-vagrant up
-```
-
-5) You can specify number of nodes you'd like to provision by modifying `num_instances` in vagrantconfig.yaml
-
-```
-num_instances: 5
-```
-
-6) Test on local built packages is available by:
-
-first, build up local yum repo
-
-```
-cd bigtop; ./gradlew alluxio-yum
-```
-
-and then enable local yum in vagrantconfig.yaml
-
-```
-enable_local_repo: true
-```
-
-finally, spin up the cluster
-```
-vagrant up
-```
-
-## Override Vagrant configurations
-You can override vagrant configurations from environment variables:
-```
-$ export REPO=http://repo.example.com/path/to
-$ vagrant up
-```
-
-or
-
-```
-$ MEMORY_SIZE=8000 vagrant up
-```
-
-## Configure Apache Hadoop ecosystem components
-* Choose the ecosystem you want to be deployed by modifying components in vagrantconfig.yaml
-
-```
-components: "hadoop, hbase, yarn,..."
-```
-
-By default, Apache Hadoop and YARN will be installed.
-See `bigtop-deploy/puppet/hieradata/site.yaml` for more details.
-
-## Sandbox VM generation
-
-Vagrant is a useful tool for times when you would like to capture the state of the VM and
-publish it as a sandbox for others to use. We provide a wrapper script for this that works
-with VirtualBox. Here's how to run it:
-
-```
-./create-sandboxes.sh [temp work dir] [version]
-```
-If you don't specify the temp work dir the current directory is going to be used, the version
-by default is 1.0.
-
-Once the script is done running you will have vagrant-bigtop-v<version>.box and
-vagrant-bigtop-v<version>.ova VM images available to you under the temp work dir.
diff --git a/provisioner/vagrant/Vagrantfile b/provisioner/vagrant/Vagrantfile
deleted file mode 100755
index ae099631..00000000
--- a/provisioner/vagrant/Vagrantfile
+++ /dev/null
@@ -1,162 +0,0 @@
-# -*- mode: ruby -*-
-# vi: set ft=ruby :
-
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-require "yaml"
-
-_config = YAML.load(File.open(File.join(File.dirname(__FILE__), "vagrantconfig.yaml"), File::RDONLY).read)
-CONF = _config
-
-# Override vagrant configurations using environment variables
-keys = CONF.keys
-keys.each do |k|
- if ENV[k.upcase] != nil then
- puts "Overide from environment variable: " + k.upcase + " = " + ENV[k.upcase]
- if /^\d+/.match(ENV[k.upcase])
- CONF[k] = Integer(ENV[k.upcase])
- else
- CONF[k] = ENV[k.upcase]
- end
- end
-end
-
-# Repository
-# Example for testing a Release candidate.
-repo = CONF['repo']
-# repo = "http://bigtop.s3.amazonaws.com/releases/0.7.0/redhat/6/x86_64"
-
-# Which Linux Distribution to use. Right now only centos is tested
-distro = CONF['distro']
-
-# number of instances
-num_instances = CONF['num_instances']
-
-# hadoop ecosystem components
-components = CONF['components']
-
-# Whether to run smoke tests
-run_smoke_tests = CONF['run_smoke_tests']
-
-# Smoke test Components to run
-smoke_test_components = CONF['smoke_test_components'].join(',')
-
-# This is a update to allow dev packages
-# Force success - not worried if this step fails, since we generally only use it for development.
-enable_local_repo = CONF['enable_local_repo']
-puts "vagrant conf local repo enabled: #{enable_local_repo}"
-
-# master node hostname
-bigtop_master = "bigtop1.vagrant"
-
-$script = <<SCRIPT
-service iptables stop
-service firewalld stop
-chkconfig iptables off
-# Remove 127.0.0.1 entry since vagrant's hostname setting will map it to FQDN,
-# which miss leads some daemons to bind on 127.0.0.1 instead of public or private IP
-sed -i "/127.0.0.1/d" /etc/hosts
-echo "Bigtop yum repo = #{repo}"
-# Prepare host manipulation files needed in standalone box
-cp /bigtop-home/provisioner/vagrant/config_hosts /etc/init.d
-cp /bigtop-home/provisioner/vagrant/gen_hosts.sh /usr/bin
-chkconfig --add config_hosts
-# Prepare puppet configuration file
-mkdir -p /etc/puppet/hieradata
-cp /bigtop-home/bigtop-deploy/puppet/hiera.yaml /etc/puppet
-cp -r /bigtop-home/bigtop-deploy/puppet/hieradata/bigtop/ /etc/puppet/hieradata/
-cat > /etc/puppet/hieradata/site.yaml << EOF
-bigtop::hadoop_head_node: #{bigtop_master}
-hadoop::hadoop_storage_dirs: [/data/1, /data/2]
-bigtop::bigtop_repo_uri: #{repo}
-hadoop_cluster_node::cluster_components: #{components}
-hadoop::common_hdfs::testonly_hdfs_sshkeys: "yes"
-EOF
-SCRIPT
-
-VAGRANTFILE_API_VERSION = "2"
-Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
- # manage /etc/hosts by hostmanager plugin(https://github.com/smdahlen/vagrant-hostmanager)
- # use vagrant-cachier to cache packages at local(https://github.com/fgrehm/vagrant-cachier)
- config.hostmanager.enabled = true
-
- # use vagrant-cachier to cache packages at local(https://github.com/fgrehm/vagrant-cachier)
- if Vagrant.has_plugin?("vagrant-cachier")
- config.cache.scope = :box
- end
-
- # nodes definition
- (1..num_instances).each do |i|
- config.vm.define "bigtop#{i}" do |bigtop|
-
- bigtop.vm.box = CONF['box']
- bigtop_hostname="bigtop#{i}"
- bigtop_fqdn="#{bigtop_hostname}.vagrant"
- bigtop_ip="10.10.10.1#{i}"
-
- bigtop.vm.provider :virtualbox do |vb|
- vb.customize ["modifyvm", :id, "--memory", CONF['memory_size']]
- vb.customize ['modifyvm', :id, '--cpus', CONF['number_cpus']]
- end
-
- bigtop.vm.network :private_network, ip: bigtop_ip
- bigtop.vm.hostname = bigtop_fqdn
-
- # two levels up is the bigtop "home" directory.
- # the current directory has puppet recipes which we need for provisioning.
- bigtop.vm.synced_folder "../../", "/bigtop-home"
-
- # We also add the bigtop-home output/ dir, so that locally built rpms will be available.
- puts "Adding output/ repo ? #{enable_local_repo}"
-
- # Install puppet
- bigtop.vm.provision :shell do |shell|
- shell.path = "../../bigtop_toolchain/bin/puppetize.sh"
- end
-
- # carry on w/ installation
- bigtop.vm.provision :shell do |shell|
- shell.path = "../utils/setup-env-" + distro + ".sh"
- shell.args = ["#{enable_local_repo}"]
- end
- bigtop.vm.provision "shell", inline: $script
- # Add the ip to FQDN and hostname mapping in /etc/hosts
- bigtop.hostmanager.aliases = "#{bigtop_fqdn} #{bigtop_hostname}"
-
- # deploy Apache Hadoop and it's ecosystem
- bigtop.vm.provision :puppet do |puppet|
- puppet.module_path = "../../bigtop-deploy/puppet/modules/"
- puppet.manifests_path = "../../bigtop-deploy/puppet/manifests/"
- puppet.manifest_file = "."
- puppet.options = "--parser future"
- end
-
- if run_smoke_tests then
- if i==num_instances then
- puts "creating provisioner directive for running tests"
- bigtop.vm.provision :shell do |shell|
- shell.path = "../utils/smoke-tests.sh"
- shell.args = ["#{smoke_test_components}"]
- end
- else
- puts "Not creating provisioner directive for tests yet... only on vm #{i} of #{num_instances}"
- end
- end
-
- end
- end
-
-end
diff --git a/provisioner/vagrant/config_hosts b/provisioner/vagrant/config_hosts
deleted file mode 100755
index 07e1d5ff..00000000
--- a/provisioner/vagrant/config_hosts
+++ /dev/null
@@ -1,45 +0,0 @@
-#!/bin/bash
-
-# Hosts reconfiguration script
-# chkconfig: 345 70 1
-# description: config_hosts
-
-# Licensed to Hortonworks, Inc. under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. Hortonworks, Inc. licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-NAME="config_hosts"
-LOG="/var/log/config_hosts.log"
-
-source /etc/rc.d/init.d/functions
-
-case "$1" in
-start)
- printf "%-50s\n" "Starting $NAME..." | tee -a $LOG
- printf "%-50s\n" "Updating IP..." | tee -a $LOG
- bash /usr/bin/gen_hosts.sh
-;;
-
-stop)
- date >> $LOG
-;;
-
-restart)
-;;
-
-*)
- echo "Usage: $0 {start|stop|restart}"
- exit 1
-esac
diff --git a/provisioner/vagrant/create-sandboxes.sh b/provisioner/vagrant/create-sandboxes.sh
deleted file mode 100755
index e90e821c..00000000
--- a/provisioner/vagrant/create-sandboxes.sh
+++ /dev/null
@@ -1,32 +0,0 @@
-#!/bin/bash
-
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-set -ex
-WORKDIR=${1:-$0.work.dir}
-echo "Cleaning $WORKDIR"
-rm -rf $WORKDIR
-mkdir -p $WORKDIR
-VERSION=${2:-1.0}
-NAME="vagrant-bigtop-v$VERSION"
-# clean everything
-vagrant destroy -f
-vagrant up
-vagrant package --output $WORKDIR/${NAME}.box
-vagrant halt
-vmname=`VBoxManage list vms | cut -d \" -f 2 | grep vagrant_bigtop`
-VBoxManage export $vmname --output=$WORKDIR/${NAME}.ova --ovf20
-vagrant destroy -f
diff --git a/provisioner/vagrant/gen_hosts.sh b/provisioner/vagrant/gen_hosts.sh
deleted file mode 100755
index c629c85a..00000000
--- a/provisioner/vagrant/gen_hosts.sh
+++ /dev/null
@@ -1,44 +0,0 @@
-# Licensed to Hortonworks, Inc. under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. Hortonworks, Inc. licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-echo "# Do not remove the following line, or various programs" > /etc/hosts
-echo "# that require network functionality will fail." >> /etc/hosts
-echo "127.0.0.1 localhost.localdomain localhost" >> /etc/hosts
-
-function get_inet_iface(){
- route | grep default | awk '{print $8}'
-}
-
-function get_ip() {
- ip addr | grep 'inet ' | grep -E "($(get_inet_iface))" | awk '{ print $2 }' | awk -F'/' '{print $1}'
-}
-
-HOST=$(get_ip)
-NUM=5
-while [ -z "$HOST" ]; do
- HOST=$(get_ip)
- sleep 5
- NUM=$(($NUM-1))
- if [ $NUM -le 0 ]; then
- HOST="127.0.0.1"
- echo "Failed to update IP"
- break
- fi
-done
-echo "$HOST `hostname`" >> /etc/hosts
-
-echo 0 > /proc/sys/kernel/hung_task_timeout_secs
-ethtool -K $(get_inet_iface) tso off
diff --git a/provisioner/vagrant/vagrantconfig.yaml b/provisioner/vagrant/vagrantconfig.yaml
deleted file mode 100644
index a25690d2..00000000
--- a/provisioner/vagrant/vagrantconfig.yaml
+++ /dev/null
@@ -1,25 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-memory_size: 4096
-number_cpus: 1
-box: "puppetlabs/centos-7.2-64-nocm"
-repo: "http://repos.bigtop.apache.org/releases/1.3.0/centos/7/x86_64"
-num_instances: 1
-distro: centos
-components: [hdfs, yarn, mapreduce]
-enable_local_repo: false
-run_smoke_tests: false
-smoke_test_components: [hdfs, yarn, mapreduce]
diff --git a/src/site/xdoc/index.xml b/src/site/xdoc/index.xml
index 1e2d917d..5f966a2f 100644
--- a/src/site/xdoc/index.xml
+++ b/src/site/xdoc/index.xml
@@ -36,8 +36,7 @@
Bigtop provides an integrated smoke testing framework, alongside a suite of over 50 test files.
</td>
<td>
- Bigtop provides vagrant recipes, raw images, and (work-in-progress) docker recipes for deploying
- Hadoop from zero.
+ Bigtop provides docker recipes and raw images for deploying Hadoop from zero.
</td>
</tr>
</table>