You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bigtop.apache.org by "Stefan Gower (JIRA)" <ji...@apache.org> on 2016/03/14 15:39:33 UTC
[jira] [Commented] (BIGTOP-1666) Test and confirm vagrant puppet
recipes work on Windows VMs.
[ https://issues.apache.org/jira/browse/BIGTOP-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15193375#comment-15193375 ]
Stefan Gower commented on BIGTOP-1666:
--------------------------------------
This is regarding a Windows install of Bigtop.
I attempt to install Apache Bigtop 1.0 on Windows 7 machine failed.
Errors message reported failure occurred because puppet was not on the path. NOTE: I got the same error message when attempting a Bigtop
install on a Mac (El Capitan). Unless I find a similar issue logged, I will log a separate issue for the Mac. This comment here is, however, just for the attempted Windows 7 install.
STEPS I DID
Windows 7 machine. HP Pentium.
Turned on virtualization via the BIOS. By default it was turned off.
So turning it on is necessary.
Installed Vagrant.
Installed VirtualBox.
ssh is installed and is the path. Used ssh from a Git install.
Please note that other VMs (not Bigtop) were successfully created with Vagrant. 'Vagrant ssh' commands into these VMs were successful.
So vagrant itself using VirtualBox *is* working.
Per instructions on the README file, I installed the vagrant plugins
vagrant plugin install vagrant-hostmanager
vagrant plugin install vagrant-cachier
Modified vagrantconfig.yaml file to use Debian.
Added a few items to the components list.
See Vagrantconfig.yaml below...
Along with log files including the error regarding puppet not being on the PATH.
Vagrantconfig.yaml
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
memory_size: 8096
number_cpus: 1
box: "puppetlabs/debian-7.8-64-nocm"
repo: http://bigtop-repos.s3.amazonaws.com/releases/1.0.0/debian/8/x86_64
num_instances: 3
distro: debian
components: [hadoop,pig,mapreduce,yarn,hbase]
enable_local_repo: false
run_smoke_tests: true
smoke_test_components: [mapreduce, pig,yarn,hbase]
jdk: "java-1.7.0-openjdk-devel.x86_64"
----------------------------
Note: I made no changes to the Vagrantfile. It is just 'as is'.
VAGRANTFILE
# -*- mode: ruby -*-
# vi: set ft=ruby :
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
require "yaml"
_config = YAML.load(File.open(File.join(File.dirname(__FILE__), "vagrantconfig.yaml"), File::RDONLY).read)
CONF = _config
# Override vagrant configurations using environment variables
keys = CONF.keys
keys.each do |k|
if ENV[k.upcase] != nil then
puts "Overide from environment variable: " + k.upcase + " = " + ENV[k.upcase]
if /^\d+/.match(ENV[k.upcase])
CONF[k] = Integer(ENV[k.upcase])
else
CONF[k] = ENV[k.upcase]
end
end
end
# Repository
# Example for testing a Release candidate.
repo = CONF['repo']
# repo = "http://bigtop.s3.amazonaws.com/releases/0.7.0/redhat/6/x86_64"
# Which Linux Distribution to use. Right now only centos is tested
distro = CONF['distro']
# number of instances
num_instances = CONF['num_instances']
# hadoop ecosystem components
components = CONF['components']
# Whether to run smoke tests
run_smoke_tests = CONF['run_smoke_tests']
# Smoke test Components to run
smoke_test_components = CONF['smoke_test_components'].join(',')
# This is a update to allow dev packages
# Force success - not worried if this step fails, since we generally only use it for development.
enable_local_repo = CONF['enable_local_repo']
puts "vagrant conf local repo enabled: #{enable_local_repo}"
# JDK package name
jdk = CONF['jdk']
# master node hostname
bigtop_master = "bigtop1.vagrant"
$script = <<SCRIPT
service iptables stop
chkconfig iptables off
# Remove 127.0.0.1 entry since vagrant's hostname setting will map it to FQDN,
# which miss leads some daemons to bind on 127.0.0.1 instead of public or private IP
sed -i "/127.0.0.1/d" /etc/hosts
echo "Bigtop yum repo = #{repo}"
# Prepare host manipulation files needed in standalone box
cp /bigtop-home/bigtop-deploy/vm/vagrant-puppet-vm/config_hosts /etc/init.d
cp /bigtop-home/bigtop-deploy/vm/vagrant-puppet-vm/gen_hosts.sh /usr/bin
chkconfig --add config_hosts
# Prepare puppet configuration file
mkdir -p /etc/puppet/hieradata
cp /bigtop-home/bigtop-deploy/puppet/hiera.yaml /etc/puppet
cp -r /bigtop-home/bigtop-deploy/puppet/hieradata/bigtop/ /etc/puppet/hieradata/
cat > /etc/puppet/hieradata/site.yaml << EOF
bigtop::hadoop_head_node: #{bigtop_master}
hadoop::hadoop_storage_dirs: [/data/1, /data/2]
bigtop::bigtop_repo_uri: #{repo}
hadoop_cluster_node::cluster_components: #{components}
bigtop::jdk_package_name: #{jdk}
hadoop::common_hdfs::testonly_hdfs_sshkeys: "yes"
EOF
SCRIPT
VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
# manage /etc/hosts by hostmanager plugin(https://github.com/smdahlen/vagrant-hostmanager)
# use vagrant-cachier to cache packages at local(https://github.com/fgrehm/vagrant-cachier)
config.hostmanager.enabled = true
# use vagrant-cachier to cache packages at local(https://github.com/fgrehm/vagrant-cachier)
if Vagrant.has_plugin?("vagrant-cachier")
config.cache.scope = :box
end
# nodes definition
(1..num_instances).each do |i|
config.vm.define "bigtop#{i}" do |bigtop|
bigtop.vm.box = CONF['box']
bigtop_hostname="bigtop#{i}"
bigtop_fqdn="#{bigtop_hostname}.vagrant"
bigtop_ip="10.10.10.1#{i}"
bigtop.vm.provider :virtualbox do |vb|
vb.customize ["modifyvm", :id, "--memory", CONF['memory_size']]
vb.customize ['modifyvm', :id, '--cpus', CONF['number_cpus']]
end
bigtop.vm.network :private_network, ip: bigtop_ip
bigtop.vm.hostname = bigtop_fqdn
# three levels up is the bigtop "home" directory.
# the current directory has puppet recipes which we need for provisioning.
bigtop.vm.synced_folder "../../../", "/bigtop-home"
# We also add the bigtop-home output/ dir, so that locally built rpms will be available.
puts "Adding output/ repo ? #{enable_local_repo}"
# carry on w/ installation
bigtop.vm.provision :shell do |shell|
shell.path = "../utils/setup-env-" + distro + ".sh"
shell.args = ["#{enable_local_repo}"]
end
bigtop.vm.provision "shell", inline: $script
# Add the ip to FQDN and hostname mapping in /etc/hosts
bigtop.hostmanager.aliases = "#{bigtop_fqdn} #{bigtop_hostname}"
# deploy Apache Hadoop and it's ecosystem
bigtop.vm.provision :puppet do |puppet|
puppet.module_path = "../../puppet/modules/"
puppet.manifests_path = "../../puppet/manifests/"
puppet.manifest_file = "site.pp"
puppet.options = "--debug"
end
if run_smoke_tests then
if i==num_instances then
puts "creating provisioner directive for running tests"
bigtop.vm.provision :shell do |shell|
shell.path = "../utils/smoke-tests.sh"
shell.args = ["#{smoke_test_components}"]
end
else
puts "Not creating provisioner directive for tests yet... only on vm #{i} of #{num_instances}"
end
end
end
end
end
---------------------------------------------
Now comes the 'vagrant up' command.
This ends in a statement that puppet is not on the PATH,
and further provisioning stops.
C:\vagrants\bigtop\bigtop\bigtop-deploy\vm\vagrant-puppet-vm>vagrant up
vagrant conf local repo enabled: false
Adding output/ repo ? false
Not creating provisioner directive for tests yet... only on vm 1 of 3
Adding output/ repo ? false
Not creating provisioner directive for tests yet... only on vm 2 of 3
Adding output/ repo ? false
creating provisioner directive for running tests
Bringing machine 'bigtop1' up with 'virtualbox' provider...
Bringing machine 'bigtop2' up with 'virtualbox' provider...
Bringing machine 'bigtop3' up with 'virtualbox' provider...
==> bigtop1: Importing base box 'puppetlabs/debian-7.8-64-nocm'...
==> bigtop1: Matching MAC address for NAT networking...
==> bigtop1: Checking if box 'puppetlabs/debian-7.8-64-nocm' is up to date...
==> bigtop1: A newer version of the box 'puppetlabs/debian-7.8-64-nocm' is available! You currently
==> bigtop1: have version '1.0.3'. The latest is version '1.0.4'. Run
==> bigtop1: `vagrant box update` to update.
==> bigtop1: Setting the name of the VM: vagrant-puppet-vm_bigtop1_1457900258158_76732
==> bigtop1: Clearing any previously set network interfaces...
==> bigtop1: Preparing network interfaces based on configuration...
bigtop1: Adapter 1: nat
bigtop1: Adapter 2: hostonly
==> bigtop1: Forwarding ports...
bigtop1: 22 (guest) => 2222 (host) (adapter 1)
==> bigtop1: Running 'pre-boot' VM customizations...
==> bigtop1: Booting VM...
==> bigtop1: Waiting for machine to boot. This may take a few minutes...
bigtop1: SSH address: 127.0.0.1:2222
bigtop1: SSH username: vagrant
bigtop1: SSH auth method: private key
bigtop1:
bigtop1: Vagrant insecure key detected. Vagrant will automatically replace
bigtop1: this with a newly generated keypair for better security.
bigtop1:
bigtop1: Inserting generated public key within guest...
bigtop1: Removing insecure key from the guest if it's present...
bigtop1: Key inserted! Disconnecting and reconnecting using new SSH key...
==> bigtop1: Machine booted and ready!
==> bigtop1: Checking for guest additions in VM...
bigtop1: The guest additions on this VM do not match the installed version of
bigtop1: VirtualBox! In most cases this is fine, but in rare cases it can
bigtop1: prevent things such as shared folders from working properly. If you see
bigtop1: shared folder errors, please make sure the guest additions within the
bigtop1: virtual machine match the version of VirtualBox you have installed on
bigtop1: your host and reload your VM.
bigtop1:
bigtop1: Guest Additions Version: 4.3.22
bigtop1: VirtualBox Version: 5.0
==> bigtop1: Setting hostname...
==> bigtop1: Configuring and enabling network interfaces...
==> bigtop1: Mounting shared folders...
bigtop1: /vagrant => C:/vagrants/bigtop/bigtop/bigtop-deploy/vm/vagrant-puppet-vm
bigtop1: /bigtop-home => C:/vagrants/bigtop/bigtop
bigtop1: /tmp/vagrant-cache => C:/Users/Gowers/.vagrant.d/cache/puppetlabs/debian-7.8-64-nocm
bigtop1: /tmp/vagrant-puppet/modules-35b17fc3f410fee52b756fc9dfa62c23 => C:/vagrants/bigtop/bigtop/bigtop-
deploy/puppet/modules
bigtop1: /tmp/vagrant-puppet/manifests-f76206f1841b10444f0cea13160540f3 => C:/vagrants/bigtop/bigtop/bigto
p-deploy/puppet/manifests
==> bigtop1: Updating /etc/hosts file on active guest machines...
==> bigtop1: Configuring cache buckets...
==> bigtop1: Running provisioner: shell...
bigtop1: Running: C:/Users/Gowers/AppData/Local/Temp/vagrant-shell20160313-2708-u1a07h.sh
==> bigtop1: stdin: is not a tty
==> bigtop1: /bigtop-home/bigtop_toolchain/bin/puppetize.sh: line 16: $'\r': command not found
==> bigtop1: /bigtop-home/bigtop_toolchain/bin/puppetize.sh: line 18: $'\r': command not found
==> bigtop1: /bigtop-home/bigtop_toolchain/bin/puppetize.sh: line 33: syntax error near unexpected token `$'in
\r''
'=> bigtop1: /bigtop-home/bigtop_toolchain/bin/puppetize.sh: line 33: `case ${ID}-${VERSION_ID} in
==> bigtop1: Reading package lists...
==> bigtop1: Building dependency tree...
==> bigtop1: Reading state information...
==> bigtop1: The following NEW packages will be installed:
==> bigtop1: rng-tools
==> bigtop1: dpkg-preconfigure: unable to re-open stdin: No such file or directory
==> bigtop1: 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
==> bigtop1: Need to get 0 B/46.8 kB of archives.
==> bigtop1: After this operation, 209 kB of additional disk space will be used.
==> bigtop1: Selecting previously unselected package rng-tools.
==> bigtop1: (Reading database ...
==> bigtop1: 38656 files and directories currently installed.)
==> bigtop1: Unpacking rng-tools (from .../rng-tools_2-unofficial-mt.14-1_amd64.deb) ...
==> bigtop1: Processing triggers for man-db ...
==> bigtop1: Setting up rng-tools (2-unofficial-mt.14-1) ...
==> bigtop1: Starting Hardware RNG entropy gatherer daemon:
==> bigtop1: (Hardware RNG device inode not found)
==> bigtop1: /etc/init.d/rng-tools: Cannot find a hardware RNG device to use.
==> bigtop1: invoke-rc.d: initscript rng-tools, action "start" failed.
==> bigtop1: Starting Hardware RNG entropy gatherer daemon:
==> bigtop1: rngd.
==> bigtop1: local apt = false ; NOT Enabling local apt. Packages will be pulled from remote...
==> bigtop1: Configuring cache buckets...
==> bigtop1: Running provisioner: shell...
bigtop1: Running: inline script
==> bigtop1: stdin: is not a tty
==> bigtop1: iptables: unrecognized service
==> bigtop1: /tmp/vagrant-shell: line 2: chkconfig: command not found
==> bigtop1: Bigtop yum repo = http://bigtop-repos.s3.amazonaws.com/releases/1.0.0/debian/8/x86_64
==> bigtop1: /tmp/vagrant-shell: line 10: chkconfig: command not found
==> bigtop1: Configuring cache buckets...
==> bigtop1: Running provisioner: puppet...
The `puppet` binary appears not to be in the PATH of the guest. This
could be because the PATH is not properly setup or perhaps Puppet is not
installed on this guest. Puppet provisioning can not continue without
Puppet properly installed
----------
> Test and confirm vagrant puppet recipes work on Windows VMs.
> ------------------------------------------------------------
>
> Key: BIGTOP-1666
> URL: https://issues.apache.org/jira/browse/BIGTOP-1666
> Project: Bigtop
> Issue Type: Bug
> Components: vm
> Environment: windows
> Reporter: jay vyas
> Fix For: backlog
>
> Attachments: vagrant-log-1.txt
>
>
> I've heard reports that bigtop vagrant recipes aren't working in windows because of some text munging errors.
> If anyone wants to test this for us that would be great.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)