You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by "Evans Ye (JIRA)" <ji...@apache.org> on 2017/05/06 10:24:04 UTC
[jira] [Created] (BIGTOP-2756) Get rid of Permission Denied when
destroying Docker Provisioner cluster
Evans Ye created BIGTOP-2756:
--------------------------------
Summary: Get rid of Permission Denied when destroying Docker Provisioner cluster
Key: BIGTOP-2756
URL: https://issues.apache.org/jira/browse/BIGTOP-2756
Project: Bigtop
Issue Type: Bug
Components: docker, provisioner
Affects Versions: 1.2.0
Reporter: Evans Ye
Assignee: Evans Ye
Priority: Minor
Currently operations agains ```provisioner/docker/config/hosts``` may result in *Permision Denied" if running docker provisioner as non-root user.
{code}
+ ./gradlew -Pconfig=config_ubuntu_xenial_Bigtop-trunk-deployments.yaml -Pnum_instances=1 docker-provisioner
:buildSrc:compileJava UP-TO-DATE
:buildSrc:compileGroovy UP-TO-DATE
:buildSrc:processResources UP-TO-DATE
:buildSrc:classes UP-TO-DATE
:buildSrc:jar UP-TO-DATE
:buildSrc:assemble UP-TO-DATE
:buildSrc:compileTestJava UP-TO-DATE
:buildSrc:compileTestGroovy UP-TO-DATE
:buildSrc:processTestResources UP-TO-DATE
:buildSrc:testClasses UP-TO-DATE
:buildSrc:test UP-TO-DATE
:buildSrc:check UP-TO-DATE
:buildSrc:build UP-TO-DATE
:docker-provisioner
Environment check...
Check docker:
Docker version 1.11.2, build b9f10c9/1.11.2
Check docker-compose:
docker-compose version 1.6.2, build 4d72027
Check ruby:
ruby 2.0.0p648 (2015-12-16) [x86_64-linux]
Creating and starting 1 ...
Pulling bigtop (bigtop/puppet:ubuntu-16.04)...
ubuntu-16.04: Pulling from bigtop/puppet
Digest: sha256:71d7918e5458527c35c82eabca36401d5a58d0ac3bf6fbc99d0fe303535d6de4
Status: Downloaded newer image for bigtop/puppet:ubuntu-16.04
[1A[2K
Creating and starting 1 ... done
[1BBigtop Puppet configurations are shared between instances, and can be modified under config/hieradata
‘../../bigtop-deploy/puppet/hieradata/site.yaml’ -> ‘./config/hieradata/site.yaml’
‘../../bigtop-deploy/puppet/hieradata/bigtop’ -> ‘./config/hieradata/bigtop’
‘../../bigtop-deploy/puppet/hieradata/bigtop/ha.yaml’ -> ‘./config/hieradata/bigtop/ha.yaml’
‘../../bigtop-deploy/puppet/hieradata/bigtop/noha.yaml’ -> ‘./config/hieradata/bigtop/noha.yaml’
‘../../bigtop-deploy/puppet/hieradata/bigtop/cluster.yaml’ -> ‘./config/hieradata/bigtop/cluster.yaml’
./docker-hadoop.sh: line 81: ./config/hosts: Permission denied
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
rng-tools
0 upgraded, 1 newly installed, 0 to remove and 4 not upgraded.
Need to get 21.9 kB of archives.
After this operation, 139 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu xenial/universe amd64 rng-tools amd64 5-0ubuntu3 [21.9 kB]
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (Can't locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.22.1 /usr/local/share/perl/5.22.1 /usr/lib/x86_64-linux-gnu/perl5/5.22 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.22 /usr/share/perl/5.22 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base .) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7, <> line 1.)
debconf: falling back to frontend: Teletype
dpkg-preconfigure: unable to re-open stdin:
Fetched 21.9 kB in 0s (89.8 kB/s)
Selecting previously unselected package rng-tools.
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 13889 files and directories currently installed.)
Preparing to unpack .../rng-tools_5-0ubuntu3_amd64.deb ...
Unpacking rng-tools (5-0ubuntu3) ...
Processing triggers for systemd (229-4ubuntu12) ...
Setting up rng-tools (5-0ubuntu3) ...
Trying to create /dev/hwrng device inode...
invoke-rc.d: policy-rc.d denied execution of start.
Processing triggers for systemd (229-4ubuntu12) ...
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
apt-transport-https
0 upgraded, 1 newly installed, 0 to remove and 4 not upgraded.
Need to get 26.0 kB of archives.
After this operation, 213 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 apt-transport-https amd64 1.2.15ubuntu0.2 [26.0 kB]
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (Can't locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.22.1 /usr/local/share/perl/5.22.1 /usr/lib/x86_64-linux-gnu/perl5/5.22 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.22 /usr/share/perl/5.22 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base .) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7, <> line 1.)
debconf: falling back to frontend: Teletype
dpkg-preconfigure: unable to re-open stdin:
Fetched 26.0 kB in 0s (107 kB/s)
Selecting previously unselected package apt-transport-https.
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 13904 files and directories currently installed.)
Preparing to unpack .../apt-transport-https_1.2.15ubuntu0.2_amd64.deb ...
Unpacking apt-transport-https (1.2.15ubuntu0.2) ...
Setting up apt-transport-https (1.2.15ubuntu0.2) ...
local apt = false ; NOT Enabling local apt. Packages will be pulled from remote...
[1;31mWarning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Hash. There is further documentation for validate_legacy function in the README.
(at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')[0m
[1;31mWarning: This method is deprecated, please use match expressions with Stdlib::Compat::Bool instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions.
(at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')[0m
[1;31mWarning: This method is deprecated, please use match expressions with Stdlib::Compat::Array instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions.
(at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')[0m
[1;31mWarning: This method is deprecated, please use the stdlib validate_legacy function, with Pattern[]. There is further documentation for validate_legacy function in the README.
(at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')[0m
[1;31mWarning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Bool. There is further documentation for validate_legacy function in the README.
(at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')[0m
[1;31mWarning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::String. There is further documentation for validate_legacy function in the README.
(at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')[0m
[mNotice: Compiled catalog for f1823aaec65e.bigtop.apache.org in environment production in 2.51 seconds[0m
[mNotice: /Stage[main]/Hadoop_cluster_node/User[testuser]/ensure: created[0m
[mNotice: /Stage[main]/Hadoop_cluster_node/User[jenkins]/ensure: created[0m
[mNotice: /Stage[main]/Hadoop::Namenode/Hadoop::Create_storage_dir[/data/2/namenode]/Exec[mkdir /data/2/namenode]/returns: executed successfully[0m
[mNotice: /Stage[main]/Hadoop::Namenode/Hadoop::Create_storage_dir[/data/1/namenode]/Exec[mkdir /data/1/namenode]/returns: executed successfully[0m
[mNotice: /Stage[main]/Hadoop::Mapred_app/Hadoop::Create_storage_dir[/data/2/mapred]/Exec[mkdir /data/2/mapred]/returns: executed successfully[0m
[mNotice: /Stage[main]/Hadoop::Mapred_app/Hadoop::Create_storage_dir[/data/1/mapred]/Exec[mkdir /data/1/mapred]/returns: executed successfully[0m
[mNotice: /Stage[main]/Main/Apt::Conf[disable_keys]/Apt::Setting[conf-disable_keys]/File[/etc/apt/apt.conf.d/50disable_keys]/ensure: created[0m
[mNotice: /Stage[main]/Hadoop_cluster_node/User[hudson]/ensure: created[0m
[mNotice: /Stage[main]/Apt/Apt::Setting[conf-update-stamp]/File[/etc/apt/apt.conf.d/15update-stamp]/ensure: defined content as '{md5}0962d70c4ec78bbfa6f3544ae0c41974'[0m
[mNotice: /Stage[main]/Main/Apt::Source[Bigtop]/Apt::Setting[list-Bigtop]/File[/etc/apt/sources.list.d/Bigtop.list]/ensure: created[0m
[mNotice: /Stage[main]/Apt/File[preferences]/ensure: created[0m
[mNotice: /Stage[main]/Apt::Update/Exec[apt_update]: Triggered 'refresh' from 1 events[0m
[mNotice: /Stage[main]/Spark::Master/Package[spark-master]/ensure: ensure changed 'purged' to 'latest'[0m
[mNotice: /Stage[main]/Spark::Common/Package[spark-datanucleus]/ensure: ensure changed 'purged' to 'latest'[0m
[mNotice: /Stage[main]/Spark::Worker/Package[spark-worker]/ensure: ensure changed 'purged' to 'latest'[0m
[mNotice: /Stage[main]/Spark::Common/File[/etc/spark/conf/spark-defaults.conf]/ensure: defined content as '{md5}a3a08db3bf31654e86b22a51928c23e7'[0m
[mNotice: /Stage[main]/Spark::Common/File[/etc/spark/conf/log4j.properties]/ensure: defined content as '{md5}89ef3bfce554862a78bf4f2e1dafd4f6'[0m
[mNotice: /Stage[main]/Main/Package[jdk]/ensure: ensure changed 'purged' to 'present'[0m
[mNotice: /Stage[main]/Hadoop::Resourcemanager/Package[hadoop-yarn-resourcemanager]/ensure: ensure changed 'purged' to 'latest'[0m
[mNotice: /Stage[main]/Hadoop::Namenode/Package[hadoop-hdfs-namenode]/ensure: ensure changed 'purged' to 'latest'[0m
[mNotice: /Stage[main]/Hadoop::Namenode/File[/etc/default/hadoop-hdfs-namenode]/content: content changed '{md5}0c3a4aff527c8fb16d826bf59c9bb985' to '{md5}8aaf7169ae9f4cf6863f2e8c96e8ecb8'[0m
[mNotice: /Stage[main]/Hadoop::Proxyserver/Package[hadoop-yarn-proxyserver]/ensure: ensure changed 'purged' to 'latest'[0m
[mNotice: /Stage[main]/Hadoop::Historyserver/Package[hadoop-mapreduce-historyserver]/ensure: ensure changed 'purged' to 'latest'[0m
[mNotice: /Stage[main]/Hadoop::Common/File[/etc/hadoop/conf/hadoop-env.sh]/content: content changed '{md5}829beb2d3c16fa22040afdc0fa277ba1' to '{md5}b4e97a45010e60de2c7d4ca3abecfdde'[0m
[mNotice: /Stage[main]/Hadoop::Mapred_app/File[/data/2/mapred]/owner: owner changed 'root' to 'yarn'[0m
[mNotice: /Stage[main]/Hadoop::Mapred_app/File[/data/2/mapred]/group: group changed 'root' to 'yarn'[0m
[mNotice: /Stage[main]/Hadoop::Client/Package[hadoop-doc]/ensure: ensure changed 'purged' to 'latest'[0m
[mNotice: /Stage[main]/Hadoop::Client/Package[libhdfs0-dev]/ensure: ensure changed 'purged' to 'latest'[0m
[mNotice: /Stage[main]/Hadoop::Client/Package[hadoop-hdfs-fuse]/ensure: ensure changed 'purged' to 'latest'[0m
[mNotice: /Stage[main]/Hadoop::Common_hdfs/File[/etc/hadoop/conf/core-site.xml]/content: content changed '{md5}315d2770c60bc9a0b3c7dc04e41fda76' to '{md5}285212e176c67db3db3670500de81713'[0m
[mNotice: /Stage[main]/Hadoop::Common_mapred_app/File[/etc/hadoop/conf/taskcontroller.cfg]/ensure: defined content as '{md5}fa9d342175e04ccf0fa1aba8291bedba'[0m
[mNotice: /Stage[main]/Hadoop::Common_yarn/File[/etc/hadoop/conf/container-executor.cfg]/content: content changed '{md5}8afd041c79a90945ebfdd10ccbc43d9d' to '{md5}6aa9a3efa8965dfee06966b8fcb22e19'[0m
[mNotice: /Stage[main]/Hadoop::Common_hdfs/File[/etc/hadoop/conf/hdfs-site.xml]/content: content changed '{md5}06f0cc5cad0018c16ef3dcb404efb74e' to '{md5}da8fee38e5068bb587ef909127118f01'[0m
[mNotice: /Stage[main]/Hadoop::Common_mapred_app/File[/etc/hadoop/conf/mapred-site.xml]/content: content changed '{md5}1b058ae210b2e2dafd3407b6bf7dad43' to '{md5}1ce1de27de34ef29d33d3d4ade6b0019'[0m
[mNotice: /Stage[main]/Hadoop::Common_yarn/File[/etc/hadoop/conf/yarn-site.xml]/content: content changed '{md5}f679a7750b76612f7468f61a7c6b75e7' to '{md5}cab241efcdeba5517e1555eb33204a5c'[0m
[mNotice: /Stage[main]/Hadoop::Proxyserver/Service[hadoop-yarn-proxyserver]/ensure: ensure changed 'stopped' to 'running'[0m
[mNotice: /Stage[main]/Hadoop::Resourcemanager/Service[hadoop-yarn-resourcemanager]/ensure: ensure changed 'stopped' to 'running'[0m
[mNotice: /Stage[main]/Hadoop::Historyserver/Service[hadoop-mapreduce-historyserver]/ensure: ensure changed 'stopped' to 'running'[0m
[mNotice: /Stage[main]/Hadoop::Nodemanager/Package[hadoop-yarn-nodemanager]/ensure: ensure changed 'purged' to 'latest'[0m
[mNotice: /Stage[main]/Hadoop::Nodemanager/Hadoop::Create_storage_dir[/data/1/yarn]/Exec[mkdir /data/1/yarn]/returns: executed successfully[0m
[mNotice: /Stage[main]/Hadoop::Nodemanager/Hadoop::Create_storage_dir[/data/2/yarn]/Exec[mkdir /data/2/yarn]/returns: executed successfully[0m
[mNotice: /Stage[main]/Hadoop::Nodemanager/File[/data/2/yarn]/owner: owner changed 'root' to 'yarn'[0m
[mNotice: /Stage[main]/Hadoop::Nodemanager/File[/data/2/yarn]/group: group changed 'root' to 'yarn'[0m
[mNotice: /Stage[main]/Hadoop::Nodemanager/File[/data/1/yarn]/owner: owner changed 'root' to 'yarn'[0m
[mNotice: /Stage[main]/Hadoop::Nodemanager/File[/data/1/yarn]/group: group changed 'root' to 'yarn'[0m
[mNotice: /Stage[main]/Hadoop::Nodemanager/Service[hadoop-yarn-nodemanager]/ensure: ensure changed 'stopped' to 'running'[0m
[mNotice: /Stage[main]/Hadoop::Namenode/File[/data/1/namenode]/owner: owner changed 'root' to 'hdfs'[0m
[mNotice: /Stage[main]/Hadoop::Namenode/File[/data/1/namenode]/group: group changed 'root' to 'hdfs'[0m
[mNotice: /Stage[main]/Hadoop::Namenode/File[/data/1/namenode]/mode: mode changed '0755' to '0700'[0m
[mNotice: /Stage[main]/Hadoop::Namenode/File[/data/2/namenode]/owner: owner changed 'root' to 'hdfs'[0m
[mNotice: /Stage[main]/Hadoop::Namenode/File[/data/2/namenode]/group: group changed 'root' to 'hdfs'[0m
[mNotice: /Stage[main]/Hadoop::Namenode/File[/data/2/namenode]/mode: mode changed '0755' to '0700'[0m
[mNotice: /Stage[main]/Hadoop::Namenode/Exec[namenode format]/returns: executed successfully[0m
[mNotice: /Stage[main]/Hadoop::Namenode/Service[hadoop-hdfs-namenode]/ensure: ensure changed 'stopped' to 'running'[0m
[mNotice: /Stage[main]/Hadoop::Datanode/Package[hadoop-hdfs-datanode]/ensure: ensure changed 'purged' to 'latest'[0m
[mNotice: /Stage[main]/Hadoop::Datanode/Hadoop::Create_storage_dir[/data/2/hdfs]/Exec[mkdir /data/2/hdfs]/returns: executed successfully[0m
[mNotice: /Stage[main]/Hadoop::Datanode/Hadoop::Create_storage_dir[/data/1/hdfs]/Exec[mkdir /data/1/hdfs]/returns: executed successfully[0m
[mNotice: /Stage[main]/Hadoop::Datanode/File[/data/1/hdfs]/owner: owner changed 'root' to 'hdfs'[0m
[mNotice: /Stage[main]/Hadoop::Datanode/File[/data/1/hdfs]/group: group changed 'root' to 'hdfs'[0m
[mNotice: /Stage[main]/Hadoop::Datanode/File[/data/2/hdfs]/owner: owner changed 'root' to 'hdfs'[0m
[mNotice: /Stage[main]/Hadoop::Datanode/File[/data/2/hdfs]/group: group changed 'root' to 'hdfs'[0m
[mNotice: /Stage[main]/Hadoop::Datanode/File[/etc/default/hadoop-hdfs-datanode]/content: content changed '{md5}0c3a4aff527c8fb16d826bf59c9bb985' to '{md5}8aaf7169ae9f4cf6863f2e8c96e8ecb8'[0m
[mNotice: /Stage[main]/Hadoop::Datanode/Service[hadoop-hdfs-datanode]/ensure: ensure changed 'stopped' to 'running'[0m
[mNotice: /Stage[main]/Hadoop::Init_hdfs/Exec[init hdfs]/returns: executed successfully[0m
[mNotice: /Stage[main]/Hadoop::Mapred_app/File[/data/1/mapred]/owner: owner changed 'root' to 'yarn'[0m
[mNotice: /Stage[main]/Hadoop::Mapred_app/File[/data/1/mapred]/group: group changed 'root' to 'yarn'[0m
[mNotice: /Stage[main]/Spark::Common/File[/etc/spark/conf/spark-env.sh]/content: content changed '{md5}0532a742483de44c845a66794a696e5e' to '{md5}5043b3e515ec7dd34ac4d0cb848aa2e6'[0m
[mNotice: /Stage[main]/Spark::Worker/Service[spark-worker]/ensure: ensure changed 'stopped' to 'running'[0m
[mNotice: /Stage[main]/Spark::Master/Service[spark-master]/ensure: ensure changed 'stopped' to 'running'[0m
[mNotice: Finished catalog run in 221.64 seconds[0m
BUILD SUCCESSFUL
Total time: 4 mins 46.916 secs
This build could be faster, please consider using the Gradle Daemon: https://docs.gradle.org/2.12/userguide/gradle_daemon.html
+ ./gradlew docker-provisioner-destroy
:buildSrc:compileJava UP-TO-DATE
:buildSrc:compileGroovy UP-TO-DATE
:buildSrc:processResources UP-TO-DATE
:buildSrc:classes UP-TO-DATE
:buildSrc:jar UP-TO-DATE
:buildSrc:assemble UP-TO-DATE
:buildSrc:compileTestJava UP-TO-DATE
:buildSrc:compileTestGroovy UP-TO-DATE
:buildSrc:processTestResources UP-TO-DATE
:buildSrc:testClasses UP-TO-DATE
:buildSrc:test UP-TO-DATE
:buildSrc:check UP-TO-DATE
:buildSrc:build UP-TO-DATE
:docker-provisioner-destroyThe DOCKER_IMAGE variable is not set. Defaulting to a blank string.
The MEM_LIMIT variable is not set. Defaulting to a blank string.
The DOCKER_IMAGE variable is not set. Defaulting to a blank string.
The MEM_LIMIT variable is not set. Defaulting to a blank string.
Stopping 20170506094608r8013_bigtop_1 ...
[1A[2K
Stopping 20170506094608r8013_bigtop_1 ... done
[1BThe DOCKER_IMAGE variable is not set. Defaulting to a blank string.
The MEM_LIMIT variable is not set. Defaulting to a blank string.
Removing 20170506094608r8013_bigtop_1 ...
[1A[2K
Removing 20170506094608r8013_bigtop_1 ... done
[1B
Going to remove 20170506094608r8013_bigtop_1
./docker-hadoop.sh: line 130: ./config/hosts: Permission denied
removed ‘./config/hieradata/bigtop/ha.yaml’
removed ‘./config/hieradata/bigtop/noha.yaml’
removed ‘./config/hieradata/bigtop/cluster.yaml’
removed directory: ‘./config/hieradata/bigtop’
removed ‘./config/hieradata/site.yaml’
removed ‘.provision_id’
BUILD SUCCESSFUL
Total time: 32.182 secs
{code}
The way to fix it is to operate the file inside container.
Meanwhile, wee need to unmount /etc/hosts and remove the file before container destroyed. Otherwise there will be a root:root file left in the repository which can not be remove by user w/o sudo permission.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)