You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@bigtop.apache.org by ev...@apache.org on 2019/01/07 09:06:17 UTC
[bigtop] branch master updated (0bb54e5 -> 676b292)
This is an automated email from the ASF dual-hosted git repository.
evansye pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/bigtop.git.
from 0bb54e5 BIGTOP-3118. [Puppet] Failed to deploy GPDB
new 471276e BIGTOP-3126. [Puppet] Failed to deploy QFS due to Permission denied error at initailization
new 676b292 BIGTOP-3124. Remove package installation in provisioner/utils/smoke-tests.sh and polish the script
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
bigtop-deploy/puppet/modules/qfs/manifests/init.pp | 6 ++--
provisioner/utils/smoke-tests.sh | 41 ++++++++++++----------
2 files changed, 27 insertions(+), 20 deletions(-)
[bigtop] 02/02: BIGTOP-3124. Remove package installation in
provisioner/utils/smoke-tests.sh and polish the script
Posted by ev...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
evansye pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/bigtop.git
commit 676b2925764cca429c783f9804c06b940352d921
Author: Evans Ye <ev...@apache.org>
AuthorDate: Sat Jan 5 02:35:23 2019 +0800
BIGTOP-3124. Remove package installation in provisioner/utils/smoke-tests.sh and polish the script
---
provisioner/utils/smoke-tests.sh | 41 ++++++++++++++++++++++------------------
1 file changed, 23 insertions(+), 18 deletions(-)
diff --git a/provisioner/utils/smoke-tests.sh b/provisioner/utils/smoke-tests.sh
index 503d3b8..1a60bf7 100755
--- a/provisioner/utils/smoke-tests.sh
+++ b/provisioner/utils/smoke-tests.sh
@@ -15,7 +15,12 @@
# limitations under the License.
HCFS_USER="hdfs"
-SMOKE_TESTS=${1:-mapreduce,pig}
+SMOKE_TESTS=$1
+
+if [ -z "$SMOKE_TESTS" ]; then
+ >&2 echo -e "\nSMOKE_TESTS VARIABLE IS NOT DEFINED. CHECK THE INPUT OF `basename $0` \n"
+ exit 2
+fi
# Autodetect JAVA_HOME
if [ -e /usr/lib/bigtop-utils/bigtop-detect-javahome ]; then
@@ -25,19 +30,24 @@ else
exit 2
fi
-echo -e "\n===== START TO RUN SMOKE TESTS: $SMOKE_TESTS =====\n"
+echo -e "\n===== EXPORTING VARIABLES =====\n"
-export HADOOP_CONF_DIR=/etc/hadoop/conf/
-export HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce/
-export HIVE_HOME=/usr/lib/hive/
-export PIG_HOME=/usr/lib/pig/
-export FLUME_HOME=/usr/lib/flume/
-export SQOOP_HOME=/usr/lib/sqoop/
-export HIVE_CONF_DIR=/etc/hive/conf/
-export MAHOUT_HOME="/usr/lib/mahout"
-export HBASE_HOME=/usr/lib/hbase
-export HBASE_CONF_DIR=/usr/lib/hbase/conf
-export ZOOKEEPER_HOME=/usr/lib/zookeeper
+export ALLUXIO_HOME=${ALLUXIO_HOME:-/usr/lib/alluxio}
+export AMBARI_URL=${AMBARI_URL:-http://localhost:8080}
+export FLUME_HOME=${FLUME_HOME:-/usr/lib/flume}
+export GPDB_HOME=${GPDB_HOME:-/usr/lib/gpdb}
+export HADOOP_MAPRED_HOME=${HADOOP_MAPRED_HOME:-/usr/lib/hadoop-mapreduce}
+export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/etc/hadoop/conf}
+export HBASE_HOME=${HBASE_HOME:-/usr/lib/hbase}
+export HBASE_CONF_DIR=${HBASE_CONF_DIR:-/usr/lib/hbase/conf}
+export HIVE_HOME=${HIVE_HOME:-/usr/lib/hive}
+export HIVE_CONF_DIR=${HIVE_CONF_DIR:-/etc/hive/conf}
+export MAHOUT_HOME=${MAHOUT_HOME:-/usr/lib/mahout}
+export SPARK_HOME=${SPARK_HOME:-/usr/lib/spark}
+export SQOOP_HOME=${SQOOP_HOME:-/usr/lib/sqoop}
+export ZOOKEEPER_HOME=${ZOOKEEPER_HOME:-/usr/lib/zookeeper}
+
+echo -e "\n===== START TO RUN SMOKE TESTS: $SMOKE_TESTS =====\n"
prep() {
HADOOP_COMMAND=$1
@@ -51,11 +61,6 @@ if [[ $SMOKE_TESTS == *"qfs"* ]]; then
prep hadoop-qfs
fi
-if [ -f /etc/debian_version ] ; then
- apt-get -y install pig hive flume mahout sqoop
-else
- yum install -y pig hive flume mahout sqoop
-fi
ALL_SMOKE_TASKS=""
for s in `echo $SMOKE_TESTS | sed -e 's#,# #g'`; do
ALL_SMOKE_TASKS="$ALL_SMOKE_TASKS bigtop-tests:smoke-tests:$s:test"
[bigtop] 01/02: BIGTOP-3126. [Puppet] Failed to deploy QFS due to
Permission denied error at initailization
Posted by ev...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
evansye pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/bigtop.git
commit 471276e1b743f143d7ef714e7cbc1ea83c2bc287
Author: Evans Ye <ev...@apache.org>
AuthorDate: Sat Jan 5 02:30:54 2019 +0800
BIGTOP-3126. [Puppet] Failed to deploy QFS due to Permission denied error at initailization
---
bigtop-deploy/puppet/modules/qfs/manifests/init.pp | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/bigtop-deploy/puppet/modules/qfs/manifests/init.pp b/bigtop-deploy/puppet/modules/qfs/manifests/init.pp
index 82de818..e0fdf54 100644
--- a/bigtop-deploy/puppet/modules/qfs/manifests/init.pp
+++ b/bigtop-deploy/puppet/modules/qfs/manifests/init.pp
@@ -36,8 +36,8 @@ class qfs {
hadoop::create_storage_dir { $qfs::common::storage_dirs: } ->
file { $qfs::common::storage_dirs:
ensure => directory,
- owner => root,
- group => root,
+ owner => qfs,
+ group => qfs,
mode => '0755',
}
}
@@ -74,6 +74,8 @@ class qfs {
exec { "mkfs":
command => "/usr/bin/metaserver -c $metaserver_conf",
creates => "${qfs::common::storage_dirs[0]}/metaserver/checkpoint/latest",
+ # BIGTOP-3126: workaround that the qfs init script requires to run under a permitted directory
+ cwd => "${qfs::common::storage_dirs[0]}",
user => qfs,
group => qfs,
require => [