You are viewing a plain text version of this content. The canonical link for it is here.
Posted to codereview@trafodion.apache.org by amandamoran <gi...@git.apache.org> on 2016/04/19 02:27:43 UTC
[GitHub] incubator-trafodion pull request: Vanilla Apache HBase support and...
GitHub user amandamoran opened a pull request:
https://github.com/apache/incubator-trafodion/pull/440
Vanilla Apache HBase support and Multi-Cluster support (disabled)
Pre-work needed from Hadoop Install
Install: Hadoop, Zookeeper, HBase, and optional Hive
hadoop-env.sh
set JAVA_HOME
hdfs-site.xml
dfs.namenode.acls.enabled true
hbase-site.xml
hbase.cluster.distributed true
hbase.coprocessor.region.classes org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionObserver, org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionEndpoint, org.apache.hadoop.hbase.coprocessor.AggregateImplementation
hbase.hregion.impl org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion
hbase-env.sh
export HBASE_MANAGES_ZK=false # don't use HBase's zookeeper
export JAVA_HOME=/path/to/java/home
zookeeper-env.sh
set JAVA_HOME
Trafodion Install
New Questions:
Specify the version of Hadoop installed (1: Cloudera, 2: Hortonworks, 3: Other):
Enter Hadoop installed full path (example: '/opt/hadoop-2.6.0'), default is [/opt/hadoop-2.6.0]:
Enter HBase installed full path (example: '/opt/hbase-1.2.0'), default is [/opt/hbase-1.0.2]:
Enter Zookeeper installed full path (example: '/opt/zoo'), default is [/opt/zookeeper-3.4.6]:
Is Hive Installed (Y/N), default is N:
Enter Hive installed full path, default is []:
Enter list of all nodes in this cluster (blank separated), default [vanilla-1 vanilla-2]:
Enter list of all HBase nodes in this cluster (blank separated), default [vanilla-1 vanilla-2]:
Enter list of all HDFS nodes in this cluster (blank separated), default [vanilla-1 vanilla-2]:
Old Questions (NEW ANSWERS):
Enter HDFS username or username running HDFS, default is [centos]:
Enter HBase username or username running HBase, default is [centos]:
Enter HBase group, default is [centos]:
Enter Zookeeper username of username running Zookeeper, default is [centos]:
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/amandamoran/incubator-trafodion installbugfixes
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-trafodion/pull/440.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #440
----
commit 5357a654658df1a18566c9bf78c0dac84713152a
Author: Amanda Moran <am...@apache.com>
Date: 2016-04-14T22:20:30Z
Merge branch 'master' of github.com:apache/incubator-trafodion into installbugfixes
commit ad27625f2bfc4b0892802fce0e8dcc46424c193c
Author: Amanda Moran <am...@apache.com>
Date: 2016-04-18T16:30:11Z
Merge branch 'master' of github.com:apache/incubator-trafodion into installbugfixes
commit ac4fa8f2c36ee5dc13726529d05f4113cb87f7be
Author: Amanda Moran <am...@apache.com>
Date: 2016-04-19T00:07:22Z
Merge branch 'master' of github.com:apache/incubator-trafodion into installbugfixes
commit 17cef884982714d6f9a319f16179c777454b49a5
Author: Amanda Moran <am...@apache.com>
Date: 2016-04-19T00:19:14Z
[[TRAFODION 1927]] [[TRAFODION 1692]] [[TRAFODION 1572]]
Install on multi-cluster (cloudera setup) --disabled
Vanilla Apache HBase support
Copy error in traf_*mods
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: Vanilla Apache HBase support and...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-trafodion/pull/440
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: Vanilla Apache HBase support and...
Posted by amandamoran <gi...@git.apache.org>.
Github user amandamoran commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/440#discussion_r60457285
--- Diff: install/installer/traf_add_sudoAccess ---
@@ -40,8 +40,8 @@ function backupRestore {
source $TRAF_CONFIG
-echo "## Allow $BACKUP_USER id to run commands needed for backup and restore" > $BACKUP_USER
-echo "%$BACKUP_USER ALL =(hbase) NOPASSWD: /usr/bin/hbase" >> $BACKUP_USER
+sudo echo "## Allow $BACKUP_USER id to run commands needed for backup and restore" > $BACKUP_USER
+sudo echo "%$BACKUP_USER ALL =(hbase) NOPASSWD: /usr/bin/hbase" >> $BACKUP_USER
--- End diff --
Good find. I will remove that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: Vanilla Apache HBase support and...
Posted by amandamoran <gi...@git.apache.org>.
Github user amandamoran commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/440#discussion_r60455673
--- Diff: install/installer/traf_apache_mods ---
@@ -0,0 +1,267 @@
+#!/bin/bash
+# @@@ START COPYRIGHT @@@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# @@@ END COPYRIGHT @@@
+#
+# This script will configure HBase with HBase-trx
+# and co-processors needed for Trafodion. It uses
+# Ambari's configs.sh script to do this.
+#
+# NOTE: Only for Ambari installations
+
+TRAF_CONFIG=/etc/trafodion/trafodion_config
+source $TRAF_CONFIG
+
+export PDSH="pdsh -R exec"
+export PDSH_SSH_CMD="ssh -q -n %h"
+export PDCP="pdcp -R ssh"
+
+export PDSH_HADOOP_NODES="$PDSH $MY_HBASE_NODES $PDSH_SSH_CMD"
+export PDCP_HADOOP_NODES="$PDCP $MY_HBASE_NODES"
+#=====================================
+# copy Trafodion hbase trx jar to /usr/lib/hbase/lib
+
+cd $UNTAR_DIR
+
+HDFS_NODE=$(echo $HDFS_NODES | head -n1 | awk '{print $1;}')
+HBASE_NODE=$(echo $HBASE_NODES | head -n1 | awk '{print $1;}')
+echo "export HDFS_NODE=\"$HDFS_NODE\"" >> $TRAF_CONFIG
+echo "export HBASE_NODE=\"$HBASE_NODE\"" >> $TRAF_CONFIG
+sudo chmod 777 $TRAF_CONFIG
+source $TRAF_CONFIG
+
+
+hbase_trx_jar="hbase-trx-apache1_0_2-2.0.0.jar"
+
+traf_util_jar="trafodion-utility-*.jar"
+
+
+# The permissions the Trafodion build process creates on the hbase-trx jar
+# files does not work well with the installation process so we change them
+sudo chmod -R 777 $UNTAR_DIR/export/lib
+
+if [ ! -f $UNTAR_DIR/export/lib/$hbase_trx_jar ]; then
+ echo "***ERROR: unable to find $UNTAR_DIR/export/lib/$hbase_trx_jar"
+ exit -1
+fi
+
+# if more than one node then copy to all nodes
+echo "***INFO: copying $hbase_trx_jar to all nodes"
+if [ $node_count -ne 1 ]; then
+ $PDSH_HADOOP_NODES sudo rm -rf $HBASE_HOME/lib/hbase-trx* 2>/dev/null
+ $TRAF_PDSH mkdir -p $LOCAL_WORKDIR 2>/dev/null
+ $PDSH_HADOOP_NODES mkdir -p $LOCAL_WORKDIR 2>/dev/null
+ cp $UNTAR_DIR/export/lib/$hbase_trx_jar $LOCAL_WORKDIR
+ cp $UNTAR_DIR/export/lib/$traf_util_jar $LOCAL_WORKDIR
+ $PDCP_HADOOP_NODES $LOCAL_WORKDIR/$hbase_trx_jar $LOCAL_WORKDIR
+ $PDCP_HADOOP_NODES $LOCAL_WORKDIR/$traf_util_jar $LOCAL_WORKDIR
+ $PDSH_HADOOP_NODES sudo cp $LOCAL_WORKDIR/$traf_util_jar $HBASE_HOME/lib
+ $PDSH_HADOOP_NODES sudo cp $LOCAL_WORKDIR/$hbase_trx_jar $HBASE_HOME/lib
+ $PDSH_HADOOP_NODES sudo chmod 644 $HBASE_HOME/lib/$hbase_trx_jar
+ $PDSH_HADOOP_NODES sudo chmod 644 $HBASE_HOME/lib/$traf_util_jar
+
+ $PDSH_HADOOP_NODES rm $LOCAL_WORKDIR/$hbase_trx_jar 2>/dev/null
+ $PDSH_HADOOP_NODES rm $LOCAL_WORKDIR/$traf_util_jar 2>/dev/null
+else
+ for node in $HBASE_NODES
+ do
+ ssh -q -n $node sudo rm -rf $HBASE_HOME/lib/hbase-trx* 2>/dev/null
+ ssh -q -n $node sudo mkdir -p $TRAF_WORKDIR 2>/dev/null
+ ssh -q -n $node sudo chmod 777 $TRAF_WORKDIR
+ scp -q $UNTAR_DIR/export/lib/$hbase_trx_jar $(whoami)@$node:$TRAF_WORKDIR
+ scp -q $UNTAR_DIR/export/lib/$traf_util_jar $(whoami)@$node:$TRAF_WORKDIR
+ ssh -q -n $node sudo cp $TRAF_WORKDIR/$hbase_trx_jar $HBASE_HOME/lib
+ ssh -q -n $node sudo cp $TRAF_WORKDIR/$traf_util_jar $HBASE_HOME/lib
+ ssh -q -n $node sudo chmod 644 $HADOOP_PATH/$hbase_trx_jar
+ ssh -q -n $node sudo chmod 644 $HADOOP_PATH/$traf_util_jar
+ done
+fi
+
+#=======================================
+#Check that HBase-trx copied to all nodes
+
+for node in $HBASE_NODES
+do
+ copiedOver=$(ssh -q -n $node sudo ls $HBASE_HOME/lib/hbase-trx* | wc -l)
+ if [[ $copiedOver -ne "1" ]]; then
+ echo "***ERROR: $hbase_trx_jar was not copied on $node"
+ echo "***ERROR: Please investigate why this happened"
+ echo "***ERROR: Trafodion can not start without this. EXITING..."
+ exit -1
+ fi
+done
+
+echo "***INFO: $hbase_trx_jar copied correctly! Huzzah."
+
+
+
+#Copy hbase-site.xml file
+ssh -q -n $HBASE_NODE sudo cp $HBASE_HOME/conf/hbase-site.xml $HOME
+ssh -q -n $HBASE_NODE sudo chown $(whoami).$(whoami) $HOME/hbase-site.xml
+ssh -q -n $HBASE_NODE sudo chmod 777 $HOME/hbase-site.xml
+
+scp -q $(whoami)@$HBASE_NODE:$HOME/hbase-site.xml $HOME
+if [[ $? -gt 1 ]]; then
+ echo "***ERROR: Unable to find $HBASE_HOME/conf/hbase-site.xml file on $HBASE_NODE or unable to copy."
+ exit -1
+fi
+sudo cp $HOME/hbase-site.xml $TRAF_WORKDIR
+sudo chown trafodion.trafodion $TRAF_WORKDIR/hbase-site.xml
+
+#=====================================
+# create new directories for bulkload and lobs if not already there
+rm $LOCAL_WORKDIR/traf_temp_output 2>/dev/null
+
+ssh -q -n $HDFS_NODE 'sudo su' "$HDFS_USER" '--command "' "$HADOOP_PREFIX"'/bin/hdfs dfs -mkdir /hbase-staging" 2> $HOME/traf_temp_output'
+if [ $? != 0 ]; then
+ # ok if directory already exists
--- End diff --
You are right! This would be much cleaner! Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: Vanilla Apache HBase support and...
Posted by amandamoran <gi...@git.apache.org>.
Github user amandamoran commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/440#discussion_r60458634
--- Diff: install/installer/traf_config_check ---
@@ -652,15 +685,16 @@ function checkRoleGroups {
}
+
function checkClouderaVersion {
if [[ $CDH_5_3_HDP_2_2_SUPPORT == "N" ]]; then
#Check that Cloudera 5.2 or 5.3 are not installed.
if [[ "$CDH_5_4_SUPPORT" == "Y" ]] || [[ "$CDH_5_5_SUPPORT" == "Y" ]]; then
- nameOfVersion=$(ssh -q -n $node grep "Version" $HOME/hbaseVersion.txt | sed 's/,.*//' | sed 's/.*\-//' | grep cdh5.[4-5].*)
+ nameOfVersion=$(ssh -q -n $node grep "Version" $HOME/hbaseVersion.txt | sed 's/,.*//' | sed 's/.*\-//' | grep cdh5.[4-6].*)
#Check that Cloudera 5.[n>4].* is not installed.
if [[ -z $nameOfVersion ]]; then
- versionInstalled=$(ssh -q -n $node grep "Version" $HOME/hbaseVersion.txt | sed 's/,.*//' | sed 's/.*\-//' | grep cdh[6-9].[0-9].* | wc -l)
+ versionInstalled=$(ssh -q -n $node grep "Version" $HOME/hbaseVersion.txt | sed 's/,.*//' | sed 's/.*\-//' | grep cdh[5-9].[7-9].* | wc -l)
if [[ $versionInstalled -gt "0" ]]; then
--- End diff --
https://issues.apache.org/jira/browse/TRAFODION-1945
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: Vanilla Apache HBase support and...
Posted by svarnau <gi...@git.apache.org>.
Github user svarnau commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/440#discussion_r60450966
--- Diff: install/installer/traf_config_check ---
@@ -545,7 +545,7 @@ else
hadoopVersion=$(curl -k -su $ADMIN:$PASSWORD $URL/api/v1/clusters | grep version | grep -c CDH)
- if [[ $hadoopVersion -ne "1" ]]; then
+ if [[ $hadoopVersion -lt "1" ]]; then
hadoopVersion=$(curl -k -su $ADMIN:$PASSWORD $URL/api/v1/clusters | grep version | grep -c HDP)
--- End diff --
will numeric compare work if hadoop version has mre than two parts?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: Vanilla Apache HBase support and...
Posted by amandamoran <gi...@git.apache.org>.
Github user amandamoran commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/440#discussion_r60459435
--- Diff: install/installer/trafodion_install ---
@@ -539,7 +543,9 @@ source $TRAF_CONFIG
#============================================
#Check to make sure HBase Version is the correct version
-checkHBaseVersion
+if [[ $HADOOP_TYPE != "apache" ]]; then
+ checkHBaseVersion
+fi
#==============================================
--- End diff --
https://issues.apache.org/jira/browse/TRAFODION-1947
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: Vanilla Apache HBase support and...
Posted by svarnau <gi...@git.apache.org>.
Github user svarnau commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/440#discussion_r60449732
--- Diff: install/installer/traf_apache_mods ---
@@ -0,0 +1,267 @@
+#!/bin/bash
+# @@@ START COPYRIGHT @@@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# @@@ END COPYRIGHT @@@
+#
+# This script will configure HBase with HBase-trx
+# and co-processors needed for Trafodion. It uses
+# Ambari's configs.sh script to do this.
+#
+# NOTE: Only for Ambari installations
+
+TRAF_CONFIG=/etc/trafodion/trafodion_config
+source $TRAF_CONFIG
+
+export PDSH="pdsh -R exec"
+export PDSH_SSH_CMD="ssh -q -n %h"
+export PDCP="pdcp -R ssh"
+
+export PDSH_HADOOP_NODES="$PDSH $MY_HBASE_NODES $PDSH_SSH_CMD"
+export PDCP_HADOOP_NODES="$PDCP $MY_HBASE_NODES"
+#=====================================
+# copy Trafodion hbase trx jar to /usr/lib/hbase/lib
+
+cd $UNTAR_DIR
+
+HDFS_NODE=$(echo $HDFS_NODES | head -n1 | awk '{print $1;}')
+HBASE_NODE=$(echo $HBASE_NODES | head -n1 | awk '{print $1;}')
+echo "export HDFS_NODE=\"$HDFS_NODE\"" >> $TRAF_CONFIG
+echo "export HBASE_NODE=\"$HBASE_NODE\"" >> $TRAF_CONFIG
+sudo chmod 777 $TRAF_CONFIG
+source $TRAF_CONFIG
+
+
+hbase_trx_jar="hbase-trx-apache1_0_2-2.0.0.jar"
+
+traf_util_jar="trafodion-utility-*.jar"
+
+
+# The permissions the Trafodion build process creates on the hbase-trx jar
+# files does not work well with the installation process so we change them
+sudo chmod -R 777 $UNTAR_DIR/export/lib
+
+if [ ! -f $UNTAR_DIR/export/lib/$hbase_trx_jar ]; then
+ echo "***ERROR: unable to find $UNTAR_DIR/export/lib/$hbase_trx_jar"
+ exit -1
+fi
+
+# if more than one node then copy to all nodes
+echo "***INFO: copying $hbase_trx_jar to all nodes"
+if [ $node_count -ne 1 ]; then
+ $PDSH_HADOOP_NODES sudo rm -rf $HBASE_HOME/lib/hbase-trx* 2>/dev/null
+ $TRAF_PDSH mkdir -p $LOCAL_WORKDIR 2>/dev/null
+ $PDSH_HADOOP_NODES mkdir -p $LOCAL_WORKDIR 2>/dev/null
+ cp $UNTAR_DIR/export/lib/$hbase_trx_jar $LOCAL_WORKDIR
+ cp $UNTAR_DIR/export/lib/$traf_util_jar $LOCAL_WORKDIR
+ $PDCP_HADOOP_NODES $LOCAL_WORKDIR/$hbase_trx_jar $LOCAL_WORKDIR
+ $PDCP_HADOOP_NODES $LOCAL_WORKDIR/$traf_util_jar $LOCAL_WORKDIR
+ $PDSH_HADOOP_NODES sudo cp $LOCAL_WORKDIR/$traf_util_jar $HBASE_HOME/lib
+ $PDSH_HADOOP_NODES sudo cp $LOCAL_WORKDIR/$hbase_trx_jar $HBASE_HOME/lib
+ $PDSH_HADOOP_NODES sudo chmod 644 $HBASE_HOME/lib/$hbase_trx_jar
+ $PDSH_HADOOP_NODES sudo chmod 644 $HBASE_HOME/lib/$traf_util_jar
+
+ $PDSH_HADOOP_NODES rm $LOCAL_WORKDIR/$hbase_trx_jar 2>/dev/null
+ $PDSH_HADOOP_NODES rm $LOCAL_WORKDIR/$traf_util_jar 2>/dev/null
+else
+ for node in $HBASE_NODES
+ do
+ ssh -q -n $node sudo rm -rf $HBASE_HOME/lib/hbase-trx* 2>/dev/null
+ ssh -q -n $node sudo mkdir -p $TRAF_WORKDIR 2>/dev/null
+ ssh -q -n $node sudo chmod 777 $TRAF_WORKDIR
+ scp -q $UNTAR_DIR/export/lib/$hbase_trx_jar $(whoami)@$node:$TRAF_WORKDIR
+ scp -q $UNTAR_DIR/export/lib/$traf_util_jar $(whoami)@$node:$TRAF_WORKDIR
+ ssh -q -n $node sudo cp $TRAF_WORKDIR/$hbase_trx_jar $HBASE_HOME/lib
+ ssh -q -n $node sudo cp $TRAF_WORKDIR/$traf_util_jar $HBASE_HOME/lib
+ ssh -q -n $node sudo chmod 644 $HADOOP_PATH/$hbase_trx_jar
+ ssh -q -n $node sudo chmod 644 $HADOOP_PATH/$traf_util_jar
+ done
+fi
+
+#=======================================
+#Check that HBase-trx copied to all nodes
+
+for node in $HBASE_NODES
+do
+ copiedOver=$(ssh -q -n $node sudo ls $HBASE_HOME/lib/hbase-trx* | wc -l)
+ if [[ $copiedOver -ne "1" ]]; then
+ echo "***ERROR: $hbase_trx_jar was not copied on $node"
+ echo "***ERROR: Please investigate why this happened"
+ echo "***ERROR: Trafodion can not start without this. EXITING..."
+ exit -1
+ fi
+done
+
+echo "***INFO: $hbase_trx_jar copied correctly! Huzzah."
+
+
+
+#Copy hbase-site.xml file
+ssh -q -n $HBASE_NODE sudo cp $HBASE_HOME/conf/hbase-site.xml $HOME
+ssh -q -n $HBASE_NODE sudo chown $(whoami).$(whoami) $HOME/hbase-site.xml
+ssh -q -n $HBASE_NODE sudo chmod 777 $HOME/hbase-site.xml
+
+scp -q $(whoami)@$HBASE_NODE:$HOME/hbase-site.xml $HOME
+if [[ $? -gt 1 ]]; then
+ echo "***ERROR: Unable to find $HBASE_HOME/conf/hbase-site.xml file on $HBASE_NODE or unable to copy."
+ exit -1
+fi
+sudo cp $HOME/hbase-site.xml $TRAF_WORKDIR
+sudo chown trafodion.trafodion $TRAF_WORKDIR/hbase-site.xml
+
+#=====================================
+# create new directories for bulkload and lobs if not already there
+rm $LOCAL_WORKDIR/traf_temp_output 2>/dev/null
+
+ssh -q -n $HDFS_NODE 'sudo su' "$HDFS_USER" '--command "' "$HADOOP_PREFIX"'/bin/hdfs dfs -mkdir /hbase-staging" 2> $HOME/traf_temp_output'
+if [ $? != 0 ]; then
+ # ok if directory already exists
--- End diff --
If you use dfs -mkdir -p otion, I believe it will return zero exit if file exists.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: Vanilla Apache HBase support and...
Posted by amandamoran <gi...@git.apache.org>.
Github user amandamoran commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/440#discussion_r60456262
--- Diff: install/installer/traf_config_check ---
@@ -545,7 +545,7 @@ else
hadoopVersion=$(curl -k -su $ADMIN:$PASSWORD $URL/api/v1/clusters | grep version | grep -c CDH)
- if [[ $hadoopVersion -ne "1" ]]; then
+ if [[ $hadoopVersion -lt "1" ]]; then
hadoopVersion=$(curl -k -su $ADMIN:$PASSWORD $URL/api/v1/clusters | grep version | grep -c HDP)
--- End diff --
That a good question. I only tested on a multi-cluster setup with 2 clusters. I think it would, but I will test this in the future.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: Vanilla Apache HBase support and...
Posted by amandamoran <gi...@git.apache.org>.
Github user amandamoran commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/440#discussion_r60459219
--- Diff: install/installer/traf_config_check ---
@@ -595,10 +598,40 @@ else
fi
}
+function setPath {
+
+export PATH=$PATH:$HADOOP_PREFIX/bin:$HADOOP_PREFIX/sbin:$HBASE_HOME/bin
+
+sudo chmod 777 $TRAF_CONFIG
+sed -i '/PATH\=/d' $TRAF_CONFIG
+echo "export PATH=\"$PATH\"" >> $TRAF_CONFIG
+sudo chmod 777 $TRAF_CONFIG
+source $TRAF_CONFIG
+
+}
+
+function setHBaseDistro {
+
+export HBASE_DISTRO="APACHE"
+
--- End diff --
https://issues.apache.org/jira/browse/TRAFODION-1946
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: Vanilla Apache HBase support and...
Posted by robertamarton <gi...@git.apache.org>.
Github user robertamarton commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/440#discussion_r60450892
--- Diff: install/installer/traf_config_check ---
@@ -652,15 +685,16 @@ function checkRoleGroups {
}
+
function checkClouderaVersion {
if [[ $CDH_5_3_HDP_2_2_SUPPORT == "N" ]]; then
#Check that Cloudera 5.2 or 5.3 are not installed.
if [[ "$CDH_5_4_SUPPORT" == "Y" ]] || [[ "$CDH_5_5_SUPPORT" == "Y" ]]; then
- nameOfVersion=$(ssh -q -n $node grep "Version" $HOME/hbaseVersion.txt | sed 's/,.*//' | sed 's/.*\-//' | grep cdh5.[4-5].*)
+ nameOfVersion=$(ssh -q -n $node grep "Version" $HOME/hbaseVersion.txt | sed 's/,.*//' | sed 's/.*\-//' | grep cdh5.[4-6].*)
#Check that Cloudera 5.[n>4].* is not installed.
if [[ -z $nameOfVersion ]]; then
- versionInstalled=$(ssh -q -n $node grep "Version" $HOME/hbaseVersion.txt | sed 's/,.*//' | sed 's/.*\-//' | grep cdh[6-9].[0-9].* | wc -l)
+ versionInstalled=$(ssh -q -n $node grep "Version" $HOME/hbaseVersion.txt | sed 's/,.*//' | sed 's/.*\-//' | grep cdh[5-9].[7-9].* | wc -l)
if [[ $versionInstalled -gt "0" ]]; then
--- End diff --
Instead of hardcoding the values of CDH you want to add define somewhere simplifying the changes as we go forward.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: Vanilla Apache HBase support and...
Posted by svarnau <gi...@git.apache.org>.
Github user svarnau commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/440#discussion_r60448911
--- Diff: install/installer/traf_add_sudoAccess ---
@@ -40,8 +40,8 @@ function backupRestore {
source $TRAF_CONFIG
-echo "## Allow $BACKUP_USER id to run commands needed for backup and restore" > $BACKUP_USER
-echo "%$BACKUP_USER ALL =(hbase) NOPASSWD: /usr/bin/hbase" >> $BACKUP_USER
+sudo echo "## Allow $BACKUP_USER id to run commands needed for backup and restore" > $BACKUP_USER
+sudo echo "%$BACKUP_USER ALL =(hbase) NOPASSWD: /usr/bin/hbase" >> $BACKUP_USER
--- End diff --
sudo does no good here. The file re-direct happens as the current user (output of sudo).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: Vanilla Apache HBase support and...
Posted by amandamoran <gi...@git.apache.org>.
Github user amandamoran commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/440#discussion_r60456457
--- Diff: install/installer/traf_config_check ---
@@ -595,10 +598,40 @@ else
fi
}
+function setPath {
+
+export PATH=$PATH:$HADOOP_PREFIX/bin:$HADOOP_PREFIX/sbin:$HBASE_HOME/bin
+
+sudo chmod 777 $TRAF_CONFIG
+sed -i '/PATH\=/d' $TRAF_CONFIG
+echo "export PATH=\"$PATH\"" >> $TRAF_CONFIG
+sudo chmod 777 $TRAF_CONFIG
+source $TRAF_CONFIG
+
+}
+
+function setHBaseDistro {
+
+export HBASE_DISTRO="APACHE"
+
--- End diff --
I agree. This variable is only used with Hadoop version is Apache in sqenvcom.sh.
A JIRA to improve sqnencom.sh to be more standard and use more of the current variables would be a good idea.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: Vanilla Apache HBase support and...
Posted by svarnau <gi...@git.apache.org>.
Github user svarnau commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/440#discussion_r60451661
--- Diff: install/installer/traf_config_check ---
@@ -595,10 +598,40 @@ else
fi
}
+function setPath {
+
+export PATH=$PATH:$HADOOP_PREFIX/bin:$HADOOP_PREFIX/sbin:$HBASE_HOME/bin
+
+sudo chmod 777 $TRAF_CONFIG
+sed -i '/PATH\=/d' $TRAF_CONFIG
+echo "export PATH=\"$PATH\"" >> $TRAF_CONFIG
+sudo chmod 777 $TRAF_CONFIG
+source $TRAF_CONFIG
+
+}
+
+function setHBaseDistro {
+
+export HBASE_DISTRO="APACHE"
+
--- End diff --
hard coded to Apache here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: Vanilla Apache HBase support and...
Posted by amandamoran <gi...@git.apache.org>.
Github user amandamoran commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/440#discussion_r60455914
--- Diff: install/installer/traf_config_check ---
@@ -652,15 +685,16 @@ function checkRoleGroups {
}
+
function checkClouderaVersion {
if [[ $CDH_5_3_HDP_2_2_SUPPORT == "N" ]]; then
#Check that Cloudera 5.2 or 5.3 are not installed.
if [[ "$CDH_5_4_SUPPORT" == "Y" ]] || [[ "$CDH_5_5_SUPPORT" == "Y" ]]; then
- nameOfVersion=$(ssh -q -n $node grep "Version" $HOME/hbaseVersion.txt | sed 's/,.*//' | sed 's/.*\-//' | grep cdh5.[4-5].*)
+ nameOfVersion=$(ssh -q -n $node grep "Version" $HOME/hbaseVersion.txt | sed 's/,.*//' | sed 's/.*\-//' | grep cdh5.[4-6].*)
#Check that Cloudera 5.[n>4].* is not installed.
if [[ -z $nameOfVersion ]]; then
- versionInstalled=$(ssh -q -n $node grep "Version" $HOME/hbaseVersion.txt | sed 's/,.*//' | sed 's/.*\-//' | grep cdh[6-9].[0-9].* | wc -l)
+ versionInstalled=$(ssh -q -n $node grep "Version" $HOME/hbaseVersion.txt | sed 's/,.*//' | sed 's/.*\-//' | grep cdh[5-9].[7-9].* | wc -l)
if [[ $versionInstalled -gt "0" ]]; then
--- End diff --
Yes. I won't edit for this check-in but I will write a JIRA to improve the version checking. It's gotten too complicated. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: Vanilla Apache HBase support and...
Posted by amandamoran <gi...@git.apache.org>.
Github user amandamoran commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/440#discussion_r60456815
--- Diff: install/installer/traf_config_setup ---
@@ -407,61 +415,270 @@ else
fi
#==============================================
-#Hadoop user name
+#Enter Hadoop Type
-echo -n "Enter Hadoop admin username, default is [$ADMIN]: "
-read answer
+echo -n "Specify the version of Hadoop installed (1: Cloudera, 2: Hortonworks, 3: Other): "
+read answer1
+ if [[ -z "$answer1" ]]; then
+ echo "***ERROR: User must specify the version of Hadoop being used."
+ exit -1
+ fi
+ case "$answer1" in
+ 1) HADOOP_TYPE="cloudera"
+ echo "export HADOOP_TYPE=\"$HADOOP_TYPE\"" >> $LOCAL_TRAF_CONFIG
+ ;;
+ 2) HADOOP_TYPE="hortonworks"
+ echo "export HADOOP_TYPE=\"$HADOOP_TYPE\"" >> $LOCAL_TRAF_CONFIG
+ ;;
+ 3) HADOOP_TYPE="apache"
+ echo "export HADOOP_TYPE=\"$HADOOP_TYPE\"" >> $LOCAL_TRAF_CONFIG
+ esac
-if [ -z $answer ]; then
- echo "export ADMIN=\"$ADMIN\"" >> $LOCAL_TRAF_CONFIG
- username=$ADMIN
-else
- echo "export ADMIN=\"$answer\"" >> $LOCAL_TRAF_CONFIG
- username=$answer
-fi
+export $HADOOP_TYPE
#==============================================
-#Hadoop Password
+if [[ "$HADOOP_TYPE" == "apache" ]]; then
+ echo -n "Enter Hadoop installed full path (example: '/opt/hadoop-2.6.0'), default is [$HADOOP_PREFIX]: "
+ read answer
-echo -n "Enter Hadoop admin password, default is [$PASSWORD]: "
-read answer
+ if [ -z $answer ]; then
+ if [[ -z $HADOOP_PREFIX ]]; then
+ echo "***ERROR: Must enter Hadoop installed path"
+ exit -1
+ fi
+ else
+ if [[ -e $answer ]]; then
+ HADOOP_PREFIX=$answer
+ else
+ echo "***ERROR: Hadoop installed path doesn't exist"
+ exit -1
+ fi
+ fi
+ echo "export HADOOP_PREFIX=\"$HADOOP_PREFIX\"" >> $LOCAL_TRAF_CONFIG
+
+ #Hbase path
+ echo -n "Enter HBase installed full path (example: '/opt/hbase-1.2.0'), default is [$HBASE_HOME]: "
+ read answer
+
+ if [ -z $answer ]; then
+ if [[ -z $HBASE_HOME ]]; then
+ echo "***ERROR: Must enter HBase installed path"
+ exit -1
+ fi
+ else
+ if [[ -e $answer ]]; then
+ HBASE_HOME=$answer
+ else
+ echo "***ERROR: HBase installed path doesn't exist"
+ exit -1
+ fi
+ fi
+ echo "export HBASE_HOME=\"$HBASE_HOME\"" >> $LOCAL_TRAF_CONFIG
+
+ #Zookeeper path
+ echo -n "Enter Zookeeper installed full path (example: '/opt/zoo'), default is [$ZOO_HOME]: "
+ read answer0
+
--- End diff --
I think I had a reason at the time but I don't think it needs to be this now.
Thanks. I will change it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: Vanilla Apache HBase support and...
Posted by robertamarton <gi...@git.apache.org>.
Github user robertamarton commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/440#discussion_r60454090
--- Diff: install/installer/trafodion_install ---
@@ -539,7 +543,9 @@ source $TRAF_CONFIG
#============================================
#Check to make sure HBase Version is the correct version
-checkHBaseVersion
+if [[ $HADOOP_TYPE != "apache" ]]; then
+ checkHBaseVersion
+fi
#==============================================
--- End diff --
There is no version checking required for vanilla apache?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: Vanilla Apache HBase support and...
Posted by robertamarton <gi...@git.apache.org>.
Github user robertamarton commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/440#discussion_r60452411
--- Diff: install/installer/traf_config_setup ---
@@ -407,61 +415,270 @@ else
fi
#==============================================
-#Hadoop user name
+#Enter Hadoop Type
-echo -n "Enter Hadoop admin username, default is [$ADMIN]: "
-read answer
+echo -n "Specify the version of Hadoop installed (1: Cloudera, 2: Hortonworks, 3: Other): "
+read answer1
+ if [[ -z "$answer1" ]]; then
+ echo "***ERROR: User must specify the version of Hadoop being used."
+ exit -1
+ fi
+ case "$answer1" in
+ 1) HADOOP_TYPE="cloudera"
+ echo "export HADOOP_TYPE=\"$HADOOP_TYPE\"" >> $LOCAL_TRAF_CONFIG
+ ;;
+ 2) HADOOP_TYPE="hortonworks"
+ echo "export HADOOP_TYPE=\"$HADOOP_TYPE\"" >> $LOCAL_TRAF_CONFIG
+ ;;
+ 3) HADOOP_TYPE="apache"
+ echo "export HADOOP_TYPE=\"$HADOOP_TYPE\"" >> $LOCAL_TRAF_CONFIG
+ esac
-if [ -z $answer ]; then
- echo "export ADMIN=\"$ADMIN\"" >> $LOCAL_TRAF_CONFIG
- username=$ADMIN
-else
- echo "export ADMIN=\"$answer\"" >> $LOCAL_TRAF_CONFIG
- username=$answer
-fi
+export $HADOOP_TYPE
#==============================================
-#Hadoop Password
+if [[ "$HADOOP_TYPE" == "apache" ]]; then
+ echo -n "Enter Hadoop installed full path (example: '/opt/hadoop-2.6.0'), default is [$HADOOP_PREFIX]: "
+ read answer
-echo -n "Enter Hadoop admin password, default is [$PASSWORD]: "
-read answer
+ if [ -z $answer ]; then
+ if [[ -z $HADOOP_PREFIX ]]; then
+ echo "***ERROR: Must enter Hadoop installed path"
+ exit -1
+ fi
+ else
+ if [[ -e $answer ]]; then
+ HADOOP_PREFIX=$answer
+ else
+ echo "***ERROR: Hadoop installed path doesn't exist"
+ exit -1
+ fi
+ fi
+ echo "export HADOOP_PREFIX=\"$HADOOP_PREFIX\"" >> $LOCAL_TRAF_CONFIG
+
+ #Hbase path
+ echo -n "Enter HBase installed full path (example: '/opt/hbase-1.2.0'), default is [$HBASE_HOME]: "
+ read answer
+
+ if [ -z $answer ]; then
+ if [[ -z $HBASE_HOME ]]; then
+ echo "***ERROR: Must enter HBase installed path"
+ exit -1
+ fi
+ else
+ if [[ -e $answer ]]; then
+ HBASE_HOME=$answer
+ else
+ echo "***ERROR: HBase installed path doesn't exist"
+ exit -1
+ fi
+ fi
+ echo "export HBASE_HOME=\"$HBASE_HOME\"" >> $LOCAL_TRAF_CONFIG
+
+ #Zookeeper path
+ echo -n "Enter Zookeeper installed full path (example: '/opt/zoo'), default is [$ZOO_HOME]: "
+ read answer0
+
--- End diff --
Any reason why you used answer0 instead of answer here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: Vanilla Apache HBase support and...
Posted by amandamoran <gi...@git.apache.org>.
Github user amandamoran commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/440#discussion_r60456998
--- Diff: install/installer/trafodion_install ---
@@ -539,7 +543,9 @@ source $TRAF_CONFIG
#============================================
#Check to make sure HBase Version is the correct version
-checkHBaseVersion
+if [[ $HADOOP_TYPE != "apache" ]]; then
+ checkHBaseVersion
+fi
#==============================================
--- End diff --
The instructions will provide the version you need, so I didn't include a check at this time. I will write a JIRA to add that improvement.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---