You are viewing a plain text version of this content. The canonical link for it is here.
Posted to codereview@trafodion.apache.org by robertamarton <gi...@git.apache.org> on 2015/10/08 02:59:31 UTC
[GitHub] incubator-trafodion pull request: [TRAFODION-1513] Trafodion build...
GitHub user robertamarton opened a pull request:
https://github.com/apache/incubator-trafodion/pull/109
[TRAFODION-1513] Trafodion build from source tar failing
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/robertamarton/incubator-trafodion master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-trafodion/pull/109.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #109
----
commit e40f8f05149315e97c8f601cf14cf7080c22fc5e
Author: Roberta Marton <ro...@esgyn.com>
Date: 2015-10-08T00:41:17Z
Trafodion-1513 -- build fixes
The build instructions on the web site were not correctly downloading dependent
tools. To help in the process, a convenience script called traf_tools_setup.sh
located in the install directory was created to download the correct versions
into a user specified directory.
Removed some of the dependencies from the dependent tool list
- QT is not longer built, this is needed by the compiler debugging GUI.
- MAVEN, PROTOBUFs are required to be installed before building Trafodion.
This removes them from the Trafodion configuration file. Plus variables
that point to HADOOP, HBASE, and HIVE were added to the configuration file
to support standalone Apache builds.
Issues with TOOLSDIR not working correctly in sqenvcom.sh setup script.
Incorrect version of Hadoop was downloaded in install_local_hadoop which caused
build error - missing hdfs library.
install_local_hadoop tried to install an non-existent test package - made this
optional. Also report what versions of Hadoop, HBase, Hive, and MySql were downloaded for testing. Trafodion JIRA 1512 was created to fix an additional issue
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1513] Trafodion build...
Posted by robertamarton <gi...@git.apache.org>.
Github user robertamarton commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/109#discussion_r41559351
--- Diff: core/sqf/sqenvcom.sh ---
@@ -95,17 +95,20 @@ cpucnt=$(grep processor /proc/cpuinfo | wc -l)
# no number means unlimited, and will swamp the system
export MAKEFLAGS="-j$cpucnt"
+# For now set up the TOOLSDIR, it may be overwritten later when the
+# .trafodion configuration file is loaded.
if [[ -n "$CLUSTERNAME" ]]; then
- export MY_ROOT=/opt/hp
- export TOOLSDIR=${TOOLSDIR:-/home/tools}
+ if [ -z "$TOOLSDIR" ]; then
+ export TOOLSDIR=${TOOLSDIR:-/home/tools}
+ fi
--- End diff --
No, this prevents the TOOLSDIR from being overwritten to either /opt/home/tools or /home/tools. On a VM or a generic system, the tools directory does not exist. If you invoke sqenvcom.sh after setting your own TOOLSDIR envvar and don't create the .trafodion file, your tools directory variable is overwritten and a subsequent build fails.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1513] Trafodion build...
Posted by svarnau <gi...@git.apache.org>.
Github user svarnau commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/109#discussion_r41563361
--- Diff: install/traf_tools_setup.sh ---
@@ -0,0 +1,249 @@
+#!/bin/bash
+#
+# @@@ START COPYRIGHT @@@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# @@@ END COPYRIGHT @@@
+#
+
+# -----------------------------------------------------------------------------
+# script: traf_tools_setup
+#
+# Helper script that downloads and installs dependent tools required by
+# Apache Trafodion to build from a source distribution
+#
+# Tools installed:
+#
+# MPICH: Implementation of the Message Passing Interface (MPI) standard. For use
+# in Trafodion, MPICH must be built to force sockets to be used in both
+# internode and intranode message passing.
+# Bison: General-purpose parser generator.
+# Udis86: Minimalistic disassembler library (libudis86) for the x86 class of
+# instruction set architectures.
+# LLVM: Collection of modular and reusable compiler and toolchain technologies.
+# ICU: C/C++ and Java libraries providing Unicode and Globalization support for
+# software applications.
+# Zookeeper: Coordination service for distributed applications. It exposes
+# common services such as naming, configuration management, synchronization,
+# and group services.
+# Thrift: Communications and data serialization tool
+#
+# Script can be modified to meet the needs of your environment
+# May need root or SUDO access to install tools in desired location
+# -----------------------------------------------------------------------------
+
+function Usage {
+ echo
+ echo "Usage: $0 -d <downloaddir> -i <installdir>"
+ echo
+ echo " -d <downloaddir> - location of download directory"
+ echo " -i <installdir> - location of install directory"
+ echo " -v - verbose mode"
+ echo
+ echo " -h - help"
+ echo
+ echo "example: traf_tools_setup.sh -d /home/userx/download -i /home/userx/tools"
+ echo
+}
+
+ VERBOSE=0
+ TOOLSDIR=
+ BASEDIR=
+
+ while getopts "d:i:hv" arg
+ do
+ case $arg in
+ d)
+ BASEDIR=${OPTARG};
+ ;;
+ h)
+ Usage;
+ exit 1;
+ ;;
+ i)
+ TOOLSDIR=${OPTARG};
+ ;;
+ v)
+ VERBOSE=1;
+ ;;
+ *)
+ Usage;
+ exit 1;
+ ;;
+ esac
+ done
+
+
+if [ "$BASEDIR" == "" ]; then
+ echo
+ echo "ERROR: download directory (-d) is not specified"
+ Usage;
+ exit 1;
+fi
+
+if [ ! -d "$BASEDIR" ]; then
+ echo
+ echo "ERROR: download directory ($BASEDIR) does not exist"
+ echo
+ exit 1;
+fi
+
+if [ "$TOOLSDIR" == "" ]; then
+ echo
+ echo "ERROR: install directory (-t) is not specified"
+ Usage;
+ exit 1;
+fi
+
+if [ ! -d "$TOOLSDIR" ]; then
+ echo
+ echo "ERROR: install directory ($TOOLSDIR) does not exist"
+ echo
+ exit 1;
+fi
+
+if [ $VERBOSE -eq 1 ]; then
+ echo "basedir is: $BASDIR";
+ echo "toolsdir is: $TOOLSDIR":
+fi
+
+#install mpi
+cd $BASEDIR
+wget http://www.mpich.org/static/downloads/3.0.4/mpich-3.0.4.tar.gz
+tar -xzf mpich-3.0.4.tar.gz
+cd mpich-3.0.4
+./configure --prefix=$TOOLSDIR/dest-mpich-3.0.4 --with-device=ch3:sock --disable-f77 --disable-fc
+make
+make check
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of MPI"
+fi
+
+# See if need to install protobufs tested with version 2.3.0
+protoc --version
+
+# install bison
+cd $BASEDIR
+wget http://ftp.gnu.org/gnu/bison/bison-3.0.tar.gz
+tar -xzf bison-3.0.tar.gz
+cd bison-3.0
+./configure --prefix=$TOOLSDIR/bison_3_linux
+make
+make check
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of BISON"
+fi
+
+# install udis
+cd $BASEDIR
+wget http://sourceforge.net/projects/udis86/files/udis86/1.7/udis86-1.7.2.tar.gz
+tar -xzf udis86-1.7.2.tar.gz
+cd udis86-1.7.2
+./configure --prefix=$TOOLSDIR/udis86-1.7.2 --enable-shared
+make
+make check
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of UDIS"
+fi
+
+#install LLVM
+
+cd $BASEDIR
+wget http://llvm.org/releases/3.2/llvm-3.2.src.tar.gz
+tar xzf llvm-3.2.src.tar.gz
+
+export MY_UDIS_INSTALL_DIR=$TOOLSDIR/udis86-1.7.2
+export MY_LLVM_INSTALL_DIR=$TOOLSDIR/dest-llvm-3.2/
+export MY_LLVM_SRC_DIR=$BASEDIR/llvm-3.2.src
+export MY_LLVM_OBJ_DIR=$BASEDIR/llvm-3.2.obj/
+export LD_LIBRARY_PATH=$MY_UDIS_INSTALL_DIR/lib:$LD_LIBRARY_PATH
+export C_INCLUDE_PATH=$MY_UDIS_INSTALL_DIR/include
+export CPATH=$MY_UDIS_INSTALL_DIR/include
+
+# Build release version
+mkdir -p $MY_LLVM_OBJ_DIR/release
+cd $MY_LLVM_OBJ_DIR/release
+
+$MY_LLVM_SRC_DIR/configure --prefix=$MY_LLVM_INSTALL_DIR/release \
+ --enable-shared --enable-targets=x86,x86_64,cpp \
+ --with-udis86=$MY_UDIS_INSTALL_DIR/lib \
+ CFLAGS=-fgnu89-inline
+
+make libs-only
+make install-libs
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of LLVM release"
+fi
+
+# Build debug version
+mkdir -p $MY_LLVM_OBJ_DIR/debug
+cd $MY_LLVM_OBJ_DIR/debug
+
+$MY_LLVM_SRC_DIR/configure --prefix=$MY_LLVM_INSTALL_DIR/debug \
+ --enable-optimized --enable-jit \
+ --enable-debug-runtime --enable-debug-symbols \
+ --enable-shared --enable-targets=x86,x86_64,cpp \
+ --with-udis86=$MY_UDIS_INSTALL_DIR/lib \
+ CFLAGS=-fgnu89-inline
+
+make libs-only
+make install-libs
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of LLVM debug"
+fi
+
+# Build ICU
+cd $BASEDIR
+wget http://download.icu-project.org/files/icu4c/4.4/icu4c-4_4-src.tgz
+tar -xzf icu4c-4_4-src.tgz
+cd icu/source
+./configure --with-library-suffix=Nv44 --prefix=$TOOLSDIR/icu4.4/linux64
+make
+make check
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of ICU"
+fi
+
+# make zookeeper
+cd $BASEDIR
+wget https://archive.apache.org/dist/zookeeper/zookeeper-3.4.5/zookeeper-3.4.5.tar.gz
+tar -zxf zookeeper-3.4.5.tar.gz
+cd zookeeper-3.4.5/src/c
+./configure --prefix=$TOOLSDIR/zookeeper-3.4.5
+make
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of ZOOKEEPER"
+fi
+
+# make thrift
+cd $BASEDIR
+wget http://archive.apache.org/dist/thrift/0.9.0/thrift-0.9.0.tar.gz
+tar -xzf thrift-0.9.0.tar.gz
+cd thrift-0.9.0
+./configure --prefix=$TOOLSDIR/thrift-0.9.0 -without-qt
+make
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of THRIFT"
+fi
--- End diff --
If a "good enough" version can be installed from standard linux distro, then I think we should prefer that. This script should concentrate on what is not in the standard distro.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1513] Trafodion build...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/109#discussion_r41559606
--- Diff: core/sqf/sqenvcom.sh ---
@@ -95,17 +95,20 @@ cpucnt=$(grep processor /proc/cpuinfo | wc -l)
# no number means unlimited, and will swamp the system
export MAKEFLAGS="-j$cpucnt"
+# For now set up the TOOLSDIR, it may be overwritten later when the
+# .trafodion configuration file is loaded.
if [[ -n "$CLUSTERNAME" ]]; then
- export MY_ROOT=/opt/hp
- export TOOLSDIR=${TOOLSDIR:-/home/tools}
+ if [ -z "$TOOLSDIR" ]; then
+ export TOOLSDIR=${TOOLSDIR:-/home/tools}
+ fi
--- End diff --
I think what Steve is saying that the old syntax
export TOOLSDIR=${TOOLSDIR:-/home/tools}
already does that, it does nothing if TOOLSDIR is already set.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1513] Trafodion build...
Posted by robertamarton <gi...@git.apache.org>.
Github user robertamarton commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/109#discussion_r41568196
--- Diff: core/sqf/sqenvcom.sh ---
@@ -795,13 +799,9 @@ fi
# Non-standard or newer version tools
export BISON="${TOOLSDIR}/bison_3_linux/bin/bison" # Need 1.3 version or later
-
-
export LLVM="${TOOLSDIR}/dest-llvm-3.2"
export UDIS86="${TOOLSDIR}/udis86-1.7.2"
export ICU="${TOOLSDIR}/icu4.4"
-export QT_TOOLKIT="${TOOLSDIR}/Qt-4.8.5-64"
-
--- End diff --
Let me play with it. It will need additional script changes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1513] Trafodion build...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/109#discussion_r41580458
--- Diff: install/traf_tools_setup.sh ---
@@ -0,0 +1,249 @@
+#!/bin/bash
+#
+# @@@ START COPYRIGHT @@@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# @@@ END COPYRIGHT @@@
+#
+
+# -----------------------------------------------------------------------------
+# script: traf_tools_setup
+#
+# Helper script that downloads and installs dependent tools required by
+# Apache Trafodion to build from a source distribution
+#
+# Tools installed:
+#
+# MPICH: Implementation of the Message Passing Interface (MPI) standard. For use
+# in Trafodion, MPICH must be built to force sockets to be used in both
+# internode and intranode message passing.
+# Bison: General-purpose parser generator.
+# Udis86: Minimalistic disassembler library (libudis86) for the x86 class of
+# instruction set architectures.
+# LLVM: Collection of modular and reusable compiler and toolchain technologies.
+# ICU: C/C++ and Java libraries providing Unicode and Globalization support for
+# software applications.
+# Zookeeper: Coordination service for distributed applications. It exposes
+# common services such as naming, configuration management, synchronization,
+# and group services.
+# Thrift: Communications and data serialization tool
+#
+# Script can be modified to meet the needs of your environment
+# May need root or SUDO access to install tools in desired location
+# -----------------------------------------------------------------------------
+
+function Usage {
+ echo
+ echo "Usage: $0 -d <downloaddir> -i <installdir>"
+ echo
+ echo " -d <downloaddir> - location of download directory"
+ echo " -i <installdir> - location of install directory"
+ echo " -v - verbose mode"
+ echo
+ echo " -h - help"
+ echo
+ echo "example: traf_tools_setup.sh -d /home/userx/download -i /home/userx/tools"
+ echo
+}
+
+ VERBOSE=0
+ TOOLSDIR=
+ BASEDIR=
+
+ while getopts "d:i:hv" arg
+ do
+ case $arg in
+ d)
+ BASEDIR=${OPTARG};
+ ;;
+ h)
+ Usage;
+ exit 1;
+ ;;
+ i)
+ TOOLSDIR=${OPTARG};
+ ;;
+ v)
+ VERBOSE=1;
+ ;;
+ *)
+ Usage;
+ exit 1;
+ ;;
+ esac
+ done
+
+
+if [ "$BASEDIR" == "" ]; then
+ echo
+ echo "ERROR: download directory (-d) is not specified"
+ Usage;
+ exit 1;
+fi
+
+if [ ! -d "$BASEDIR" ]; then
+ echo
+ echo "ERROR: download directory ($BASEDIR) does not exist"
+ echo
+ exit 1;
+fi
+
+if [ "$TOOLSDIR" == "" ]; then
+ echo
+ echo "ERROR: install directory (-t) is not specified"
+ Usage;
+ exit 1;
+fi
+
+if [ ! -d "$TOOLSDIR" ]; then
+ echo
+ echo "ERROR: install directory ($TOOLSDIR) does not exist"
+ echo
+ exit 1;
+fi
+
+if [ $VERBOSE -eq 1 ]; then
+ echo "basedir is: $BASDIR";
+ echo "toolsdir is: $TOOLSDIR":
+fi
+
+#install mpi
+cd $BASEDIR
+wget http://www.mpich.org/static/downloads/3.0.4/mpich-3.0.4.tar.gz
+tar -xzf mpich-3.0.4.tar.gz
+cd mpich-3.0.4
+./configure --prefix=$TOOLSDIR/dest-mpich-3.0.4 --with-device=ch3:sock --disable-f77 --disable-fc
+make
+make check
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of MPI"
+fi
+
+# See if need to install protobufs tested with version 2.3.0
+protoc --version
+
+# install bison
+cd $BASEDIR
+wget http://ftp.gnu.org/gnu/bison/bison-3.0.tar.gz
+tar -xzf bison-3.0.tar.gz
+cd bison-3.0
+./configure --prefix=$TOOLSDIR/bison_3_linux
+make
+make check
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of BISON"
+fi
+
+# install udis
+cd $BASEDIR
+wget http://sourceforge.net/projects/udis86/files/udis86/1.7/udis86-1.7.2.tar.gz
+tar -xzf udis86-1.7.2.tar.gz
+cd udis86-1.7.2
+./configure --prefix=$TOOLSDIR/udis86-1.7.2 --enable-shared
+make
+make check
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of UDIS"
+fi
+
+#install LLVM
+
+cd $BASEDIR
+wget http://llvm.org/releases/3.2/llvm-3.2.src.tar.gz
+tar xzf llvm-3.2.src.tar.gz
+
+export MY_UDIS_INSTALL_DIR=$TOOLSDIR/udis86-1.7.2
+export MY_LLVM_INSTALL_DIR=$TOOLSDIR/dest-llvm-3.2/
+export MY_LLVM_SRC_DIR=$BASEDIR/llvm-3.2.src
+export MY_LLVM_OBJ_DIR=$BASEDIR/llvm-3.2.obj/
+export LD_LIBRARY_PATH=$MY_UDIS_INSTALL_DIR/lib:$LD_LIBRARY_PATH
+export C_INCLUDE_PATH=$MY_UDIS_INSTALL_DIR/include
+export CPATH=$MY_UDIS_INSTALL_DIR/include
+
+# Build release version
+mkdir -p $MY_LLVM_OBJ_DIR/release
+cd $MY_LLVM_OBJ_DIR/release
+
+$MY_LLVM_SRC_DIR/configure --prefix=$MY_LLVM_INSTALL_DIR/release \
+ --enable-shared --enable-targets=x86,x86_64,cpp \
+ --with-udis86=$MY_UDIS_INSTALL_DIR/lib \
+ CFLAGS=-fgnu89-inline
+
+make libs-only
+make install-libs
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of LLVM release"
+fi
+
+# Build debug version
+mkdir -p $MY_LLVM_OBJ_DIR/debug
+cd $MY_LLVM_OBJ_DIR/debug
+
+$MY_LLVM_SRC_DIR/configure --prefix=$MY_LLVM_INSTALL_DIR/debug \
+ --enable-optimized --enable-jit \
+ --enable-debug-runtime --enable-debug-symbols \
+ --enable-shared --enable-targets=x86,x86_64,cpp \
+ --with-udis86=$MY_UDIS_INSTALL_DIR/lib \
+ CFLAGS=-fgnu89-inline
+
+make libs-only
+make install-libs
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of LLVM debug"
+fi
+
+# Build ICU
+cd $BASEDIR
+wget http://download.icu-project.org/files/icu4c/4.4/icu4c-4_4-src.tgz
+tar -xzf icu4c-4_4-src.tgz
+cd icu/source
+./configure --with-library-suffix=Nv44 --prefix=$TOOLSDIR/icu4.4/linux64
+make
+make check
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of ICU"
+fi
+
+# make zookeeper
+cd $BASEDIR
+wget https://archive.apache.org/dist/zookeeper/zookeeper-3.4.5/zookeeper-3.4.5.tar.gz
+tar -zxf zookeeper-3.4.5.tar.gz
+cd zookeeper-3.4.5/src/c
+./configure --prefix=$TOOLSDIR/zookeeper-3.4.5
+make
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of ZOOKEEPER"
+fi
+
+# make thrift
+cd $BASEDIR
+wget http://archive.apache.org/dist/thrift/0.9.0/thrift-0.9.0.tar.gz
+tar -xzf thrift-0.9.0.tar.gz
+cd thrift-0.9.0
+./configure --prefix=$TOOLSDIR/thrift-0.9.0 -without-qt
+make
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of THRIFT"
+fi
--- End diff --
Good to know, but for now we are probably fine. I can handle the protobuf requirement inside my script for now - or we can move the code to your script (and also update all of our VM images).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1513] Trafodion build...
Posted by robertamarton <gi...@git.apache.org>.
Github user robertamarton commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/109#discussion_r41561640
--- Diff: core/sqf/sql/scripts/install_local_hadoop ---
@@ -925,8 +933,10 @@ else
if [ -f $MY_LOCAL_SW_DIST/${HADOOP_TAR} ]; then
cp $MY_LOCAL_SW_DIST/${HADOOP_TAR} .
+ echo "Downloaded Hadoop tar file: $MY_LOCAL_SW_DIST/${HADOOP_TAR}"
--- End diff --
Good point!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1513] Trafodion build...
Posted by robertamarton <gi...@git.apache.org>.
Github user robertamarton commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/109#discussion_r41561972
--- Diff: install/traf_tools_setup.sh ---
@@ -0,0 +1,249 @@
+#!/bin/bash
+#
+# @@@ START COPYRIGHT @@@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# @@@ END COPYRIGHT @@@
+#
+
+# -----------------------------------------------------------------------------
+# script: traf_tools_setup
+#
+# Helper script that downloads and installs dependent tools required by
+# Apache Trafodion to build from a source distribution
+#
+# Tools installed:
+#
+# MPICH: Implementation of the Message Passing Interface (MPI) standard. For use
+# in Trafodion, MPICH must be built to force sockets to be used in both
+# internode and intranode message passing.
+# Bison: General-purpose parser generator.
+# Udis86: Minimalistic disassembler library (libudis86) for the x86 class of
+# instruction set architectures.
+# LLVM: Collection of modular and reusable compiler and toolchain technologies.
+# ICU: C/C++ and Java libraries providing Unicode and Globalization support for
+# software applications.
+# Zookeeper: Coordination service for distributed applications. It exposes
+# common services such as naming, configuration management, synchronization,
+# and group services.
+# Thrift: Communications and data serialization tool
+#
+# Script can be modified to meet the needs of your environment
+# May need root or SUDO access to install tools in desired location
+# -----------------------------------------------------------------------------
+
+function Usage {
+ echo
+ echo "Usage: $0 -d <downloaddir> -i <installdir>"
+ echo
+ echo " -d <downloaddir> - location of download directory"
+ echo " -i <installdir> - location of install directory"
+ echo " -v - verbose mode"
+ echo
+ echo " -h - help"
+ echo
+ echo "example: traf_tools_setup.sh -d /home/userx/download -i /home/userx/tools"
+ echo
+}
+
+ VERBOSE=0
+ TOOLSDIR=
+ BASEDIR=
+
+ while getopts "d:i:hv" arg
+ do
+ case $arg in
+ d)
+ BASEDIR=${OPTARG};
+ ;;
+ h)
+ Usage;
+ exit 1;
+ ;;
+ i)
+ TOOLSDIR=${OPTARG};
+ ;;
+ v)
+ VERBOSE=1;
+ ;;
+ *)
+ Usage;
+ exit 1;
+ ;;
+ esac
+ done
+
+
+if [ "$BASEDIR" == "" ]; then
+ echo
+ echo "ERROR: download directory (-d) is not specified"
+ Usage;
+ exit 1;
+fi
+
+if [ ! -d "$BASEDIR" ]; then
+ echo
+ echo "ERROR: download directory ($BASEDIR) does not exist"
+ echo
+ exit 1;
+fi
+
+if [ "$TOOLSDIR" == "" ]; then
+ echo
+ echo "ERROR: install directory (-t) is not specified"
+ Usage;
+ exit 1;
+fi
+
+if [ ! -d "$TOOLSDIR" ]; then
+ echo
+ echo "ERROR: install directory ($TOOLSDIR) does not exist"
+ echo
+ exit 1;
+fi
+
+if [ $VERBOSE -eq 1 ]; then
+ echo "basedir is: $BASDIR";
+ echo "toolsdir is: $TOOLSDIR":
+fi
+
+#install mpi
+cd $BASEDIR
+wget http://www.mpich.org/static/downloads/3.0.4/mpich-3.0.4.tar.gz
+tar -xzf mpich-3.0.4.tar.gz
+cd mpich-3.0.4
+./configure --prefix=$TOOLSDIR/dest-mpich-3.0.4 --with-device=ch3:sock --disable-f77 --disable-fc
+make
+make check
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of MPI"
+fi
+
+# See if need to install protobufs tested with version 2.3.0
+protoc --version
+
+# install bison
+cd $BASEDIR
+wget http://ftp.gnu.org/gnu/bison/bison-3.0.tar.gz
+tar -xzf bison-3.0.tar.gz
+cd bison-3.0
+./configure --prefix=$TOOLSDIR/bison_3_linux
+make
+make check
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of BISON"
+fi
+
+# install udis
+cd $BASEDIR
+wget http://sourceforge.net/projects/udis86/files/udis86/1.7/udis86-1.7.2.tar.gz
+tar -xzf udis86-1.7.2.tar.gz
+cd udis86-1.7.2
+./configure --prefix=$TOOLSDIR/udis86-1.7.2 --enable-shared
+make
+make check
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of UDIS"
+fi
+
+#install LLVM
+
+cd $BASEDIR
+wget http://llvm.org/releases/3.2/llvm-3.2.src.tar.gz
+tar xzf llvm-3.2.src.tar.gz
+
+export MY_UDIS_INSTALL_DIR=$TOOLSDIR/udis86-1.7.2
+export MY_LLVM_INSTALL_DIR=$TOOLSDIR/dest-llvm-3.2/
+export MY_LLVM_SRC_DIR=$BASEDIR/llvm-3.2.src
+export MY_LLVM_OBJ_DIR=$BASEDIR/llvm-3.2.obj/
+export LD_LIBRARY_PATH=$MY_UDIS_INSTALL_DIR/lib:$LD_LIBRARY_PATH
+export C_INCLUDE_PATH=$MY_UDIS_INSTALL_DIR/include
+export CPATH=$MY_UDIS_INSTALL_DIR/include
+
+# Build release version
+mkdir -p $MY_LLVM_OBJ_DIR/release
+cd $MY_LLVM_OBJ_DIR/release
+
+$MY_LLVM_SRC_DIR/configure --prefix=$MY_LLVM_INSTALL_DIR/release \
+ --enable-shared --enable-targets=x86,x86_64,cpp \
+ --with-udis86=$MY_UDIS_INSTALL_DIR/lib \
+ CFLAGS=-fgnu89-inline
+
+make libs-only
+make install-libs
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of LLVM release"
+fi
+
+# Build debug version
+mkdir -p $MY_LLVM_OBJ_DIR/debug
+cd $MY_LLVM_OBJ_DIR/debug
+
+$MY_LLVM_SRC_DIR/configure --prefix=$MY_LLVM_INSTALL_DIR/debug \
+ --enable-optimized --enable-jit \
+ --enable-debug-runtime --enable-debug-symbols \
+ --enable-shared --enable-targets=x86,x86_64,cpp \
+ --with-udis86=$MY_UDIS_INSTALL_DIR/lib \
+ CFLAGS=-fgnu89-inline
+
+make libs-only
+make install-libs
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of LLVM debug"
+fi
+
+# Build ICU
+cd $BASEDIR
+wget http://download.icu-project.org/files/icu4c/4.4/icu4c-4_4-src.tgz
+tar -xzf icu4c-4_4-src.tgz
+cd icu/source
+./configure --with-library-suffix=Nv44 --prefix=$TOOLSDIR/icu4.4/linux64
+make
+make check
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of ICU"
+fi
+
+# make zookeeper
+cd $BASEDIR
+wget https://archive.apache.org/dist/zookeeper/zookeeper-3.4.5/zookeeper-3.4.5.tar.gz
+tar -zxf zookeeper-3.4.5.tar.gz
+cd zookeeper-3.4.5/src/c
+./configure --prefix=$TOOLSDIR/zookeeper-3.4.5
+make
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of ZOOKEEPER"
+fi
+
+# make thrift
+cd $BASEDIR
+wget http://archive.apache.org/dist/thrift/0.9.0/thrift-0.9.0.tar.gz
+tar -xzf thrift-0.9.0.tar.gz
+cd thrift-0.9.0
+./configure --prefix=$TOOLSDIR/thrift-0.9.0 -without-qt
+make
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of THRIFT"
+fi
--- End diff --
I did remove QT, Protocol buffers, and Maven from this list. I was thinking about adding it back in. If "protoc --version" does not produce the correct version then install it. If "mvn --version" does not produce the correct version than install it. However, I am not sure it is worth the effort. I assume that most system have these packages installed. What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1513] Trafodion build...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/109#discussion_r41560575
--- Diff: core/sqf/sqenvcom.sh ---
@@ -795,13 +799,9 @@ fi
# Non-standard or newer version tools
export BISON="${TOOLSDIR}/bison_3_linux/bin/bison" # Need 1.3 version or later
-
-
export LLVM="${TOOLSDIR}/dest-llvm-3.2"
export UDIS86="${TOOLSDIR}/udis86-1.7.2"
export ICU="${TOOLSDIR}/icu4.4"
-export QT_TOOLKIT="${TOOLSDIR}/Qt-4.8.5-64"
-
--- End diff --
That's fine, but Jenkins and most developers routinely build the GUI on our machines today, and that should continue to work. We use the DISPLAY command (invoking the GUI) all the time to debug queries.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1513] Trafodion build...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/109#discussion_r41558305
--- Diff: core/sqf/sql/scripts/install_local_hadoop ---
@@ -925,8 +933,10 @@ else
if [ -f $MY_LOCAL_SW_DIST/${HADOOP_TAR} ]; then
cp $MY_LOCAL_SW_DIST/${HADOOP_TAR} .
+ echo "Downloaded Hadoop tar file: $MY_LOCAL_SW_DIST/${HADOOP_TAR}"
--- End diff --
Given that we didn't download anything, should this message say "copied" instead of "downloaded"? Same for the following ones.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1513] Trafodion build...
Posted by robertamarton <gi...@git.apache.org>.
Github user robertamarton commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/109#discussion_r41561594
--- Diff: core/sqf/sql/scripts/install_hadoop_regr_test_env ---
@@ -134,12 +134,12 @@ fi
if [ ! -f $MY_LOCAL_SW_DIST/${TPCDS_ZIP} ]; then
# Right now there is no URL to download this tool automatically
- echo '*********************** ERROR *************************'
- echo "Please download the tpcds_kit.zip file from"
- echo "http://www.tpc.org/tpcds/dsgen-download-request.asp"
- echo "and place it into a directory pointed to by the MY_LOCAL_SW_DIST"
- echo "environment variable. Then retry installing. Sorry, this is due"
- echo "to the TPC wanting your email address."
+ # Please download the tpcds_kit.zip file from"
+ # "http://www.tpc.org/tpcds/dsgen-download-request.asp"
+ # "and place it into a directory pointed to by the MY_LOCAL_SW_DIST"
+ # "environment variable. Then retry installing. Sorry, this is due"
+ # "to the TPC wanting your email address."
+ echo "The testware tpcds_kit.zip does not exist and will not be installed"
exit 99
fi
--- End diff --
Most of the contributors that build Trafodion code are expecting things to work without change including the running of all regressions. To do this, a repository has been created that contains everything these contributors need. However, not all contributors have this luxury. As Hans pointed out, other contributors, cannot even follow the directions to download this software. That said, our regression tests require this to be installed - at least our Hive tests. Let me write a JIRA to on this. In the meantime, I can display something to the effect that some of the developer regressions will not pass.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1513] Trafodion build...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/109#discussion_r41559803
--- Diff: core/sqf/sqenvcom.sh ---
@@ -795,13 +799,9 @@ fi
# Non-standard or newer version tools
export BISON="${TOOLSDIR}/bison_3_linux/bin/bison" # Need 1.3 version or later
-
-
export LLVM="${TOOLSDIR}/dest-llvm-3.2"
export UDIS86="${TOOLSDIR}/udis86-1.7.2"
export ICU="${TOOLSDIR}/icu4.4"
-export QT_TOOLKIT="${TOOLSDIR}/Qt-4.8.5-64"
-
--- End diff --
Agree, could we leave this in? If we skip the GUI for the official Apache build script that's fine, but Jenkins and developers should continue to build it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1513] Trafodion build...
Posted by svarnau <gi...@git.apache.org>.
Github user svarnau commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/109#discussion_r41556251
--- Diff: core/sqf/sqenvcom.sh ---
@@ -95,17 +95,20 @@ cpucnt=$(grep processor /proc/cpuinfo | wc -l)
# no number means unlimited, and will swamp the system
export MAKEFLAGS="-j$cpucnt"
+# For now set up the TOOLSDIR, it may be overwritten later when the
+# .trafodion configuration file is loaded.
if [[ -n "$CLUSTERNAME" ]]; then
- export MY_ROOT=/opt/hp
- export TOOLSDIR=${TOOLSDIR:-/home/tools}
+ if [ -z "$TOOLSDIR" ]; then
+ export TOOLSDIR=${TOOLSDIR:-/home/tools}
+ fi
--- End diff --
Setting TOOLSDIR here seems to be doing same thing as before, but more complicated (3 lines instead of one). If we are explicitly checking for zero-length, then we might as well use the simple assignment:
export TOOLSDIR=/home/tools
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1513] Trafodion build...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/109#discussion_r41586523
--- Diff: core/sqf/sqenvcom.sh ---
@@ -795,13 +799,9 @@ fi
# Non-standard or newer version tools
export BISON="${TOOLSDIR}/bison_3_linux/bin/bison" # Need 1.3 version or later
-
-
export LLVM="${TOOLSDIR}/dest-llvm-3.2"
export UDIS86="${TOOLSDIR}/udis86-1.7.2"
export ICU="${TOOLSDIR}/icu4.4"
-export QT_TOOLKIT="${TOOLSDIR}/Qt-4.8.5-64"
-
--- End diff --
In my next checkin, I am planning to remove the checks for QT_TOOLKIT from the $MY_SQROOT/../bldenvchk.sh script. I'll also remove the check for the Hadoop lib and include directories.
If you want you can take out the QT_TOOLKIT from sqenvcom.sh and I can put it back in once I updated bldenvchk.sh.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1513] Trafodion build...
Posted by svarnau <gi...@git.apache.org>.
Github user svarnau commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/109#discussion_r41556665
--- Diff: core/sqf/sqenvcom.sh ---
@@ -795,13 +799,9 @@ fi
# Non-standard or newer version tools
export BISON="${TOOLSDIR}/bison_3_linux/bin/bison" # Need 1.3 version or later
-
-
export LLVM="${TOOLSDIR}/dest-llvm-3.2"
export UDIS86="${TOOLSDIR}/udis86-1.7.2"
export ICU="${TOOLSDIR}/icu4.4"
-export QT_TOOLKIT="${TOOLSDIR}/Qt-4.8.5-64"
-
--- End diff --
Not setting QT_TOOLKIT here means compiler debugger it is no longer included in the standard builds, including jenkins builds?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1513] Trafodion build...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-trafodion/pull/109
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1513] Trafodion build...
Posted by svarnau <gi...@git.apache.org>.
Github user svarnau commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/109#discussion_r41557352
--- Diff: core/sqf/sql/scripts/install_hadoop_regr_test_env ---
@@ -134,12 +134,12 @@ fi
if [ ! -f $MY_LOCAL_SW_DIST/${TPCDS_ZIP} ]; then
# Right now there is no URL to download this tool automatically
- echo '*********************** ERROR *************************'
- echo "Please download the tpcds_kit.zip file from"
- echo "http://www.tpc.org/tpcds/dsgen-download-request.asp"
- echo "and place it into a directory pointed to by the MY_LOCAL_SW_DIST"
- echo "environment variable. Then retry installing. Sorry, this is due"
- echo "to the TPC wanting your email address."
+ # Please download the tpcds_kit.zip file from"
+ # "http://www.tpc.org/tpcds/dsgen-download-request.asp"
+ # "and place it into a directory pointed to by the MY_LOCAL_SW_DIST"
+ # "environment variable. Then retry installing. Sorry, this is due"
+ # "to the TPC wanting your email address."
+ echo "The testware tpcds_kit.zip does not exist and will not be installed"
exit 99
fi
--- End diff --
Might want to give message about the implications. The hive tests (and one core test) will not work without this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1513] Trafodion build...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/109#discussion_r41559006
--- Diff: core/sqf/sql/scripts/install_local_hadoop ---
@@ -565,8 +572,9 @@ fi
# check for missing tpcds_kit.zip file
install_hadoop_regr_test_env --check
+INSTALL_TPCDS=0
if [ $? -ne 0 ]; then
- exit 1
+ INSTALL_TPCDS=1
--- End diff --
I would have expected that we set INSTALL_TPCDS to 0 here, since we do not want to attempt the install in this case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1513] Trafodion build...
Posted by robertamarton <gi...@git.apache.org>.
Github user robertamarton commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/109#discussion_r41560248
--- Diff: core/sqf/sqenvcom.sh ---
@@ -795,13 +799,9 @@ fi
# Non-standard or newer version tools
export BISON="${TOOLSDIR}/bison_3_linux/bin/bison" # Need 1.3 version or later
-
-
export LLVM="${TOOLSDIR}/dest-llvm-3.2"
export UDIS86="${TOOLSDIR}/udis86-1.7.2"
export ICU="${TOOLSDIR}/icu4.4"
-export QT_TOOLKIT="${TOOLSDIR}/Qt-4.8.5-64"
-
--- End diff --
Is there a requirement to build the QT_TOOLKIT? I asked around and it seemed to be an optional product. This would be one less dependency that the user has to provide. Given that I was unable to get QT to build on a VM, (it kept wanting more and more dependencies which I could not find). I assumed that if someone wants this tool, then we can provide instructions on how to build it. I can add a section on the build page to discuss how to build and use this tool.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1513] Trafodion build...
Posted by robertamarton <gi...@git.apache.org>.
Github user robertamarton commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/109#discussion_r41569466
--- Diff: core/sqf/sql/scripts/install_local_hadoop ---
@@ -565,8 +572,9 @@ fi
# check for missing tpcds_kit.zip file
install_hadoop_regr_test_env --check
+INSTALL_TPCDS=0
if [ $? -ne 0 ]; then
- exit 1
+ INSTALL_TPCDS=1
--- End diff --
You are correct. I thought I tested this version of the script but apparently not. Will fix it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1513] Trafodion build...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/109#discussion_r41563307
--- Diff: install/traf_tools_setup.sh ---
@@ -0,0 +1,249 @@
+#!/bin/bash
+#
+# @@@ START COPYRIGHT @@@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# @@@ END COPYRIGHT @@@
+#
+
+# -----------------------------------------------------------------------------
+# script: traf_tools_setup
+#
+# Helper script that downloads and installs dependent tools required by
+# Apache Trafodion to build from a source distribution
+#
+# Tools installed:
+#
+# MPICH: Implementation of the Message Passing Interface (MPI) standard. For use
+# in Trafodion, MPICH must be built to force sockets to be used in both
+# internode and intranode message passing.
+# Bison: General-purpose parser generator.
+# Udis86: Minimalistic disassembler library (libudis86) for the x86 class of
+# instruction set architectures.
+# LLVM: Collection of modular and reusable compiler and toolchain technologies.
+# ICU: C/C++ and Java libraries providing Unicode and Globalization support for
+# software applications.
+# Zookeeper: Coordination service for distributed applications. It exposes
+# common services such as naming, configuration management, synchronization,
+# and group services.
+# Thrift: Communications and data serialization tool
+#
+# Script can be modified to meet the needs of your environment
+# May need root or SUDO access to install tools in desired location
+# -----------------------------------------------------------------------------
+
+function Usage {
+ echo
+ echo "Usage: $0 -d <downloaddir> -i <installdir>"
+ echo
+ echo " -d <downloaddir> - location of download directory"
+ echo " -i <installdir> - location of install directory"
+ echo " -v - verbose mode"
+ echo
+ echo " -h - help"
+ echo
+ echo "example: traf_tools_setup.sh -d /home/userx/download -i /home/userx/tools"
+ echo
+}
+
+ VERBOSE=0
+ TOOLSDIR=
+ BASEDIR=
+
+ while getopts "d:i:hv" arg
+ do
+ case $arg in
+ d)
+ BASEDIR=${OPTARG};
+ ;;
+ h)
+ Usage;
+ exit 1;
+ ;;
+ i)
+ TOOLSDIR=${OPTARG};
+ ;;
+ v)
+ VERBOSE=1;
+ ;;
+ *)
+ Usage;
+ exit 1;
+ ;;
+ esac
+ done
+
+
+if [ "$BASEDIR" == "" ]; then
+ echo
+ echo "ERROR: download directory (-d) is not specified"
+ Usage;
+ exit 1;
+fi
+
+if [ ! -d "$BASEDIR" ]; then
+ echo
+ echo "ERROR: download directory ($BASEDIR) does not exist"
+ echo
+ exit 1;
+fi
+
+if [ "$TOOLSDIR" == "" ]; then
+ echo
+ echo "ERROR: install directory (-t) is not specified"
+ Usage;
+ exit 1;
+fi
+
+if [ ! -d "$TOOLSDIR" ]; then
+ echo
+ echo "ERROR: install directory ($TOOLSDIR) does not exist"
+ echo
+ exit 1;
+fi
+
+if [ $VERBOSE -eq 1 ]; then
+ echo "basedir is: $BASDIR";
+ echo "toolsdir is: $TOOLSDIR":
+fi
+
+#install mpi
+cd $BASEDIR
+wget http://www.mpich.org/static/downloads/3.0.4/mpich-3.0.4.tar.gz
+tar -xzf mpich-3.0.4.tar.gz
+cd mpich-3.0.4
+./configure --prefix=$TOOLSDIR/dest-mpich-3.0.4 --with-device=ch3:sock --disable-f77 --disable-fc
+make
+make check
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of MPI"
+fi
+
+# See if need to install protobufs tested with version 2.3.0
+protoc --version
+
+# install bison
+cd $BASEDIR
+wget http://ftp.gnu.org/gnu/bison/bison-3.0.tar.gz
+tar -xzf bison-3.0.tar.gz
+cd bison-3.0
+./configure --prefix=$TOOLSDIR/bison_3_linux
+make
+make check
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of BISON"
+fi
+
+# install udis
+cd $BASEDIR
+wget http://sourceforge.net/projects/udis86/files/udis86/1.7/udis86-1.7.2.tar.gz
+tar -xzf udis86-1.7.2.tar.gz
+cd udis86-1.7.2
+./configure --prefix=$TOOLSDIR/udis86-1.7.2 --enable-shared
+make
+make check
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of UDIS"
+fi
+
+#install LLVM
+
+cd $BASEDIR
+wget http://llvm.org/releases/3.2/llvm-3.2.src.tar.gz
+tar xzf llvm-3.2.src.tar.gz
+
+export MY_UDIS_INSTALL_DIR=$TOOLSDIR/udis86-1.7.2
+export MY_LLVM_INSTALL_DIR=$TOOLSDIR/dest-llvm-3.2/
+export MY_LLVM_SRC_DIR=$BASEDIR/llvm-3.2.src
+export MY_LLVM_OBJ_DIR=$BASEDIR/llvm-3.2.obj/
+export LD_LIBRARY_PATH=$MY_UDIS_INSTALL_DIR/lib:$LD_LIBRARY_PATH
+export C_INCLUDE_PATH=$MY_UDIS_INSTALL_DIR/include
+export CPATH=$MY_UDIS_INSTALL_DIR/include
+
+# Build release version
+mkdir -p $MY_LLVM_OBJ_DIR/release
+cd $MY_LLVM_OBJ_DIR/release
+
+$MY_LLVM_SRC_DIR/configure --prefix=$MY_LLVM_INSTALL_DIR/release \
+ --enable-shared --enable-targets=x86,x86_64,cpp \
+ --with-udis86=$MY_UDIS_INSTALL_DIR/lib \
+ CFLAGS=-fgnu89-inline
+
+make libs-only
+make install-libs
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of LLVM release"
+fi
+
+# Build debug version
+mkdir -p $MY_LLVM_OBJ_DIR/debug
+cd $MY_LLVM_OBJ_DIR/debug
+
+$MY_LLVM_SRC_DIR/configure --prefix=$MY_LLVM_INSTALL_DIR/debug \
+ --enable-optimized --enable-jit \
+ --enable-debug-runtime --enable-debug-symbols \
+ --enable-shared --enable-targets=x86,x86_64,cpp \
+ --with-udis86=$MY_UDIS_INSTALL_DIR/lib \
+ CFLAGS=-fgnu89-inline
+
+make libs-only
+make install-libs
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of LLVM debug"
+fi
+
+# Build ICU
+cd $BASEDIR
+wget http://download.icu-project.org/files/icu4c/4.4/icu4c-4_4-src.tgz
+tar -xzf icu4c-4_4-src.tgz
+cd icu/source
+./configure --with-library-suffix=Nv44 --prefix=$TOOLSDIR/icu4.4/linux64
+make
+make check
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of ICU"
+fi
+
+# make zookeeper
+cd $BASEDIR
+wget https://archive.apache.org/dist/zookeeper/zookeeper-3.4.5/zookeeper-3.4.5.tar.gz
+tar -zxf zookeeper-3.4.5.tar.gz
+cd zookeeper-3.4.5/src/c
+./configure --prefix=$TOOLSDIR/zookeeper-3.4.5
+make
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of ZOOKEEPER"
+fi
+
+# make thrift
+cd $BASEDIR
+wget http://archive.apache.org/dist/thrift/0.9.0/thrift-0.9.0.tar.gz
+tar -xzf thrift-0.9.0.tar.gz
+cd thrift-0.9.0
+./configure --prefix=$TOOLSDIR/thrift-0.9.0 -without-qt
+make
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of THRIFT"
+fi
--- End diff --
It it's not too much work it would be good to add it, but I would not say that it's required. Given how widespread the use of Maven is in Hadoop, we can probably expect people to have it.
Protoc is a real trouble maker (at least for me). I'm going to check in a script that downloads Hadoop sources and builds libhdfs. This script also builds a local copy of protoc 2.5.0, since the default for CentOS (2.3.0) is too old. Maybe I can remove that if this script sets up protoc 2.5.0.
Qt would be very good to have in this setup, since it's mostly meant for developers, who will need the compiler GUI. But, again, it's not required. Maybe Howard can help us set this up.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1513] Trafodion build...
Posted by svarnau <gi...@git.apache.org>.
Github user svarnau commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/109#discussion_r41557753
--- Diff: install/traf_tools_setup.sh ---
@@ -0,0 +1,249 @@
+#!/bin/bash
+#
+# @@@ START COPYRIGHT @@@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# @@@ END COPYRIGHT @@@
+#
+
+# -----------------------------------------------------------------------------
+# script: traf_tools_setup
+#
+# Helper script that downloads and installs dependent tools required by
+# Apache Trafodion to build from a source distribution
+#
+# Tools installed:
+#
+# MPICH: Implementation of the Message Passing Interface (MPI) standard. For use
+# in Trafodion, MPICH must be built to force sockets to be used in both
+# internode and intranode message passing.
+# Bison: General-purpose parser generator.
+# Udis86: Minimalistic disassembler library (libudis86) for the x86 class of
+# instruction set architectures.
+# LLVM: Collection of modular and reusable compiler and toolchain technologies.
+# ICU: C/C++ and Java libraries providing Unicode and Globalization support for
+# software applications.
+# Zookeeper: Coordination service for distributed applications. It exposes
+# common services such as naming, configuration management, synchronization,
+# and group services.
+# Thrift: Communications and data serialization tool
+#
+# Script can be modified to meet the needs of your environment
+# May need root or SUDO access to install tools in desired location
+# -----------------------------------------------------------------------------
+
+function Usage {
+ echo
+ echo "Usage: $0 -d <downloaddir> -i <installdir>"
+ echo
+ echo " -d <downloaddir> - location of download directory"
+ echo " -i <installdir> - location of install directory"
+ echo " -v - verbose mode"
+ echo
+ echo " -h - help"
+ echo
+ echo "example: traf_tools_setup.sh -d /home/userx/download -i /home/userx/tools"
+ echo
+}
+
+ VERBOSE=0
+ TOOLSDIR=
+ BASEDIR=
+
+ while getopts "d:i:hv" arg
+ do
+ case $arg in
+ d)
+ BASEDIR=${OPTARG};
+ ;;
+ h)
+ Usage;
+ exit 1;
+ ;;
+ i)
+ TOOLSDIR=${OPTARG};
+ ;;
+ v)
+ VERBOSE=1;
+ ;;
+ *)
+ Usage;
+ exit 1;
+ ;;
+ esac
+ done
+
+
+if [ "$BASEDIR" == "" ]; then
+ echo
+ echo "ERROR: download directory (-d) is not specified"
+ Usage;
+ exit 1;
+fi
+
+if [ ! -d "$BASEDIR" ]; then
+ echo
+ echo "ERROR: download directory ($BASEDIR) does not exist"
+ echo
+ exit 1;
+fi
+
+if [ "$TOOLSDIR" == "" ]; then
+ echo
+ echo "ERROR: install directory (-t) is not specified"
+ Usage;
+ exit 1;
+fi
+
+if [ ! -d "$TOOLSDIR" ]; then
+ echo
+ echo "ERROR: install directory ($TOOLSDIR) does not exist"
+ echo
+ exit 1;
+fi
+
+if [ $VERBOSE -eq 1 ]; then
+ echo "basedir is: $BASDIR";
+ echo "toolsdir is: $TOOLSDIR":
+fi
+
+#install mpi
+cd $BASEDIR
+wget http://www.mpich.org/static/downloads/3.0.4/mpich-3.0.4.tar.gz
+tar -xzf mpich-3.0.4.tar.gz
+cd mpich-3.0.4
+./configure --prefix=$TOOLSDIR/dest-mpich-3.0.4 --with-device=ch3:sock --disable-f77 --disable-fc
+make
+make check
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of MPI"
+fi
+
+# See if need to install protobufs tested with version 2.3.0
+protoc --version
+
+# install bison
+cd $BASEDIR
+wget http://ftp.gnu.org/gnu/bison/bison-3.0.tar.gz
+tar -xzf bison-3.0.tar.gz
+cd bison-3.0
+./configure --prefix=$TOOLSDIR/bison_3_linux
+make
+make check
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of BISON"
+fi
+
+# install udis
+cd $BASEDIR
+wget http://sourceforge.net/projects/udis86/files/udis86/1.7/udis86-1.7.2.tar.gz
+tar -xzf udis86-1.7.2.tar.gz
+cd udis86-1.7.2
+./configure --prefix=$TOOLSDIR/udis86-1.7.2 --enable-shared
+make
+make check
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of UDIS"
+fi
+
+#install LLVM
+
+cd $BASEDIR
+wget http://llvm.org/releases/3.2/llvm-3.2.src.tar.gz
+tar xzf llvm-3.2.src.tar.gz
+
+export MY_UDIS_INSTALL_DIR=$TOOLSDIR/udis86-1.7.2
+export MY_LLVM_INSTALL_DIR=$TOOLSDIR/dest-llvm-3.2/
+export MY_LLVM_SRC_DIR=$BASEDIR/llvm-3.2.src
+export MY_LLVM_OBJ_DIR=$BASEDIR/llvm-3.2.obj/
+export LD_LIBRARY_PATH=$MY_UDIS_INSTALL_DIR/lib:$LD_LIBRARY_PATH
+export C_INCLUDE_PATH=$MY_UDIS_INSTALL_DIR/include
+export CPATH=$MY_UDIS_INSTALL_DIR/include
+
+# Build release version
+mkdir -p $MY_LLVM_OBJ_DIR/release
+cd $MY_LLVM_OBJ_DIR/release
+
+$MY_LLVM_SRC_DIR/configure --prefix=$MY_LLVM_INSTALL_DIR/release \
+ --enable-shared --enable-targets=x86,x86_64,cpp \
+ --with-udis86=$MY_UDIS_INSTALL_DIR/lib \
+ CFLAGS=-fgnu89-inline
+
+make libs-only
+make install-libs
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of LLVM release"
+fi
+
+# Build debug version
+mkdir -p $MY_LLVM_OBJ_DIR/debug
+cd $MY_LLVM_OBJ_DIR/debug
+
+$MY_LLVM_SRC_DIR/configure --prefix=$MY_LLVM_INSTALL_DIR/debug \
+ --enable-optimized --enable-jit \
+ --enable-debug-runtime --enable-debug-symbols \
+ --enable-shared --enable-targets=x86,x86_64,cpp \
+ --with-udis86=$MY_UDIS_INSTALL_DIR/lib \
+ CFLAGS=-fgnu89-inline
+
+make libs-only
+make install-libs
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of LLVM debug"
+fi
+
+# Build ICU
+cd $BASEDIR
+wget http://download.icu-project.org/files/icu4c/4.4/icu4c-4_4-src.tgz
+tar -xzf icu4c-4_4-src.tgz
+cd icu/source
+./configure --with-library-suffix=Nv44 --prefix=$TOOLSDIR/icu4.4/linux64
+make
+make check
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of ICU"
+fi
+
+# make zookeeper
+cd $BASEDIR
+wget https://archive.apache.org/dist/zookeeper/zookeeper-3.4.5/zookeeper-3.4.5.tar.gz
+tar -zxf zookeeper-3.4.5.tar.gz
+cd zookeeper-3.4.5/src/c
+./configure --prefix=$TOOLSDIR/zookeeper-3.4.5
+make
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of ZOOKEEPER"
+fi
+
+# make thrift
+cd $BASEDIR
+wget http://archive.apache.org/dist/thrift/0.9.0/thrift-0.9.0.tar.gz
+tar -xzf thrift-0.9.0.tar.gz
+cd thrift-0.9.0
+./configure --prefix=$TOOLSDIR/thrift-0.9.0 -without-qt
+make
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of THRIFT"
+fi
--- End diff --
Really good to have this script. I hope we keep it up to date as we migrate to new versions!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1513] Trafodion build...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/109#discussion_r41559280
--- Diff: install/traf_tools_setup.sh ---
@@ -0,0 +1,249 @@
+#!/bin/bash
+#
+# @@@ START COPYRIGHT @@@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# @@@ END COPYRIGHT @@@
+#
+
+# -----------------------------------------------------------------------------
+# script: traf_tools_setup
+#
+# Helper script that downloads and installs dependent tools required by
+# Apache Trafodion to build from a source distribution
+#
+# Tools installed:
+#
+# MPICH: Implementation of the Message Passing Interface (MPI) standard. For use
+# in Trafodion, MPICH must be built to force sockets to be used in both
+# internode and intranode message passing.
+# Bison: General-purpose parser generator.
+# Udis86: Minimalistic disassembler library (libudis86) for the x86 class of
+# instruction set architectures.
+# LLVM: Collection of modular and reusable compiler and toolchain technologies.
+# ICU: C/C++ and Java libraries providing Unicode and Globalization support for
+# software applications.
+# Zookeeper: Coordination service for distributed applications. It exposes
+# common services such as naming, configuration management, synchronization,
+# and group services.
+# Thrift: Communications and data serialization tool
+#
+# Script can be modified to meet the needs of your environment
+# May need root or SUDO access to install tools in desired location
+# -----------------------------------------------------------------------------
+
+function Usage {
+ echo
+ echo "Usage: $0 -d <downloaddir> -i <installdir>"
+ echo
+ echo " -d <downloaddir> - location of download directory"
+ echo " -i <installdir> - location of install directory"
+ echo " -v - verbose mode"
+ echo
+ echo " -h - help"
+ echo
+ echo "example: traf_tools_setup.sh -d /home/userx/download -i /home/userx/tools"
+ echo
+}
+
+ VERBOSE=0
+ TOOLSDIR=
+ BASEDIR=
+
+ while getopts "d:i:hv" arg
+ do
+ case $arg in
+ d)
+ BASEDIR=${OPTARG};
+ ;;
+ h)
+ Usage;
+ exit 1;
+ ;;
+ i)
+ TOOLSDIR=${OPTARG};
+ ;;
+ v)
+ VERBOSE=1;
+ ;;
+ *)
+ Usage;
+ exit 1;
+ ;;
+ esac
+ done
+
+
+if [ "$BASEDIR" == "" ]; then
+ echo
+ echo "ERROR: download directory (-d) is not specified"
+ Usage;
+ exit 1;
+fi
+
+if [ ! -d "$BASEDIR" ]; then
+ echo
+ echo "ERROR: download directory ($BASEDIR) does not exist"
+ echo
+ exit 1;
+fi
+
+if [ "$TOOLSDIR" == "" ]; then
+ echo
+ echo "ERROR: install directory (-t) is not specified"
+ Usage;
+ exit 1;
+fi
+
+if [ ! -d "$TOOLSDIR" ]; then
+ echo
+ echo "ERROR: install directory ($TOOLSDIR) does not exist"
--- End diff --
Since this sets up $TOOLSDIR, should we create it here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1513] Trafodion build...
Posted by robertamarton <gi...@git.apache.org>.
Github user robertamarton commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/109#discussion_r41569557
--- Diff: install/traf_tools_setup.sh ---
@@ -0,0 +1,249 @@
+#!/bin/bash
+#
+# @@@ START COPYRIGHT @@@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# @@@ END COPYRIGHT @@@
+#
+
+# -----------------------------------------------------------------------------
+# script: traf_tools_setup
+#
+# Helper script that downloads and installs dependent tools required by
+# Apache Trafodion to build from a source distribution
+#
+# Tools installed:
+#
+# MPICH: Implementation of the Message Passing Interface (MPI) standard. For use
+# in Trafodion, MPICH must be built to force sockets to be used in both
+# internode and intranode message passing.
+# Bison: General-purpose parser generator.
+# Udis86: Minimalistic disassembler library (libudis86) for the x86 class of
+# instruction set architectures.
+# LLVM: Collection of modular and reusable compiler and toolchain technologies.
+# ICU: C/C++ and Java libraries providing Unicode and Globalization support for
+# software applications.
+# Zookeeper: Coordination service for distributed applications. It exposes
+# common services such as naming, configuration management, synchronization,
+# and group services.
+# Thrift: Communications and data serialization tool
+#
+# Script can be modified to meet the needs of your environment
+# May need root or SUDO access to install tools in desired location
+# -----------------------------------------------------------------------------
+
+function Usage {
+ echo
+ echo "Usage: $0 -d <downloaddir> -i <installdir>"
+ echo
+ echo " -d <downloaddir> - location of download directory"
+ echo " -i <installdir> - location of install directory"
+ echo " -v - verbose mode"
+ echo
+ echo " -h - help"
+ echo
+ echo "example: traf_tools_setup.sh -d /home/userx/download -i /home/userx/tools"
+ echo
+}
+
+ VERBOSE=0
+ TOOLSDIR=
+ BASEDIR=
+
+ while getopts "d:i:hv" arg
+ do
+ case $arg in
+ d)
+ BASEDIR=${OPTARG};
+ ;;
+ h)
+ Usage;
+ exit 1;
+ ;;
+ i)
+ TOOLSDIR=${OPTARG};
+ ;;
+ v)
+ VERBOSE=1;
+ ;;
+ *)
+ Usage;
+ exit 1;
+ ;;
+ esac
+ done
+
+
+if [ "$BASEDIR" == "" ]; then
+ echo
+ echo "ERROR: download directory (-d) is not specified"
+ Usage;
+ exit 1;
+fi
+
+if [ ! -d "$BASEDIR" ]; then
+ echo
+ echo "ERROR: download directory ($BASEDIR) does not exist"
+ echo
+ exit 1;
+fi
+
+if [ "$TOOLSDIR" == "" ]; then
+ echo
+ echo "ERROR: install directory (-t) is not specified"
+ Usage;
+ exit 1;
+fi
+
+if [ ! -d "$TOOLSDIR" ]; then
+ echo
+ echo "ERROR: install directory ($TOOLSDIR) does not exist"
--- End diff --
Probably. I can create it if it doesn't exit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1513] Trafodion build...
Posted by robertamarton <gi...@git.apache.org>.
Github user robertamarton commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/109#discussion_r41568928
--- Diff: install/traf_tools_setup.sh ---
@@ -0,0 +1,249 @@
+#!/bin/bash
+#
+# @@@ START COPYRIGHT @@@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# @@@ END COPYRIGHT @@@
+#
+
+# -----------------------------------------------------------------------------
+# script: traf_tools_setup
+#
+# Helper script that downloads and installs dependent tools required by
+# Apache Trafodion to build from a source distribution
+#
+# Tools installed:
+#
+# MPICH: Implementation of the Message Passing Interface (MPI) standard. For use
+# in Trafodion, MPICH must be built to force sockets to be used in both
+# internode and intranode message passing.
+# Bison: General-purpose parser generator.
+# Udis86: Minimalistic disassembler library (libudis86) for the x86 class of
+# instruction set architectures.
+# LLVM: Collection of modular and reusable compiler and toolchain technologies.
+# ICU: C/C++ and Java libraries providing Unicode and Globalization support for
+# software applications.
+# Zookeeper: Coordination service for distributed applications. It exposes
+# common services such as naming, configuration management, synchronization,
+# and group services.
+# Thrift: Communications and data serialization tool
+#
+# Script can be modified to meet the needs of your environment
+# May need root or SUDO access to install tools in desired location
+# -----------------------------------------------------------------------------
+
+function Usage {
+ echo
+ echo "Usage: $0 -d <downloaddir> -i <installdir>"
+ echo
+ echo " -d <downloaddir> - location of download directory"
+ echo " -i <installdir> - location of install directory"
+ echo " -v - verbose mode"
+ echo
+ echo " -h - help"
+ echo
+ echo "example: traf_tools_setup.sh -d /home/userx/download -i /home/userx/tools"
+ echo
+}
+
+ VERBOSE=0
+ TOOLSDIR=
+ BASEDIR=
+
+ while getopts "d:i:hv" arg
+ do
+ case $arg in
+ d)
+ BASEDIR=${OPTARG};
+ ;;
+ h)
+ Usage;
+ exit 1;
+ ;;
+ i)
+ TOOLSDIR=${OPTARG};
+ ;;
+ v)
+ VERBOSE=1;
+ ;;
+ *)
+ Usage;
+ exit 1;
+ ;;
+ esac
+ done
+
+
+if [ "$BASEDIR" == "" ]; then
+ echo
+ echo "ERROR: download directory (-d) is not specified"
+ Usage;
+ exit 1;
+fi
+
+if [ ! -d "$BASEDIR" ]; then
+ echo
+ echo "ERROR: download directory ($BASEDIR) does not exist"
+ echo
+ exit 1;
+fi
+
+if [ "$TOOLSDIR" == "" ]; then
+ echo
+ echo "ERROR: install directory (-t) is not specified"
+ Usage;
+ exit 1;
+fi
+
+if [ ! -d "$TOOLSDIR" ]; then
+ echo
+ echo "ERROR: install directory ($TOOLSDIR) does not exist"
+ echo
+ exit 1;
+fi
+
+if [ $VERBOSE -eq 1 ]; then
+ echo "basedir is: $BASDIR";
+ echo "toolsdir is: $TOOLSDIR":
+fi
+
+#install mpi
+cd $BASEDIR
+wget http://www.mpich.org/static/downloads/3.0.4/mpich-3.0.4.tar.gz
+tar -xzf mpich-3.0.4.tar.gz
+cd mpich-3.0.4
+./configure --prefix=$TOOLSDIR/dest-mpich-3.0.4 --with-device=ch3:sock --disable-f77 --disable-fc
+make
+make check
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of MPI"
+fi
+
+# See if need to install protobufs tested with version 2.3.0
+protoc --version
+
+# install bison
+cd $BASEDIR
+wget http://ftp.gnu.org/gnu/bison/bison-3.0.tar.gz
+tar -xzf bison-3.0.tar.gz
+cd bison-3.0
+./configure --prefix=$TOOLSDIR/bison_3_linux
+make
+make check
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of BISON"
+fi
+
+# install udis
+cd $BASEDIR
+wget http://sourceforge.net/projects/udis86/files/udis86/1.7/udis86-1.7.2.tar.gz
+tar -xzf udis86-1.7.2.tar.gz
+cd udis86-1.7.2
+./configure --prefix=$TOOLSDIR/udis86-1.7.2 --enable-shared
+make
+make check
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of UDIS"
+fi
+
+#install LLVM
+
+cd $BASEDIR
+wget http://llvm.org/releases/3.2/llvm-3.2.src.tar.gz
+tar xzf llvm-3.2.src.tar.gz
+
+export MY_UDIS_INSTALL_DIR=$TOOLSDIR/udis86-1.7.2
+export MY_LLVM_INSTALL_DIR=$TOOLSDIR/dest-llvm-3.2/
+export MY_LLVM_SRC_DIR=$BASEDIR/llvm-3.2.src
+export MY_LLVM_OBJ_DIR=$BASEDIR/llvm-3.2.obj/
+export LD_LIBRARY_PATH=$MY_UDIS_INSTALL_DIR/lib:$LD_LIBRARY_PATH
+export C_INCLUDE_PATH=$MY_UDIS_INSTALL_DIR/include
+export CPATH=$MY_UDIS_INSTALL_DIR/include
+
+# Build release version
+mkdir -p $MY_LLVM_OBJ_DIR/release
+cd $MY_LLVM_OBJ_DIR/release
+
+$MY_LLVM_SRC_DIR/configure --prefix=$MY_LLVM_INSTALL_DIR/release \
+ --enable-shared --enable-targets=x86,x86_64,cpp \
+ --with-udis86=$MY_UDIS_INSTALL_DIR/lib \
+ CFLAGS=-fgnu89-inline
+
+make libs-only
+make install-libs
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of LLVM release"
+fi
+
+# Build debug version
+mkdir -p $MY_LLVM_OBJ_DIR/debug
+cd $MY_LLVM_OBJ_DIR/debug
+
+$MY_LLVM_SRC_DIR/configure --prefix=$MY_LLVM_INSTALL_DIR/debug \
+ --enable-optimized --enable-jit \
+ --enable-debug-runtime --enable-debug-symbols \
+ --enable-shared --enable-targets=x86,x86_64,cpp \
+ --with-udis86=$MY_UDIS_INSTALL_DIR/lib \
+ CFLAGS=-fgnu89-inline
+
+make libs-only
+make install-libs
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of LLVM debug"
+fi
+
+# Build ICU
+cd $BASEDIR
+wget http://download.icu-project.org/files/icu4c/4.4/icu4c-4_4-src.tgz
+tar -xzf icu4c-4_4-src.tgz
+cd icu/source
+./configure --with-library-suffix=Nv44 --prefix=$TOOLSDIR/icu4.4/linux64
+make
+make check
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of ICU"
+fi
+
+# make zookeeper
+cd $BASEDIR
+wget https://archive.apache.org/dist/zookeeper/zookeeper-3.4.5/zookeeper-3.4.5.tar.gz
+tar -zxf zookeeper-3.4.5.tar.gz
+cd zookeeper-3.4.5/src/c
+./configure --prefix=$TOOLSDIR/zookeeper-3.4.5
+make
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of ZOOKEEPER"
+fi
+
+# make thrift
+cd $BASEDIR
+wget http://archive.apache.org/dist/thrift/0.9.0/thrift-0.9.0.tar.gz
+tar -xzf thrift-0.9.0.tar.gz
+cd thrift-0.9.0
+./configure --prefix=$TOOLSDIR/thrift-0.9.0 -without-qt
+make
+make install
+if [ $VERBOSE -eq 1 ]; then
+ echo "STEP completed: installation of THRIFT"
+fi
--- End diff --
I am able to build with protoc version 2.3.0. Apparently you need 2.5.0 to build libhdfs... I will write a JIRA and make these changes in a future delivery - In build instructions describe how to install later versions of protocol buffers and QT. I can also update the script to install version 2.5.0 of protocol buffers.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1513] Trafodion build...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/109#discussion_r41558135
--- Diff: core/sqf/sql/scripts/install_hadoop_regr_test_env ---
@@ -134,12 +134,12 @@ fi
if [ ! -f $MY_LOCAL_SW_DIST/${TPCDS_ZIP} ]; then
# Right now there is no URL to download this tool automatically
- echo '*********************** ERROR *************************'
- echo "Please download the tpcds_kit.zip file from"
- echo "http://www.tpc.org/tpcds/dsgen-download-request.asp"
- echo "and place it into a directory pointed to by the MY_LOCAL_SW_DIST"
- echo "environment variable. Then retry installing. Sorry, this is due"
- echo "to the TPC wanting your email address."
+ # Please download the tpcds_kit.zip file from"
+ # "http://www.tpc.org/tpcds/dsgen-download-request.asp"
+ # "and place it into a directory pointed to by the MY_LOCAL_SW_DIST"
+ # "environment variable. Then retry installing. Sorry, this is due"
+ # "to the TPC wanting your email address."
+ echo "The testware tpcds_kit.zip does not exist and will not be installed"
exit 99
fi
--- End diff --
Also, unfortunately, the TPC rewrote their test kit and we would need to update our build procedure to use the new files. So, following these instructions will actually no longer work. We are all using an old, saved copy of tpcds_kit.zip.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1513] Trafodion build...
Posted by robertamarton <gi...@git.apache.org>.
Github user robertamarton commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/109#discussion_r41567991
--- Diff: core/sqf/sqenvcom.sh ---
@@ -95,17 +95,20 @@ cpucnt=$(grep processor /proc/cpuinfo | wc -l)
# no number means unlimited, and will swamp the system
export MAKEFLAGS="-j$cpucnt"
+# For now set up the TOOLSDIR, it may be overwritten later when the
+# .trafodion configuration file is loaded.
if [[ -n "$CLUSTERNAME" ]]; then
- export MY_ROOT=/opt/hp
- export TOOLSDIR=${TOOLSDIR:-/home/tools}
+ if [ -z "$TOOLSDIR" ]; then
+ export TOOLSDIR=${TOOLSDIR:-/home/tools}
+ fi
--- End diff --
Okay, I see what Steve is saying, I can simplify the code
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---