[impala] branch master updated: IMPALA-9005: distcc server bootstrapping improvements

[ta...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

tarmstrong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git


The following commit(s) were added to refs/heads/master by this push:
     new ea79ba6  IMPALA-9005: distcc server bootstrapping improvements
ea79ba6 is described below

commit ea79ba66ece6c0793b30cec2b0114267f7c29728
Author: Tim Armstrong <ta...@cloudera.com>
AuthorDate: Thu Oct 3 14:39:12 2019 -0700

    IMPALA-9005: distcc server bootstrapping improvements
    
    Adds a script distcc_server_bootstrap.sh that can be run
    in a standalone way on an ubuntu 16.04 server.
    
    A next step could be to build a docker image that can be
    deployed as an impala distcc server.
    
    Testing:
    Ran in an ubuntu 16.04 docker image and then confirmed I could
    add it to my BUILD_FARM and compile things with it.
    
    Change-Id: I9e382e90f68a5a2fe2917184fa4603bed492a308
    Reviewed-on: http://gerrit.cloudera.org:8080/14363
    Reviewed-by: Joe McDonnell <jo...@cloudera.com>
    Tested-by: Tim Armstrong <ta...@cloudera.com>
---
 bin/distcc/distcc_server_bootstrap.sh | 75 +++++++++++++++++++++++++++++++++++
 bin/distcc/distcc_server_setup.sh     | 11 +++--
 2 files changed, 83 insertions(+), 3 deletions(-)

diff --git a/bin/distcc/distcc_server_bootstrap.sh b/bin/distcc/distcc_server_bootstrap.sh
new file mode 100755
index 0000000..2f931f5
--- /dev/null
+++ b/bin/distcc/distcc_server_bootstrap.sh
@@ -0,0 +1,75 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This script automates setup of distcc servers on Ubuntu. This
+# script can be run outside of an Impala repository and will
+# bootstrap the server from scratch.
+#
+# Usage:
+# ------
+# This script must be run as root. Arguments are forwarded to
+# distcc_server_setup.sh to specify IP ranges, etc.
+
+set -eu -o pipefail
+
+if [[ "$USER" != "root" ]]; then
+  echo "Must be run as root."
+  exit 1
+fi
+
+LSB_ID=$(lsb_release -is)
+LSB_VERSION=$(lsb_release -rs)
+if [[ "$LSB_ID" == Ubuntu ]]; then
+  if ! [[ $LSB_VERSION == 16.04 ]]; then
+    echo "This script only supports Ubuntu 16.04" >&2
+    exit 1
+  fi
+fi
+
+# Install basic packages required to get to distcc_server_setup.sh
+# git: needed to clone the repo
+# openjdk: needed because JAVA_HOME is checked by impala-config.sh
+# distcc: needed to set up the distccd user.
+# libsasl2-dev: needed to bootstrap python virtualenv
+apt-get install -y git openjdk-8-jdk-headless distcc libsasl2-dev
+
+mkdir -p /opt/distcc
+chown distccd /opt/distcc
+cd /opt/distcc
+
+# Toolchain must be owned by distcc so that it can execute binaries.
+# So check out the Impala repo and download the toolchain as the
+# distccd user.
+sudo -u distccd -H bash <<"EOF"
+  set -euo pipefail
+  set -x
+  # Set HOME as workaround for ccache trying to access /.ccache
+  HOME=$(pwd)
+  if [[ ! -d impala ]]; then
+    git clone https://github.com/apache/impala.git
+  fi
+  cd impala
+  echo 'export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64' >> bin/impala-config-local.sh
+  export IMPALA_HOME=$(pwd)
+ . bin/impala-config.sh
+ ./infra/python/deps/download_requirements
+ DOWNLOAD_CDH_COMPONENTS=false ./bin/bootstrap_toolchain.py
+EOF
+
+(cd impala && ./bin/distcc/distcc_server_setup.sh "$@")
diff --git a/bin/distcc/distcc_server_setup.sh b/bin/distcc/distcc_server_setup.sh
index 6b514e7..83bce46 100755
--- a/bin/distcc/distcc_server_setup.sh
+++ b/bin/distcc/distcc_server_setup.sh
@@ -23,6 +23,10 @@
 #
 # Usage:
 # ------
+# This script assumes that the Impala repository is checked out and that the
+# toolchain has been bootstrapped in the toolchain subdirectory of the impala
+# repository.
+#
 # The script expects to be run as root and requires that the user specify a range of
 # IPs that it should accept connections from. E.g. To configure a distcc server that
 # accepts connections from the 172.* private IP address block. The IP address argument
@@ -39,6 +43,10 @@
 # CCACHE_DIR: directory to use for distccd's ccache.
 # CCACHE_SIZE: size of ccache, passed to ccache's -M option
 set -eu -o pipefail
+
+# Find the absolute path to repo root.
+export IMPALA_HOME=$(cd "$(dirname "$0")/../.."; pwd)
+
 . $IMPALA_HOME/bin/report_build_error.sh
 setup_report_build_error
 
@@ -49,9 +57,6 @@ fi
 set -x
 ALLOWED_NETS=$1
 
-# Find the absolute path to repo root.
-IMPALA_HOME=$(cd "$(dirname "$0")/../.."; pwd)
-
 LSB_ID=$(lsb_release -is)
 LSB_VERSION=$(lsb_release -rs)
 if [[ "$LSB_ID" == Ubuntu ]]; then