You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ka...@apache.org on 2021/01/09 21:30:35 UTC
[airflow] branch master updated: Add classic installation scripts
for additional tools (#13587)
This is an automated email from the ASF dual-hosted git repository.
kamilbregula pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/master by this push:
new 1d1ea52 Add classic installation scripts for additional tools (#13587)
1d1ea52 is described below
commit 1d1ea523c58eb8fd60dfc5cefd69888c8331102c
Author: Kamil BreguĊa <mi...@users.noreply.github.com>
AuthorDate: Sat Jan 9 22:30:20 2021 +0100
Add classic installation scripts for additional tools (#13587)
---
BREEZE.rst | 48 +++++++++++++------
Dockerfile.ci | 4 +-
scripts/in_container/bin/install_gcloud.sh | 67 +++++++++++++++++++++++++++
scripts/in_container/bin/install_imgcat.sh | 40 ++++++++++++++++
scripts/in_container/bin/install_java.sh | 61 ++++++++++++++++++++++++
scripts/in_container/bin/install_kubectl.sh | 48 +++++++++++++++++++
scripts/in_container/bin/install_terraform.sh | 48 +++++++++++++++++++
scripts/in_container/run_cli_tool.sh | 25 +---------
8 files changed, 301 insertions(+), 40 deletions(-)
diff --git a/BREEZE.rst b/BREEZE.rst
index f78c98c..3ecf770 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -428,18 +428,44 @@ way as when you enter the environment. You can do it multiple times and open as
</div>
-CLIs for cloud providers
-------------------------
+Additional tools
+----------------
+
+To shrink a Docker image, not all tools are installed in the Docker image. Some of you have to install
+or yourself, but we made sure it was an easy process.
+
+Additional tools are installed in ``/files/bin``. This path is added to ``$PATH``, so you shell will
+automatically autocomplete files that are in that directory. You can also keep your tools in this directory
+if you need to.
+
+**Installation scripts**
-For development convenience we installed simple wrappers for the most common cloud providers CLIs. Those
-CLIs are not installed when you build or pull the image - they will be downloaded as docker images
-the first time you attempt to use them. It is downloaded and executed in your host's docker engine so once
-it is downloaded, it will stay until you remove the downloaded images from your host container.
+For the development convenience, we also provided install script for common used tools. They are
+installed to ``/files/opt/``, so they are preserved after restarting the Breeze environment. Each script
+is also available in ``$PATH``, so just type ``install_<TAB>`` to get a list of tools.
+
+Currently available scripts:
+
+* ``install_gcloud.sh`` - installs `the Google Cloud SDK <https://cloud.google.com/sdk>`__ including
+ ``gcloud``, ``gsutil``.
+* ``install_imgcat.sh`` - installs `imgcat - Inline Images Protocol <https://iterm2.com/documentation-images.html>`__
+ for iTem2 (Mac OS only)
+* ``install_java.sh`` - installs `the OpenJDK 8u41 <https://openjdk.java.net/>`__
+* ``install_kubectl.sh`` - installs `the Kubernetes command-line tool, kubectl <https://kubernetes.io/docs/reference/kubectl/kubectl/>`__
+* ``install_terraform.sh`` - installs `the Terraform <https://www.terraform.io/docs/index.html>`__
+
+**Docker wrappers**
+
+For tools that do not provide a simple method of installation in the custom directory, we provided simple
+wrappers for docker. Those CLIs are not installed when you build or pull the image - they will be downloaded
+as docker images the first time you attempt to use them. It is downloaded and executed in your host's
+docker engine so once it is downloaded, it will stay until you remove the downloaded images from your host
+container.
For each of those CLI credentials are taken (automatically) from the credentials you have defined in
-your ${HOME} directory on host.
+your ``${HOME}`` directory on host.
-Those tools also have host Airflow source directory mounted in /opt/airflow path
+Those tools also have host Airflow source directory mounted in ``/opt/airflow`` path
so you can directly transfer files to/from your airflow host sources.
Those are currently installed CLIs (they are available as aliases to the docker commands):
@@ -451,12 +477,6 @@ Those are currently installed CLIs (they are available as aliases to the docker
+-----------------------+----------+-------------------------------------------------+-------------------+
| Microsoft Azure | az | mcr.microsoft.com/azure-cli:latest | .azure |
+-----------------------+----------+-------------------------------------------------+-------------------+
-| Google Cloud | bq | gcr.io/google.com/cloudsdktool/cloud-sdk:latest | .config/gcloud |
-| +----------+-------------------------------------------------+-------------------+
-| | gcloud | gcr.io/google.com/cloudsdktool/cloud-sdk:latest | .config/gcloud |
-| +----------+-------------------------------------------------+-------------------+
-| | gsutil | gcr.io/google.com/cloudsdktool/cloud-sdk:latest | .config/gcloud |
-+-----------------------+----------+-------------------------------------------------+-------------------+
For each of the CLIs we have also an accompanying ``*-update`` alias (for example ``aws-update``) which
will pull the latest image for the tool. Note that all Google Cloud tools are served by one
diff --git a/Dockerfile.ci b/Dockerfile.ci
index b18656b..672ab67 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -212,7 +212,7 @@ RUN mkdir -p /opt/bats/lib/bats-file \
&& curl -sSL https://github.com/bats-core/bats-file/archive/v${BATS_FILE_VERSION}.tar.gz -o /tmp/bats-file.tgz \
&& tar -zxf /tmp/bats-file.tgz -C /opt/bats/lib/bats-file --strip 1 && rm -rf /tmp/*
-RUN echo "export PATH=/opt/bats/bin:${PATH}" >> /root/.bashrc
+ENV PATH "/opt/bats/bin/:${PATH}"
# Additional scripts for managing BATS addons
COPY scripts/docker/load.bash /opt/bats/lib/
@@ -418,7 +418,7 @@ RUN if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
WORKDIR ${AIRFLOW_SOURCES}
-ENV PATH="${HOME}:${PATH}"
+ENV PATH "/files/bin/:/opt/airflow/scripts/in_container/bin/:${HOME}:${PATH}"
# Needed to stop Gunicorn from crashing when /tmp is now mounted from host
ENV GUNICORN_CMD_ARGS="--worker-tmp-dir /dev/shm/"
diff --git a/scripts/in_container/bin/install_gcloud.sh b/scripts/in_container/bin/install_gcloud.sh
new file mode 100755
index 0000000..755369e
--- /dev/null
+++ b/scripts/in_container/bin/install_gcloud.sh
@@ -0,0 +1,67 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -euo pipefail
+
+if command -v gcloud; then
+ echo 'The "gcloud" command found. Installation not needed.'
+ exit 1
+fi
+
+CLOUD_SDK_VERSION=322.0.0
+DOWNLOAD_URL="https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-${CLOUD_SDK_VERSION}-linux-x86_64.tar.gz"
+INSTALL_DIR="/files/opt/google-cloud-sdk"
+
+if [[ -e ${INSTALL_DIR} ]]; then
+ echo "The install directory (${INSTALL_DIR}) already exists. This may mean Cloud SDK is already installed."
+ echo "Please delete this directory to start the installation."
+ exit 1
+fi
+
+TMP_DIR="$(mktemp -d)"
+# shellcheck disable=SC2064
+trap "rm -rf ${TMP_DIR}" EXIT
+
+mkdir -p "${INSTALL_DIR}"
+echo "Downloading from ${DOWNLOAD_URL}"
+curl -# --fail "${DOWNLOAD_URL}" --output "${TMP_DIR}/google-cloud-sdk.tar.gz"
+echo "Extracting archive"
+tar xzf "${TMP_DIR}/google-cloud-sdk.tar.gz" -C "${INSTALL_DIR}" --strip-components=1
+
+"${INSTALL_DIR}/install.sh" \
+ --bash-completion=false \
+ --path-update=false \
+ --usage-reporting=false \
+ --additional-components alpha beta \
+ --quiet
+
+echo 'Symlinking executables files to /files/bin'
+mkdir -p "/files/bin/"
+while IPS='' read -r line; do
+ BIN_NAME="$(basename "${line}")"
+ ln -s "${line}" "/files/bin/${BIN_NAME}"
+done < <(find "${INSTALL_DIR}/bin/" -type f)
+
+# Sanity check
+if ! command -v gcloud > /dev/null; then
+ echo 'Installation failed. The command "gcloud" was not found.'
+ exit 1
+fi
+
+echo 'Installation complete.'
diff --git a/scripts/in_container/bin/install_imgcat.sh b/scripts/in_container/bin/install_imgcat.sh
new file mode 100755
index 0000000..ac0542d
--- /dev/null
+++ b/scripts/in_container/bin/install_imgcat.sh
@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -euo pipefail
+
+if command -v imgcat; then
+ echo 'The "imgcat" command found. Installation not needed.'
+ exit 1
+fi
+
+DOWNLOAD_URL="https://iterm2.com/utilities/imgcat"
+
+mkdir -p "/files/bin/"
+echo "Downloading from ${DOWNLOAD_URL}"
+curl -# --fail "${DOWNLOAD_URL}" --output "/files/bin/imgcat"
+chmod +x "/files/bin/imgcat"
+
+# Sanity check
+if ! command -v imgcat > /dev/null; then
+ echo 'Installation failed. The command "imgcat" was not found.'
+ exit 1
+fi
+
+echo 'Installation complete.'
diff --git a/scripts/in_container/bin/install_java.sh b/scripts/in_container/bin/install_java.sh
new file mode 100755
index 0000000..8bdf48b
--- /dev/null
+++ b/scripts/in_container/bin/install_java.sh
@@ -0,0 +1,61 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -euo pipefail
+
+if command -v java; then
+ echo 'The "java" command found. Installation not needed.'
+ exit 1
+fi
+
+DOWNLOAD_URL='https://download.java.net/openjdk/jdk8u41/ri/openjdk-8u41-b04-linux-x64-14_jan_2020.tar.gz'
+INSTALL_DIR="/files/opt/java"
+
+if [[ -e ${INSTALL_DIR} ]]; then
+ echo "The install directory (${INSTALL_DIR}) already exists. This may mean java is already installed."
+ echo "Please delete this directory to start the installation."
+ exit 1
+fi
+
+TMP_DIR="$(mktemp -d)"
+# shellcheck disable=SC2064
+trap "rm -rf ${TMP_DIR}" EXIT
+
+mkdir -p "${INSTALL_DIR}"
+
+echo "Downloading from ${DOWNLOAD_URL}"
+curl -# --fail "${DOWNLOAD_URL}" --output "${TMP_DIR}/openjdk.tar.gz"
+
+echo "Extracting archive"
+tar xzf "${TMP_DIR}/openjdk.tar.gz" -C "${INSTALL_DIR}" --strip-components=1
+
+echo 'Symlinking executables files to /files/bin'
+mkdir -p "/files/bin/"
+while IPS='' read -r line; do
+ BIN_NAME="$(basename "${line}")"
+ ln -s "${line}" "/files/bin/${BIN_NAME}"
+done < <(find "${INSTALL_DIR}/bin/" -type f)
+
+# Sanity check
+if ! command -v java > /dev/null; then
+ echo 'Installation failed. The command "java" was not found.'
+ exit 1
+fi
+
+echo 'Installation complete.'
diff --git a/scripts/in_container/bin/install_kubectl.sh b/scripts/in_container/bin/install_kubectl.sh
new file mode 100755
index 0000000..ae7916d
--- /dev/null
+++ b/scripts/in_container/bin/install_kubectl.sh
@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -euo pipefail
+
+if command -v kubectl; then
+ echo 'The "kubectl" command found. Installation not needed.'
+ exit 1
+fi
+
+KUBECTL_VERSION="$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)"
+DOWNLOAD_URL="https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl"
+BIN_PATH="/files/bin/kubectl"
+
+if [[ -e ${BIN_PATH} ]]; then
+ echo "The binary file (${BIN_PATH}) already exists. This may mean kubectl is already installed."
+ echo "Please delete this file to start the installation."
+ exit 1
+fi
+
+mkdir -p "/files/bin/"
+echo "Downloading from ${DOWNLOAD_URL}"
+curl -# --fail "${DOWNLOAD_URL}" --output "${BIN_PATH}"
+chmod +x "${BIN_PATH}"
+
+# Sanity check
+if ! command -v kubectl > /dev/null; then
+ echo 'Installation failed. The command "kubectl" was not found.'
+ exit 1
+fi
+
+echo 'Installation complete.'
diff --git a/scripts/in_container/bin/install_terraform.sh b/scripts/in_container/bin/install_terraform.sh
new file mode 100755
index 0000000..6d86bdf
--- /dev/null
+++ b/scripts/in_container/bin/install_terraform.sh
@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -euo pipefail
+
+if command -v terraform; then
+ echo 'The "terraform" command found. Installation not needed.'
+ exit 1
+fi
+
+TERRAFORM_VERSION="0.14.4"
+TERRAFORM_BASE_URL="https://releases.hashicorp.com/terraform"
+TERRAFORM_ZIP="terraform_${TERRAFORM_VERSION}_$(uname | tr '[:upper:]' '[:lower:]')_amd64.zip"
+DOWNLOAD_URL="${TERRAFORM_BASE_URL}/${TERRAFORM_VERSION}/${TERRAFORM_ZIP}"
+TMP_DIR="$(mktemp -d)"
+
+# shellcheck disable=SC2064
+trap "rm -rf ${TMP_DIR}" EXIT
+
+mkdir -p "/files/bin/"
+echo "Downloading from ${DOWNLOAD_URL}"
+curl -# --fail "${DOWNLOAD_URL}" --output "${TMP_DIR}/terraform.zip"
+echo "Extracting archive"
+unzip "${TMP_DIR}/terraform.zip" -d /files/bin
+
+# Sanity check
+if ! command -v terraform > /dev/null; then
+ echo 'Installation failed. The command "terraform" was not found.'
+ exit 1
+fi
+
+echo 'Installation complete.'
diff --git a/scripts/in_container/run_cli_tool.sh b/scripts/in_container/run_cli_tool.sh
index 6e0bbac..1d8e2b7 100755
--- a/scripts/in_container/run_cli_tool.sh
+++ b/scripts/in_container/run_cli_tool.sh
@@ -47,7 +47,7 @@ SCRIPT_NAME="$( basename "${BASH_SOURCE[0]}")"
# Drop "-update" suffix, if exists
TOOL_NAME="$(echo "${SCRIPT_NAME}" | cut -d "-" -f 1)"
-SUPPORTED_TOOL_NAMES=("aws" "az" "gcloud" "bq" "gsutil" "terraform" "java")
+SUPPORTED_TOOL_NAMES=("aws" "az")
if [ ! -L "${BASH_SOURCE[0]}" ]
then
@@ -93,10 +93,6 @@ COMMON_DOCKER_ARGS=(
AWS_CREDENTIALS_DOCKER_ARGS=(-v "${HOST_HOME}/.aws:/root/.aws")
AZURE_CREDENTIALS_DOCKER_ARGS=(-v "${HOST_HOME}/.azure:/root/.azure")
-GOOGLE_CREDENTIALS_DOCKER_ARGS=(
- -v "${HOST_HOME}/.config/gcloud:/root/.config/gcloud"
- -v "${HOST_AIRFLOW_SOURCES}/files/.kube:/root/.kube"
-)
COMMAND=("${@}")
@@ -110,25 +106,6 @@ case "${TOOL_NAME}" in
COMMON_DOCKER_ARGS+=("${AZURE_CREDENTIALS_DOCKER_ARGS[@]}")
IMAGE_NAME="mcr.microsoft.com/azure-cli:latest"
;;
- gcloud | bq | gsutil )
- COMMON_DOCKER_ARGS+=("${GOOGLE_CREDENTIALS_DOCKER_ARGS[@]}")
- IMAGE_NAME="gcr.io/google.com/cloudsdktool/cloud-sdk:latest"
- COMMAND=("$TOOL_NAME" "${@}")
- ;;
- terraform )
- COMMON_DOCKER_ARGS+=(
- "${GOOGLE_CREDENTIALS_DOCKER_ARGS[@]}"
- "${AZURE_CREDENTIALS_DOCKER_ARGS[@]}"
- "${AWS_CREDENTIALS_DOCKER_ARGS[@]}"
- )
- IMAGE_NAME="hashicorp/terraform:latest"
- ;;
- java )
- # TODO: Should we add other credentials?
- COMMON_DOCKER_ARGS+=("${GOOGLE_CREDENTIALS_DOCKER_ARGS[@]}")
- IMAGE_NAME="openjdk:8-jre-slim"
- COMMAND=("/usr/local/openjdk-8/bin/java" "${@}")
- ;;
* )
echo
echo "${COLOR_RED}ERROR: Unsupported tool name: ${TOOL_NAME} ${COLOR_RESET}"