You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2020/06/20 16:53:06 UTC
[airflow] 07/25: Support additional apt dependencies (#9189)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 55d15ae6ee71b839697d752bd8626c01c8b3ddc0
Author: zikun <33...@users.noreply.github.com>
AuthorDate: Wed Jun 10 05:05:43 2020 +0800
Support additional apt dependencies (#9189)
* Add ADDITONAL_DEV_DEPS and ADDITONAL_RUNTIME_DEPS
* Add examples for additional apt dev and runtime dependencies
* Update comment
* Fix typo
(cherry picked from commit 82c8343ab6294168104cb2f25018656b681d2de9)
---
Dockerfile | 12 ++++++++++--
Dockerfile.ci | 10 +++++++++-
IMAGES.rst | 44 ++++++++++++++++++++++++++++++++++++++++++++
docs/concepts.rst | 2 +-
4 files changed, 64 insertions(+), 4 deletions(-)
diff --git a/Dockerfile b/Dockerfile
index ac27823..7c722cf 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -79,7 +79,10 @@ RUN apt-get update \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
-# Install basic apt dependencies
+ARG ADDITIONAL_DEV_DEPS=""
+ENV ADDITIONAL_DEV_DEPS=${ADDITIONAL_DEV_DEPS}
+
+# Install basic and additional apt dependencies
RUN curl --fail --location https://deb.nodesource.com/setup_10.x | bash - \
&& curl https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - > /dev/null \
&& echo "deb https://dl.yarnpkg.com/debian/ stable main" > /etc/apt/sources.list.d/yarn.list \
@@ -121,6 +124,7 @@ RUN curl --fail --location https://deb.nodesource.com/setup_10.x | bash - \
unixodbc \
unixodbc-dev \
yarn \
+ ${ADDITIONAL_DEV_DEPS} \
&& apt-get autoremove -yqq --purge \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
@@ -242,13 +246,16 @@ ENV PYTHON_BASE_IMAGE=${PYTHON_BASE_IMAGE}
ARG AIRFLOW_VERSION
ENV AIRFLOW_VERSION=${AIRFLOW_VERSION}
+ARG ADDITIONAL_RUNTIME_DEPS=""
+ENV ADDITIONAL_RUNTIME_DEPS=${ADDITIONAL_RUNTIME_DEPS}
+
# Make sure noninteractive debian install is used and language variables set
ENV DEBIAN_FRONTEND=noninteractive LANGUAGE=C.UTF-8 LANG=C.UTF-8 LC_ALL=C.UTF-8 \
LC_CTYPE=C.UTF-8 LC_MESSAGES=C.UTF-8
# Note missing man directories on debian-buster
# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=863199
-# Install basic apt dependencies
+# Install basic and additional apt dependencies
RUN mkdir -pv /usr/share/man/man1 \
&& mkdir -pv /usr/share/man/man7 \
&& apt-get update \
@@ -277,6 +284,7 @@ RUN mkdir -pv /usr/share/man/man1 \
sqlite3 \
sudo \
unixodbc \
+ ${ADDITIONAL_RUNTIME_DEPS} \
&& apt-get autoremove -yqq --purge \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
diff --git a/Dockerfile.ci b/Dockerfile.ci
index 8051431..1549214 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -54,7 +54,10 @@ RUN apt-get update \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
-# Install basic apt dependencies
+ARG ADDITIONAL_DEV_DEPS=""
+ENV ADDITIONAL_DEV_DEPS=${ADDITIONAL_DEV_DEPS}
+
+# Install basic and additional apt dependencies
RUN curl --fail --location https://deb.nodesource.com/setup_10.x | bash - \
&& curl https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - > /dev/null \
&& echo "deb https://dl.yarnpkg.com/debian/ stable main" > /etc/apt/sources.list.d/yarn.list \
@@ -83,6 +86,7 @@ RUN curl --fail --location https://deb.nodesource.com/setup_10.x | bash - \
sasl2-bin \
sudo \
yarn \
+ ${ADDITIONAL_DEV_DEPS} \
&& apt-get autoremove -yqq --purge \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
@@ -115,6 +119,9 @@ RUN adduser airflow \
&& echo "airflow ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/airflow \
&& chmod 0440 /etc/sudoers.d/airflow
+ARG ADDITIONAL_RUNTIME_DEPS=""
+ENV ADDITIONAL_RUNTIME_DEPS=${ADDITIONAL_RUNTIME_DEPS}
+
# Note missing man directories on debian-buster
# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=863199
RUN mkdir -pv /usr/share/man/man1 \
@@ -142,6 +149,7 @@ RUN mkdir -pv /usr/share/man/man1 \
tmux \
unzip \
vim \
+ ${ADDITIONAL_RUNTIME_DEPS} \
&& apt-get autoremove -yqq --purge \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
diff --git a/IMAGES.rst b/IMAGES.rst
index 0a34140..0d4bd8c 100644
--- a/IMAGES.rst
+++ b/IMAGES.rst
@@ -203,6 +203,12 @@ The following build arguments (``--build-arg`` in docker build command) can be u
| ``ADDITIONAL_PYTHON_DEPS`` | \```\` | additional python dependencies to |
| | | install |
+------------------------------------------+------------------------------------------+------------------------------------------+
+| ``ADDITIONAL_DEV_DEPS`` | ```` | additional apt dev dependencies to |
+| | | install |
++------------------------------------------+------------------------------------------+------------------------------------------+
+| ``ADDITIONAL_RUNTIME_DEPS`` | ```` | additional apt runtime dependencies to |
+| | | install |
++------------------------------------------+------------------------------------------+------------------------------------------+
Here are some examples of how CI images can built manually. CI is always built from local sources.
@@ -236,6 +242,20 @@ This builds the CI image in version 3.6 with "mssql" additional package added.
docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
--build-arg PYTHON_MAJOR_MINOR_VERSION=3.6 --build-arg ADDITIONAL_PYTHON_DEPS="mssql"
+This builds the CI image in version 3.6 with "gcc" and "g++" additional apt dev dependencies added.
+
+.. code-block::
+
+ docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
+ --build-arg PYTHON_MAJOR_MINOR_VERSION=3.6 --build-arg ADDITIONAL_DEV_DEPS="gcc g++"
+
+This builds the CI image in version 3.6 with "jdbc" extra and "default-jre-headless" additional apt runtime dependencies added.
+
+.. code-block::
+
+ docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
+ --build-arg PYTHON_MAJOR_MINOR_VERSION=3.6 --build-arg AIRFLOW_EXTRAS=jdbc --build-arg ADDITIONAL_RUNTIME_DEPS="default-jre-headless"
+
Production images
@@ -277,6 +297,12 @@ The following build arguments (``--build-arg`` in docker build command) can be u
| ``ADDITIONAL_PYTHON_DEPS`` | ```` | Optional python packages to extend |
| | | the image with some extra dependencies |
+------------------------------------------+------------------------------------------+------------------------------------------+
+| ``ADDITIONAL_DEV_DEPS`` | ```` | additional apt dev dependencies to |
+| | | install |
++------------------------------------------+------------------------------------------+------------------------------------------+
+| ``ADDITIONAL_RUNTIME_DEPS`` | ```` | additional apt runtime dependencies to |
+| | | install |
++------------------------------------------+------------------------------------------+------------------------------------------+
| ``AIRFLOW_HOME`` | ``/opt/airflow`` | Airflow’s HOME (that’s where logs and |
| | | sqlite databases are stored) |
+------------------------------------------+------------------------------------------+------------------------------------------+
@@ -402,6 +428,24 @@ additional python dependencies.
--build-arg ADDITIONAL_AIRFLOW_EXTRAS="mssql,hdfs"
--build-arg ADDITIONAL_PYTHON_DEPS="sshtunnel oauth2client"
+This builds the production image in version 3.7 with additional airflow extras from 1.10.10 Pypi package and
+additional apt dev and runtime dependencies.
+
+.. code-block::
+
+ docker build . \
+ --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
+ --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \
+ --build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \
+ --build-arg AIRFLOW_INSTALL_VERSION="==1.10.10" \
+ --build-arg CONSTRAINT_REQUIREMENTS="https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt" \
+ --build-arg ENTRYPOINT_FILE="https://raw.githubusercontent.com/apache/airflow/1.10.10/entrypoint.sh" \
+ --build-arg AIRFLOW_SOURCES_FROM="entrypoint.sh" \
+ --build-arg AIRFLOW_SOURCES_TO="/entrypoint" \
+ --build-arg ADDITIONAL_AIRFLOW_EXTRAS="jdbc"
+ --build-arg ADDITIONAL_DEV_DEPS="gcc g++"
+ --build-arg ADDITIONAL_RUNTIME_DEPS="default-jre-headless"
+
Image manifests
---------------
diff --git a/docs/concepts.rst b/docs/concepts.rst
index 3a9c4c2..603a729 100644
--- a/docs/concepts.rst
+++ b/docs/concepts.rst
@@ -522,7 +522,7 @@ with the same ``conn_id``, the :py:meth:`~airflow.hooks.base_hook.BaseHook.get_c
provide basic load balancing and fault tolerance, when used in conjunction with retries.
Airflow also provides a mechanism to store connections outside the database, e.g. in :ref:`environment variables <environment_variables_secrets_backend>`.
-Additonal sources may be enabled, e.g. :ref:`AWS SSM Parameter Store <ssm_parameter_store_secrets>`, or you may
+Additional sources may be enabled, e.g. :ref:`AWS SSM Parameter Store <ssm_parameter_store_secrets>`, or you may
:ref:`roll your own secrets backend <roll_your_own_secrets_backend>`.
Many hooks have a default ``conn_id``, where operators using that hook do not