You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2020/10/11 23:35:37 UTC
[airflow] 08/14: Constraints and PIP packages can be installed from
local sources (#11382)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit dd3a4048ee25af90e2666ca7a4501cb7485ba9b0
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Sat Oct 10 12:58:09 2020 +0200
Constraints and PIP packages can be installed from local sources (#11382)
* Constraints and PIP packages can be installed from local sources
This is the final part of implementing #11171 based on feedback
from enterprise customers we worked with. They want to have
a capability of building the image using binary wheel packages
that are locally available and the official Dockerfile. This means
that besides the official APT sources the Dockerfile build should
not needd GitHub, nor any other external files pulled from outside
including PIP repository.
This change also includes documentation on how to prepare set of
such binaries ready for inspection and review by security teams
in Enterprise environment. Such sets of "known-working-binary-whl"
files can then be separately committed, tracked and scrutinized
in an artifact repository of such an Enterprise.
Fixes: #11171
* Update docs/production-deployment.rst
(cherry picked from commit 04973904c3652fac4a8efc168d2b36f8a9245257)
---
.dockerignore | 4 ++
BREEZE.rst | 48 +++++++++++++----
Dockerfile | 28 ++++++----
IMAGES.rst | 25 +++++++--
breeze | 40 ++++++++++----
breeze-complete | 2 +-
docker-context-files/README.md | 31 +++++++++++
docs/production-deployment.rst | 66 ++++++++++++++++++++++++
scripts/ci/libraries/_build_images.sh | 23 ++++++++-
scripts/ci/libraries/_initialization.sh | 12 +++++
scripts/in_container/run_generate_constraints.sh | 2 +-
11 files changed, 246 insertions(+), 35 deletions(-)
diff --git a/.dockerignore b/.dockerignore
index fb9c80e..8a8bb52 100644
--- a/.dockerignore
+++ b/.dockerignore
@@ -56,6 +56,10 @@
!empty
!.pypirc
+# This folder is for you if you want to add any files to the docker context when you build your own
+# docker image. most of other files and any new folder you add will be excluded by default
+!docker-context-files
+
# Avoid triggering context change on README change (new companies using Airflow)
# So please do not uncomment this line ;)
# !README.md
diff --git a/BREEZE.rst b/BREEZE.rst
index 5b23c84..6bf6f7e 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -1212,9 +1212,6 @@ This is the current syntax for `./breeze <./breeze>`_:
--image-tag TAG
Additional tag in the image.
- --disable-pip-cache
- Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
-
--additional-extras ADDITIONAL_EXTRAS
Additional extras to pass to build images The default is no additional extras.
@@ -1260,6 +1257,19 @@ This is the current syntax for `./breeze <./breeze>`_:
Disables installation of the mysql client which might be problematic if you are building
image in controlled environment. Only valid for production image.
+ --constraints-location
+ Url to the constraints file. In case of the production image it can also be a path to the
+ constraint file placed in 'docker-context-files' folder, in which case it has to be
+ in the form of '/docker-context-files/<NAME_OF_THE_FILE>'
+
+ --disable-pip-cache
+ Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
+
+ --install-local-pip-wheels
+ This flag is only used in production image building. If it is used then instead of
+ installing Airflow from PyPI, the packages are installed from the .whl packages placed
+ in the 'docker-context-files' folder. It implies '--disable-pip-cache'
+
-C, --force-clean-images
Force build images with cache disabled. This will remove the pulled or build images
and start building images from scratch. This might take a long time.
@@ -1700,9 +1710,6 @@ This is the current syntax for `./breeze <./breeze>`_:
--image-tag TAG
Additional tag in the image.
- --disable-pip-cache
- Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
-
--additional-extras ADDITIONAL_EXTRAS
Additional extras to pass to build images The default is no additional extras.
@@ -1748,6 +1755,19 @@ This is the current syntax for `./breeze <./breeze>`_:
Disables installation of the mysql client which might be problematic if you are building
image in controlled environment. Only valid for production image.
+ --constraints-location
+ Url to the constraints file. In case of the production image it can also be a path to the
+ constraint file placed in 'docker-context-files' folder, in which case it has to be
+ in the form of '/docker-context-files/<NAME_OF_THE_FILE>'
+
+ --disable-pip-cache
+ Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
+
+ --install-local-pip-wheels
+ This flag is only used in production image building. If it is used then instead of
+ installing Airflow from PyPI, the packages are installed from the .whl packages placed
+ in the 'docker-context-files' folder. It implies '--disable-pip-cache'
+
-C, --force-clean-images
Force build images with cache disabled. This will remove the pulled or build images
and start building images from scratch. This might take a long time.
@@ -2051,9 +2071,6 @@ This is the current syntax for `./breeze <./breeze>`_:
--image-tag TAG
Additional tag in the image.
- --disable-pip-cache
- Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
-
--additional-extras ADDITIONAL_EXTRAS
Additional extras to pass to build images The default is no additional extras.
@@ -2099,6 +2116,19 @@ This is the current syntax for `./breeze <./breeze>`_:
Disables installation of the mysql client which might be problematic if you are building
image in controlled environment. Only valid for production image.
+ --constraints-location
+ Url to the constraints file. In case of the production image it can also be a path to the
+ constraint file placed in 'docker-context-files' folder, in which case it has to be
+ in the form of '/docker-context-files/<NAME_OF_THE_FILE>'
+
+ --disable-pip-cache
+ Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
+
+ --install-local-pip-wheels
+ This flag is only used in production image building. If it is used then instead of
+ installing Airflow from PyPI, the packages are installed from the .whl packages placed
+ in the 'docker-context-files' folder. It implies '--disable-pip-cache'
+
-C, --force-clean-images
Force build images with cache disabled. This will remove the pulled or build images
and start building images from scratch. This might take a long time.
diff --git a/Dockerfile b/Dockerfile
index cb46c69..f257606 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -140,6 +140,8 @@ ARG INSTALL_MYSQL_CLIENT="true"
ENV INSTALL_MYSQL_CLIENT=${INSTALL_MYSQL_CLIENT}
COPY scripts/docker scripts/docker
+COPY docker-context-files /docker-context-files
+
RUN ./scripts/docker/install_mysql.sh dev
ARG AIRFLOW_REPO=apache/airflow
@@ -153,8 +155,8 @@ ARG ADDITIONAL_AIRFLOW_EXTRAS=""
ENV AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}${ADDITIONAL_AIRFLOW_EXTRAS:+,}${ADDITIONAL_AIRFLOW_EXTRAS}
ARG AIRFLOW_CONSTRAINTS_REFERENCE="constraints-master"
-ARG AIRFLOW_CONSTRAINTS_URL="https://raw.githubusercontent.com/apache/airflow/${AIRFLOW_CONSTRAINTS_REFERENCE}/constraints-${PYTHON_MAJOR_MINOR_VERSION}.txt"
-ENV AIRFLOW_CONSTRAINTS_URL=${AIRFLOW_CONSTRAINTS_URL}
+ARG AIRFLOW_CONSTRAINTS_LOCATION="https://raw.githubusercontent.com/apache/airflow/${AIRFLOW_CONSTRAINTS_REFERENCE}/constraints-${PYTHON_MAJOR_MINOR_VERSION}.txt"
+ENV AIRFLOW_CONSTRAINTS_LOCATION=${AIRFLOW_CONSTRAINTS_LOCATION}
ENV PATH=${PATH}:/root/.local/bin
RUN mkdir -p /root/.local/bin
@@ -170,7 +172,7 @@ RUN if [[ ${AIRFLOW_PRE_CACHED_PIP_PACKAGES} == "true" ]]; then \
fi; \
pip install --user \
"https://github.com/${AIRFLOW_REPO}/archive/${AIRFLOW_BRANCH}.tar.gz#egg=apache-airflow[${AIRFLOW_EXTRAS}]" \
- --constraint "${AIRFLOW_CONSTRAINTS_URL}" && pip uninstall --yes apache-airflow; \
+ --constraint "${AIRFLOW_CONSTRAINTS_LOCATION}" && pip uninstall --yes apache-airflow; \
fi
ARG AIRFLOW_SOURCES_FROM="."
@@ -196,6 +198,9 @@ ENV AIRFLOW_INSTALL_SOURCES=${AIRFLOW_INSTALL_SOURCES}
ARG AIRFLOW_INSTALL_VERSION=""
ENV AIRFLOW_INSTALL_VERSION=${AIRFLOW_INSTALL_VERSION}
+ARG AIRFLOW_LOCAL_PIP_WHEELS=""
+ENV AIRFLOW_LOCAL_PIP_WHEELS=${AIRFLOW_LOCAL_PIP_WHEELS}
+
ARG SLUGIFY_USES_TEXT_UNIDECODE=""
ENV SLUGIFY_USES_TEXT_UNIDECODE=${SLUGIFY_USES_TEXT_UNIDECODE}
@@ -205,12 +210,17 @@ WORKDIR /opt/airflow
RUN if [[ ${INSTALL_MYSQL_CLIENT} != "true" ]]; then \
AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS/mysql,}; \
fi; \
- pip install --user "${AIRFLOW_INSTALL_SOURCES}[${AIRFLOW_EXTRAS}]${AIRFLOW_INSTALL_VERSION}" \
- --constraint "${AIRFLOW_CONSTRAINTS_URL}" && \
- if [ -n "${ADDITIONAL_PYTHON_DEPS}" ]; then pip install --user ${ADDITIONAL_PYTHON_DEPS} \
- --constraint "${AIRFLOW_CONSTRAINTS_URL}"; fi && \
- find /root/.local/ -name '*.pyc' -print0 | xargs -0 rm -r && \
- find /root/.local/ -type d -name '__pycache__' -print0 | xargs -0 rm -r
+ if [[ ${AIRFLOW_LOCAL_PIP_WHEELS} != "true" ]]; then \
+ pip install --user "${AIRFLOW_INSTALL_SOURCES}[${AIRFLOW_EXTRAS}]${AIRFLOW_INSTALL_VERSION}" \
+ --constraint "${AIRFLOW_CONSTRAINTS_LOCATION}"; \
+ if [ -n "${ADDITIONAL_PYTHON_DEPS}" ]; then \
+ pip install --user ${ADDITIONAL_PYTHON_DEPS} --constraint "${AIRFLOW_CONSTRAINTS_LOCATION}"; \
+ fi; \
+ else \
+ pip install --user /docker-context-files/*.whl; \
+ fi \
+ && find /root/.local/ -name '*.pyc' -print0 | xargs -0 rm -r \
+ && find /root/.local/ -type d -name '__pycache__' -print0 | xargs -0 rm -r
RUN AIRFLOW_SITE_PACKAGE="/root/.local/lib/python${PYTHON_MAJOR_MINOR_VERSION}/site-packages/airflow"; \
if [[ -f "${AIRFLOW_SITE_PACKAGE}/www_rbac/package.json" ]]; then \
diff --git a/IMAGES.rst b/IMAGES.rst
index 68467c6..ecfc388 100644
--- a/IMAGES.rst
+++ b/IMAGES.rst
@@ -22,8 +22,13 @@ Airflow docker images
Airflow has two images (build from Dockerfiles):
-* Production image (Dockerfile) - that can be used to build your own production-ready Airflow installation
-* CI image (Dockerfile.ci) - used for running tests and local development
+ * Production image (Dockerfile) - that can be used to build your own production-ready Airflow installation
+ You can read more about building and using the production image in the
+ `Production Deployments <docs/production-deployment.rst>`_ document. The image is built using
+ `Dockerfile <Dockerfile>`_
+
+ * CI image (Dockerfile.ci) - used for running tests and local development. The image is built using
+ `Dockerfile.ci <Dockerfile.ci>`_
Image naming conventions
========================
@@ -332,7 +337,6 @@ based on example in `this comment <https://github.com/apache/airflow/issues/8605
--build-arg ADDITIONAL_RUNTIME_ENV_VARS="ACCEPT_EULA=Y" \
--tag my-image
-
CI image build arguments
........................
@@ -378,6 +382,21 @@ The following build arguments (``--build-arg`` in docker build command) can be u
| | | dependencies from the repository from |
| | | scratch |
+------------------------------------------+------------------------------------------+------------------------------------------+
+| ``AIRFLOW_CONSTRAINTS_LOCATION`` | | If not empty, it will override the |
+| | | source of the constraints with the |
+| | | specified URL or file. Note that the |
+| | | file has to be in docker context so |
+| | | it's best to place such file in |
+| | | one of the folders included in |
+| | | dockerignore |
++------------------------------------------+------------------------------------------+------------------------------------------+
+| ``AIRFLOW_LOCAL_PIP_WHEELS`` | ``false`` | If set to true, Airflow and it's |
+| | | dependencies are installed from locally |
+| | | downloaded .whl files placed in the |
+| | | ``docker-context-files``. Implies |
+| | | ``AIRFLOW_PRE_CACHED_PIP_PACKAGES`` |
+| | | to be false. |
++------------------------------------------+------------------------------------------+------------------------------------------+
| ``AIRFLOW_EXTRAS`` | ``all`` | extras to install |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``AIRFLOW_PRE_CACHED_PIP_PACKAGES`` | ``true`` | Allows to pre-cache airflow PIP packages |
diff --git a/breeze b/breeze
index 1878e31..fc74cac 100755
--- a/breeze
+++ b/breeze
@@ -910,13 +910,6 @@ function breeze::parse_arguments() {
# if not set here, docker cached is determined later, depending on type of image to be build
shift
;;
- -B | --disable-pip-cache)
- echo "Disable PIP cache during build"
- echo
- export AIRFLOW_PRE_CACHED_PIP_PACKAGES="false"
- shift
- ;;
-
-P | --force-pull-images)
echo "Force pulling images before build. Uses pulled images as cache."
echo
@@ -1004,6 +997,23 @@ function breeze::parse_arguments() {
echo "Install MySQL client: ${INSTALL_MYSQL_CLIENT}"
shift
;;
+ --constraints-location)
+ export AIRFLOW_CONSTRAINTS_LOCATION="${2}"
+ echo "Constraints location: ${AIRFLOW_CONSTRAINTS_LOCATION}"
+ shift 2
+ ;;
+ --disable-pip-cache)
+ echo "Disable PIP cache during build"
+ echo
+ export AIRFLOW_PRE_CACHED_PIP_PACKAGES="false"
+ shift
+ ;;
+ --install-local-pip-wheels)
+ export AIRFLOW_LOCAL_PIP_WHEELS="true"
+ export AIRFLOW_PRE_CACHED_PIP_PACKAGES="false"
+ echo "Install from local wheels and disable pip cache"
+ shift
+ ;;
--image-tag)
export IMAGE_TAG="${2}"
echo "Tag to add to the image: ${IMAGE_TAG}"
@@ -2175,9 +2185,6 @@ ${FORMATTED_DEFAULT_PROD_EXTRAS}
--image-tag TAG
Additional tag in the image.
---disable-pip-cache
- Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
-
--additional-extras ADDITIONAL_EXTRAS
Additional extras to pass to build images The default is no additional extras.
@@ -2223,6 +2230,19 @@ Build options:
Disables installation of the mysql client which might be problematic if you are building
image in controlled environment. Only valid for production image.
+--constraints-location
+ Url to the constraints file. In case of the production image it can also be a path to the
+ constraint file placed in 'docker-context-files' folder, in which case it has to be
+ in the form of '/docker-context-files/<NAME_OF_THE_FILE>'
+
+--disable-pip-cache
+ Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
+
+--install-local-pip-wheels
+ This flag is only used in production image building. If it is used then instead of
+ installing Airflow from PyPI, the packages are installed from the .whl packages placed
+ in the 'docker-context-files' folder. It implies '--disable-pip-cache'
+
-C, --force-clean-images
Force build images with cache disabled. This will remove the pulled or build images
and start building images from scratch. This might take a long time.
diff --git a/breeze-complete b/breeze-complete
index 5c58e50..6a62eaf 100644
--- a/breeze-complete
+++ b/breeze-complete
@@ -136,7 +136,7 @@ build-cache-local build-cache-pulled build-cache-disabled disable-pip-cache
dockerhub-user: dockerhub-repo: github-registry github-repository: github-image-id:
postgres-version: mysql-version:
additional-extras: additional-python-deps: additional-dev-deps: additional-runtime-deps: image-tag:
-disable-mysql-client-installation
+disable-mysql-client-installation constraints-location: disable-pip-cache install-local-pip-wheels
additional-extras: additional-python-deps:
dev-apt-deps: additional-dev-apt-deps: dev-apt-command: additional-dev-apt-command: additional-dev-apt-env:
runtime-apt-deps: additional-runtime-apt-deps: runtime-apt-command: additional-runtime-apt-command: additional-runtime-apt-env:
diff --git a/docker-context-files/README.md b/docker-context-files/README.md
new file mode 100644
index 0000000..f53fbfe
--- /dev/null
+++ b/docker-context-files/README.md
@@ -0,0 +1,31 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+This folder is par of the Docker context.
+
+Most of other folders in Airflow are not part of the context in order to make the context smaller.
+
+The Production [Dockerfile](../Dockerfile) copies th [docker-context-files](.) folder to the "build"
+stage of the production image (it is not used in the CI image) and content of the folder is available
+in the `/docker-context-file` folder inside the build image. You can store constraint files and wheel
+packages there that you want to install as PYPI packages and refer to those packages using
+`--constraint-location` flag for constraints or by using `--install-local-pip-wheels` flag.
+
+By default, the content of this folder is .gitignored so that any binaries and files you put here are only
+used for local builds and not committed to the repository.
diff --git a/docs/production-deployment.rst b/docs/production-deployment.rst
index 95e08ed..5e6cad2 100644
--- a/docs/production-deployment.rst
+++ b/docs/production-deployment.rst
@@ -189,6 +189,71 @@ based on example in `this comment <https://github.com/apache/airflow/issues/8605
--build-arg ADDITIONAL_RUNTIME_ENV_VARS="ACCEPT_EULA=Y" \
--tag my-image
+Customizing images in high security restricted environments
+...........................................................
+
+You can also make sure your image is only build using local constraint file and locally downloaded
+wheel files. This is often useful in Enterprise environments where the binary files are verified and
+vetted by the security teams.
+
+This builds below builds the production image in version 3.7 with packages and constraints used from the local
+``docker-context-files`` rather than installed from PyPI or GitHub. It also disables MySQL client
+installation as it is using external installation method.
+
+Note that as a prerequisite - you need to have downloaded wheel files. In the example below we
+first download such constraint file locally and then use ``pip download`` to get the .whl files needed
+but in most likely scenario, those wheel files should be copied from an internal repository of such .whl
+files. Note that ``AIRFLOW_INSTALL_VERSION`` is only there for reference, the apache airflow .whl file
+in the right version is part of the .whl files downloaded.
+
+Note that 'pip download' will only works on Linux host as some of the packages need to be compiled from
+sources and you cannot install them providing ``--platform`` switch. They also need to be downloaded using
+the same python version as the target image.
+
+The ``pip download`` might happen in a separate environment. The files can be committed to a separate
+binary repository and vetted/verified by the security team and used subsequently to build images
+of Airflow when needed on an air-gaped system.
+
+Preparing the constraint files and wheel files:
+
+.. code-block:: bash
+
+ rm docker-context-files/*.whl docker-context-files/*.txt
+
+ curl -Lo "docker-context-files/constraints-1-10.txt" \
+ https://raw.githubusercontent.com/apache/airflow/constraints-1-10/constraints-3.7.txt
+
+ pip download --dest docker-context-files \
+ --constraint docker-context-files/constraints-1-10.txt \
+ apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,ssh,statsd,virtualenv]==1.10.12
+
+
+Building the image (after copying the files downloaded to the "docker-context-files" directory:
+
+.. code-block:: bash
+
+ ./breeze build-image \
+ --production-image --python 3.7 --install-airflow-version=1.10.12 \
+ --disable-mysql-client-installation --disable-pip-cache --install-local-pip-wheels \
+ --constraints-location="/docker-context-files/constraints-1-10.txt"
+
+or
+
+.. code-block:: bash
+
+ docker build . \
+ --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
+ --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \
+ --build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \
+ --build-arg AIRFLOW_INSTALL_VERSION="==1.10.12" \
+ --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1-10" \
+ --build-arg AIRFLOW_SOURCES_FROM="empty" \
+ --build-arg AIRFLOW_SOURCES_TO="/empty" \
+ --build-arg INSTALL_MYSQL_CLIENT="false" \
+ --build-arg AIRFLOW_PRE_CACHED_PIP_PACKAGES="false" \
+ --build-arg AIRFLOW_LOCAL_PIP_WHEELS="true" \
+ --build-arg AIRFLOW_CONSTRAINTS_LOCATION="/docker-context-files/constraints-1-10.txt"
+
Customizing & extending the image together
..........................................
@@ -524,6 +589,7 @@ additional apt dev and runtime dependencies.
--build-arg ADDITIONAL_RUNTIME_APT_DEPS="default-jre-headless"
+
More details about the images
-----------------------------
diff --git a/scripts/ci/libraries/_build_images.sh b/scripts/ci/libraries/_build_images.sh
index 9f391e0..5454116 100644
--- a/scripts/ci/libraries/_build_images.sh
+++ b/scripts/ci/libraries/_build_images.sh
@@ -26,6 +26,16 @@ function build_images::add_build_args_for_remote_install() {
"--build-arg" "AIRFLOW_SOURCES_FROM=empty"
"--build-arg" "AIRFLOW_SOURCES_TO=/empty"
)
+ if [[ ${AIRFLOW_CONSTRAINTS_REFERENCE} != "" ]]; then
+ EXTRA_DOCKER_PROD_BUILD_FLAGS+=(
+ "--build-arg" "AIRFLOW_CONSTRAINTS_REFERENCE=${AIRFLOW_CONSTRAINTS_REFERENCE}"
+ )
+ fi
+ if [[ "${AIRFLOW_CONSTRAINTS_LOCATION}" != "" ]]; then
+ EXTRA_DOCKER_PROD_BUILD_FLAGS+=(
+ "--build-arg" "AIRFLOW_CONSTRAINTS_LOCATION=${AIRFLOW_CONSTRAINTS_LOCATION}"
+ )
+ fi
if [[ ${AIRFLOW_VERSION} =~ [^0-9]*1[^0-9]*10[^0-9]([0-9]*) ]]; then
# All types of references/versions match this regexp for 1.10 series
# for example v1_10_test, 1.10.10, 1.10.9 etc. ${BASH_REMATCH[1]} matches last
@@ -530,6 +540,13 @@ function build_images::build_ci_image() {
EXTRA_DOCKER_CI_BUILD_FLAGS=(
"--build-arg" "AIRFLOW_CONSTRAINTS_REFERENCE=${DEFAULT_CONSTRAINTS_BRANCH}"
)
+
+ if [[ "${AIRFLOW_CONSTRAINTS_LOCATION}" != "" ]]; then
+ EXTRA_DOCKER_CI_BUILD_FLAGS+=(
+ "--build-arg" "AIRFLOW_CONSTRAINTS_LOCATION=${AIRFLOW_CONSTRAINTS_LOCATION}"
+ )
+ fi
+
if [[ -n ${SPIN_PID:=""} ]]; then
kill -HUP "${SPIN_PID}" || true
wait "${SPIN_PID}" || true
@@ -722,13 +739,14 @@ function build_images::build_prod_images() {
--build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
--build-arg AIRFLOW_BRANCH="${AIRFLOW_BRANCH_FOR_PYPI_PRELOADING}" \
--build-arg AIRFLOW_EXTRAS="${AIRFLOW_EXTRAS}" \
- --build-arg AIRFLOW_PRE_CACHED_PIP_PACKAGES="${AIRFLOW_PRE_CACHED_PIP_PACKAGES}" \
--build-arg ADDITIONAL_AIRFLOW_EXTRAS="${ADDITIONAL_AIRFLOW_EXTRAS}" \
--build-arg ADDITIONAL_PYTHON_DEPS="${ADDITIONAL_PYTHON_DEPS}" \
"${additional_dev_args[@]}" \
--build-arg ADDITIONAL_DEV_APT_COMMAND="${ADDITIONAL_DEV_APT_COMMAND}" \
--build-arg ADDITIONAL_DEV_APT_DEPS="${ADDITIONAL_DEV_APT_DEPS}" \
--build-arg ADDITIONAL_DEV_APT_ENV="${ADDITIONAL_DEV_APT_ENV}" \
+ --build-arg AIRFLOW_PRE_CACHED_PIP_PACKAGES="${AIRFLOW_PRE_CACHED_PIP_PACKAGES}" \
+ --build-arg AIRFLOW_LOCAL_PIP_WHEELS="${AIRFLOW_LOCAL_PIP_WHEELS}" \
--build-arg BUILD_ID="${CI_BUILD_ID}" \
--build-arg COMMIT_SHA="${COMMIT_SHA}" \
"${DOCKER_CACHE_PROD_BUILD_DIRECTIVE[@]}" \
@@ -755,10 +773,11 @@ function build_images::build_prod_images() {
--build-arg ADDITIONAL_RUNTIME_APT_COMMAND="${ADDITIONAL_RUNTIME_APT_COMMAND}" \
--build-arg ADDITIONAL_RUNTIME_APT_DEPS="${ADDITIONAL_RUNTIME_APT_DEPS}" \
--build-arg ADDITIONAL_RUNTIME_APT_ENV="${ADDITIONAL_RUNTIME_APT_ENV}" \
+ --build-arg AIRFLOW_PRE_CACHED_PIP_PACKAGES="${AIRFLOW_PRE_CACHED_PIP_PACKAGES}" \
+ --build-arg AIRFLOW_LOCAL_PIP_WHEELS="${AIRFLOW_LOCAL_PIP_WHEELS}" \
--build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
--build-arg AIRFLOW_BRANCH="${AIRFLOW_BRANCH_FOR_PYPI_PRELOADING}" \
--build-arg AIRFLOW_EXTRAS="${AIRFLOW_EXTRAS}" \
- --build-arg AIRFLOW_PRE_CACHED_PIP_PACKAGES="${AIRFLOW_PRE_CACHED_PIP_PACKAGES}" \
--build-arg BUILD_ID="${CI_BUILD_ID}" \
--build-arg COMMIT_SHA="${COMMIT_SHA}" \
"${additional_dev_args[@]}" \
diff --git a/scripts/ci/libraries/_initialization.sh b/scripts/ci/libraries/_initialization.sh
index 2139aa6..1b80364 100644
--- a/scripts/ci/libraries/_initialization.sh
+++ b/scripts/ci/libraries/_initialization.sh
@@ -356,6 +356,14 @@ function initialization::initialize_image_build_variables() {
export INSTALL_MYSQL_CLIENT=${INSTALL_MYSQL_CLIENT:="true"}
# additional tag for the image
export IMAGE_TAG=${IMAGE_TAG:=""}
+
+ # whether installation should be performed from the local wheel packages in "docker-context-files" folder
+ export AIRFLOW_LOCAL_PIP_WHEELS="${AIRFLOW_LOCAL_PIP_WHEELS:="false"}"
+ # reference to CONSTRAINTS. they can be overwritten manually or replaced with AIRFLOW_CONSTRAINTS_LOCATION
+ export AIRFLOW_CONSTRAINTS_REFERENCE="${AIRFLOW_CONSTRAINTS_REFERENCE:=""}"
+ # direct constraints Location - can be URL or path to local file. If empty, it will be calculated
+ # based on which Airflow version is installed and from where
+ export AIRFLOW_CONSTRAINTS_LOCATION="${AIRFLOW_CONSTRAINTS_LOCATION:=""}"
}
# Determine version suffixes used to build backport packages
@@ -673,6 +681,10 @@ function initialization::make_constants_read_only() {
readonly IMAGE_TAG
readonly AIRFLOW_PRE_CACHED_PIP_PACKAGES
+ readonly AIRFLOW_LOCAL_PIP_WHEELS
+ readonly AIRFLOW_CONSTRAINTS_REFERENCE
+ readonly AIRFLOW_CONSTRAINTS_LOCATION
+
# AIRFLOW_EXTRAS are made readonly by the time the image is built (either PROD or CI)
readonly ADDITIONAL_AIRFLOW_EXTRAS
readonly ADDITIONAL_PYTHON_DEPS
diff --git a/scripts/in_container/run_generate_constraints.sh b/scripts/in_container/run_generate_constraints.sh
index d361b0e..dabf698 100755
--- a/scripts/in_container/run_generate_constraints.sh
+++ b/scripts/in_container/run_generate_constraints.sh
@@ -27,7 +27,7 @@ CURRENT_CONSTRAINT_FILE="${CONSTRAINTS_DIR}/constraints-${PYTHON_MAJOR_MINOR_VER
mkdir -pv "${CONSTRAINTS_DIR}"
-curl "${AIRFLOW_CONSTRAINTS_URL}" --output "${LATEST_CONSTRAINT_FILE}"
+curl "${AIRFLOW_CONSTRAINTS_LOCATION}" --output "${LATEST_CONSTRAINT_FILE}"
echo
echo "Freezing constraints to ${CURRENT_CONSTRAINT_FILE}"