You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2020/06/09 09:04:33 UTC
[airflow] 20/36: You can push with Breeze as separate command and
to cache (#8976)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit eed7d335982404d3cda12f587799f2da4b176bc2
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Tue Jun 2 10:24:47 2020 +0200
You can push with Breeze as separate command and to cache (#8976)
Breeze had --push-images switch to also push images to repo
but it was often needed to build and push images separately.
We have now a possibility to push an already built image with
separate push-image command instead and also you can choose
to push to cache registry in GitHub rather than to DockerHub
with --registry-cache switch.
(cherry picked from commit a6216a760ca7449cfbf5aeee0f74caeda2f3f999)
---
.github/workflows/ci.yml | 2 +-
BREEZE.rst | 148 ++++++++++++++++++-------
breeze | 106 +++++++++++++-----
breeze-complete | 13 ++-
scripts/ci/_utils.sh | 35 +++---
scripts/ci/ci_fix_ownership.sh | 9 ++
scripts/ci/in_container/_in_container_utils.sh | 2 +-
7 files changed, 234 insertions(+), 81 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index e4f16e7..afa911c 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -32,7 +32,7 @@ env:
VERBOSE: "true"
UPGRADE_TO_LATEST_REQUIREMENTS: "false"
PYTHON_MAJOR_MINOR_VERSION: 3.5
- CACHE_REGISTRY: "docker.pkg.github.com"
+ ENABLE_REGISTRY_CACHE: "true"
CACHE_IMAGE_PREFIX: ${{ github.repository }}
CACHE_REGISTRY_USERNAME: ${{ github.actor }}
CACHE_REGISTRY_PASSWORD: ${{ secrets.GITHUB_TOKEN }}
diff --git a/BREEZE.rst b/BREEZE.rst
index 262ddcd..0d90763 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -78,36 +78,46 @@ Docker Compose
- **Permissions**: Configure to run the ``docker-compose`` command.
-Docker in WSL
--------------
+Docker in WSL 2
+---------------
+
+- **WSL 2 installation** :
+ Install WSL 2 and a Linux Distro (e.g. Ubuntu) see
+ `WSL 2 Installation Guide <https://docs.microsoft.com/en-us/windows/wsl/install-win10>`_ for details.
-- **WSL installation** :
- `WSL Installation Guide <https://docs.microsoft.com/en-us/windows/wsl/install-win10>`_ for details.
+- **Docker Desktop installation** :
+ Install Docker Desktop for Windows. For Windows Home follow the
+ `Docker Windows Home Installation Guide <https://docs.docker.com/docker-for-windows/install-windows-home>`_.
+ For Windows Pro, Enterprise, or Education follow the
+ `Docker Windows Installation Guide <https://docs.docker.com/docker-for-windows/install/>`_.
-- **Docker installation** :
- You should install docker in WSL.
- follow `Docker Installtion Guide <https://docs.docker.com/install/linux/docker-ce/ubuntu/>`_
- only docker-ce without docker-ce-cli containerd.io.
- **Docker setting** :
- You should expose Docker daemon,
+ WSL integration needs to be enabled
-.. image:: images/docker_expose_daemon.png
+.. image:: images/docker_wsl_integration.png
:align: left
- :alt: Docker expose daemon
-
-and set env variable DOCKER_HOST.
-
-.. code-block:: bash
-
- echo "export DOCKER_HOST=tcp://localhost:2375" >> ~/.bashrc && source ~/.bashrc
-
-- **WSL problems** :
- There is a mounting problem in docker because docker could not recognize ``/mnt/c``, ``/mnt/d`` driver path.
- run this command in Windows Version 18.03+ and reboot Windows
-
-.. code-block:: bash
-
- printf '[automount]\nroot = /\n options = "metadata"\n' >> /etc/wsl.conf
+ :alt: Docker WSL2 integration
+
+- **WSL 2 Filesystem Performance** :
+ Accessing the host Windows filesystem incurs a performance penalty,
+ it is therefore recommended to do development on the Linux filesystem.
+ E.g. Run ``cd ~`` and create a development folder in your Linux distro home
+ and git pull the Airflow repo there.
+
+- **WSL 2 Memory Usage** :
+ WSL 2 can consume a lot of memory under the process name "Vmmem". To reclaim
+ the memory after development you can:
+ * On the Linux distro clear cached memory: ``sudo sysctl -w vm.drop_caches=3``
+ * If no longer using Docker you can quit Docker Desktop
+ (right click system try icon and select "Quit Docker Desktop")
+ * If no longer using WSL you can shut it down on the Windows Host
+ with the following command: ``wsl --shutdown``
+
+- **Developing in WSL 2** :
+ You can use all the standard Linux command line utilities to develop on WSL 2.
+ Further VS Code supports developing in Windows but remotely executing in WSL.
+ If VS Code is installed on the Windows host system then in the WSL Linux Distro
+ you can run ``code .`` in the root directory of you Airflow repo to launch VS Code.
Docker Images Used by Breeze
----------------------------
@@ -182,6 +192,8 @@ On macOS, 2GB of RAM are available for your Docker containers by default, but mo
(4GB should be comfortable). For details see
`Docker for Mac - Advanced tab <https://docs.docker.com/v17.12/docker-for-mac/#advanced-tab>`_.
+On Windows WSL 2 expect the Linux Disto and Docker containers to use 7 - 8 GB of RAM.
+
Airflow Directory Structure inside Docker
-----------------------------------------
@@ -655,6 +667,7 @@ This is the current syntax for `./breeze <./breeze>`_:
cleanup-image Cleans up the container image created
exec Execs into running breeze container in new terminal
generate-requirements Generates pinned requirements for pip dependencies
+ push-image Pushes images to registry
initialize-local-virtualenv Initializes local virtualenv
setup-autocomplete Sets up autocomplete for breeze
stop Stops the docker-compose environment
@@ -729,7 +742,7 @@ This is the current syntax for `./breeze <./breeze>`_:
breeze [FLAGS] build-image -- <EXTRA_ARGS>
Builds docker image (CI or production) without entering the container. You can pass
- aditional options to this command, such as '--force-build-image',
+ additional options to this command, such as '--force-build-image',
'--force-pull-image' '--python' '--use-local-cache'' in order to modify build behaviour.
You can also pass '--production-image' flag to build production image rather than CI image.
@@ -788,17 +801,23 @@ This is the current syntax for `./breeze <./breeze>`_:
Uses local cache to build images. No pulled images will be used, but results of local
builds in the Docker cache are used instead.
- -u, --push-images
- After building - uploads the images to DockerHub
- It is useful in case you use your own DockerHub user to store images and you want
- to build them locally. Note that you need to use 'docker login' before you upload images.
-
-D, --dockerhub-user
DockerHub user used to pull, push and build images. Default: apache.
-H, --dockerhub-repo
DockerHub repository used to pull, push, build images. Default: airflow.
+ -c, --registry-cache
+ If registry cache is enabled, pulls and pushes are done from the registry cache in github.
+ You need to be logged in to the registry in order to be able to pull/push from it and you
+ need to be committer to push to airflow registry.
+
+ -G, --github-organisation
+ GitHub organisation used to pull, push images when cache is used. Default: apache.
+
+ -g, --github-repo
+ GitHub repository used to pull, push images when cache is used. Default: airflow.
+
-v, --verbose
Show verbose information about executed commands (enabled by default for running test).
Note that you can further increase verbosity and see all the commands executed by breeze
@@ -876,6 +895,55 @@ This is the current syntax for `./breeze <./breeze>`_:
####################################################################################################
+ Detailed usage for command: push-image
+
+ breeze [FLAGS] push-image -- <EXTRA_ARGS>
+
+ Pushes images to docker registry. You can push the images to DockerHub registry (default)
+ or to the GitHub cache registry (if --registry-cache flag is used).
+
+ For DockerHub pushes --dockerhub-user and --dockerhub-repo flags can be used to specify
+ the repository to push to. For GitHub repository --github-organisation and --github-repo
+ flags can be used for the same purpose.
+
+ You can also add --production-image flag to switch to production image (default is CI one)
+
+ Examples:
+
+ 'breeze push-image' or
+ 'breeze push-image --dockerhub-user user' to push to your private registry or
+ 'breeze push-image --production-image' - to push production image or
+ 'breeze push-image --registry-cache' - to push to GitHub cache or
+ 'breeze push-image --registry-cache --github-organisation org' - for other organisation
+
+ Flags:
+
+ -D, --dockerhub-user
+ DockerHub user used to pull, push and build images. Default: apache.
+
+ -H, --dockerhub-repo
+ DockerHub repository used to pull, push, build images. Default: airflow.
+
+ -c, --registry-cache
+ If registry cache is enabled, pulls and pushes are done from the registry cache in github.
+ You need to be logged in to the registry in order to be able to pull/push from it and you
+ need to be committer to push to airflow registry.
+
+ -G, --github-organisation
+ GitHub organisation used to pull, push images when cache is used. Default: apache.
+
+ -g, --github-repo
+ GitHub repository used to pull, push images when cache is used. Default: airflow.
+
+ -v, --verbose
+ Show verbose information about executed commands (enabled by default for running test).
+ Note that you can further increase verbosity and see all the commands executed by breeze
+ by running 'export VERBOSE_COMMANDS="true"' before running breeze.
+
+
+ ####################################################################################################
+
+
Detailed usage for command: initialize-local-virtualenv
breeze [FLAGS] initialize-local-virtualenv -- <EXTRA_ARGS>
@@ -1333,12 +1401,7 @@ This is the current syntax for `./breeze <./breeze>`_:
builds in the Docker cache are used instead.
****************************************************************************************************
- Flags for pushing Docker images (both CI and production)
-
- -u, --push-images
- After building - uploads the images to DockerHub
- It is useful in case you use your own DockerHub user to store images and you want
- to build them locally. Note that you need to use 'docker login' before you upload images.
+ Flags for pulling/pushing Docker images (both CI and production)
-D, --dockerhub-user
DockerHub user used to pull, push and build images. Default: apache.
@@ -1346,6 +1409,17 @@ This is the current syntax for `./breeze <./breeze>`_:
-H, --dockerhub-repo
DockerHub repository used to pull, push, build images. Default: airflow.
+ -c, --registry-cache
+ If registry cache is enabled, pulls and pushes are done from the registry cache in github.
+ You need to be logged in to the registry in order to be able to pull/push from it and you
+ need to be committer to push to airflow registry.
+
+ -G, --github-organisation
+ GitHub organisation used to pull, push images when cache is used. Default: apache.
+
+ -g, --github-repo
+ GitHub repository used to pull, push images when cache is used. Default: airflow.
+
****************************************************************************************************
Increase verbosity of the scripts
diff --git a/breeze b/breeze
index 75c00eb..fcea655 100755
--- a/breeze
+++ b/breeze
@@ -67,7 +67,7 @@ function setup_default_breeze_variables() {
# We have different versions of images depending on the python version used. We keep up with the
# Latest patch-level changes in Python (this is done automatically during CI builds) so we have
- # To only take into account MAJOR and MINOR version of python. This variable keeps the major/mninor
+ # To only take into account MAJOR and MINOR version of python. This variable keeps the major/minor
# version of python in X.Y format (3.6, 3.7 etc).
export PYTHON_MAJOR_MINOR_VERSION="${PYTHON_MAJOR_MINOR_VERSION:=$(read_from_file PYTHON_MAJOR_MINOR_VERSION)}"
@@ -158,7 +158,7 @@ function setup_default_breeze_variables() {
# Initializes development-friendly virtualenv if you are already in such env. It installs all the necessary
# packages from PyPI and it case of problems it provides useful hints on what prerequisites should be
-# installed. It also removes and resets the existing AIRFOW_HOME installation to make sure that you
+# installed. It also removes and resets the existing AIRFLOW_HOME installation to make sure that you
# have it synchronized with the version of airflow installed. It resets the airflow's sqlite database to
# a clean state. You can use this function if your virtualenv is broken, to clean it up
function initialize_virtualenv() {
@@ -792,13 +792,24 @@ function parse_arguments() {
echo
export FORWARD_CREDENTIALS="true"
shift 1 ;;
- -u|--push-images)
+ -c|--registry-cache)
echo
- echo "Pushing images to DockerHub"
+ echo "Use cache for the container registry"
echo
- export PUSH_IMAGES="true"
- export FORCE_BUILD_IMAGES="true"
+ export ENABLE_REGISTRY_CACHE="true"
shift ;;
+ -G|--github-organisation)
+ echo
+ echo "GitHub organisation"
+ echo
+ export GITHUB_ORGANISATION="${2}"
+ shift 2;;
+ -g|--github-repo)
+ echo
+ echo "GitHub repository"
+ echo
+ export GITHUB_REPOSITORY="${2}"
+ shift 2;;
--)
shift ;
break ;;
@@ -860,6 +871,15 @@ function parse_arguments() {
generate-requirements)
LAST_SUBCOMMAND="${1}"
COMMAND_TO_RUN="perform_generate_requirements"
+ # if you want to generate requirement - you want to build the image too :)
+ export FORCE_ANSWER_TO_QUESTIONS="yes"
+ # and assume you want to build it no matter if it is needed
+ export FORCE_BUILD_IMAGES="true"
+ shift ;;
+ push-image)
+ LAST_SUBCOMMAND="${1}"
+ COMMAND_TO_RUN="perform_push_image"
+ SKIP_CHECK_REMOTE_IMAGE="true"
shift ;;
initialize-local-virtualenv)
LAST_SUBCOMMAND="${1}"
@@ -1050,6 +1070,7 @@ function prepare_usage() {
export USAGE_FLAGS="Shows all breeze's flags"
export USAGE_GENERATE_REQUIREMENTS="Generates pinned requirements for pip dependencies"
export USAGE_INITIALIZE_LOCAL_VIRTUALENV="Initializes local virtualenv"
+ export USAGE_PUSH_IMAGE="Pushes images to registry"
export USAGE_SETUP_AUTOCOMPLETE="Sets up autocomplete for breeze"
export USAGE_STOP="Stops the docker-compose environment"
export USAGE_RESTART="Stops the docker-compose environment including DB cleanup"
@@ -1100,7 +1121,7 @@ $(flag_footer)
# shellcheck disable=SC2089
DETAILED_USAGE_BUILD_IMAGE="
Builds docker image (CI or production) without entering the container. You can pass
- aditional options to this command, such as '--force-build-image',
+ additional options to this command, such as '--force-build-image',
'--force-pull-image' '--python' '--use-local-cache'' in order to modify build behaviour.
You can also pass '--production-image' flag to build production image rather than CI image.
@@ -1109,7 +1130,7 @@ $(flag_airflow_variants)
$(flag_choose_different_airflow_version)
$(flag_production_image)
$(flag_build_docker_images)
-$(flag_push_docker_images)
+$(flag_pull_push_docker_images)
$(flag_verbosity)
"
# shellcheck disable=SC2090
@@ -1187,6 +1208,31 @@ $(flag_airflow_variants)
"
# shellcheck disable=SC2090
export DETAILED_USAGE_INITIALIZE_LOCAL_VIRTUALENV
+ # shellcheck disable=SC2089
+ DETAILED_USAGE_PUSH_IMAGE="
+ Pushes images to docker registry. You can push the images to DockerHub registry (default)
+ or to the GitHub cache registry (if --registry-cache flag is used).
+
+ For DockerHub pushes --dockerhub-user and --dockerhub-repo flags can be used to specify
+ the repository to push to. For GitHub repository --github-organisation and --github-repo
+ flags can be used for the same purpose.
+
+ You can also add --production-image flag to switch to production image (default is CI one)
+
+ Examples:
+
+ '${CMDNAME} push-image' or
+ '${CMDNAME} push-image --dockerhub-user user' to push to your private registry or
+ '${CMDNAME} push-image --production-image' - to push production image or
+ '${CMDNAME} push-image --registry-cache' - to push to GitHub cache or
+ '${CMDNAME} push-image --registry-cache --github-organisation org' - for other organisation
+
+Flags:
+$(flag_pull_push_docker_images)
+$(flag_verbosity)
+"
+ # shellcheck disable=SC2090
+ export DETAILED_USAGE_PUSH_IMAGE
export DETAILED_USAGE_SETUP_AUTOCOMPLETE="
Sets up autocomplete for breeze commands. Once you do it you need to re-enter the bash
shell and when typing breeze command <TAB> will provide autocomplete for
@@ -1548,19 +1594,26 @@ ${FORMATTED_DEFAULT_PROD_EXTRAS}
"
}
-# Prints flags controlling docker push process
-function flag_push_docker_images() {
+# Prints flags controlling docker pull and push process
+function flag_pull_push_docker_images() {
echo "
--u, --push-images
- After building - uploads the images to DockerHub
- It is useful in case you use your own DockerHub user to store images and you want
- to build them locally. Note that you need to use 'docker login' before you upload images.
-
-D, --dockerhub-user
DockerHub user used to pull, push and build images. Default: ${_BREEZE_DEFAULT_DOCKERHUB_USER:=}.
-H, --dockerhub-repo
DockerHub repository used to pull, push, build images. Default: ${_BREEZE_DEFAULT_DOCKERHUB_REPO:=}.
+
+-c, --registry-cache
+ If registry cache is enabled, pulls and pushes are done from the registry cache in github.
+ You need to be logged in to the registry in order to be able to pull/push from it and you
+ need to be committer to push to airflow registry.
+
+-G, --github-organisation
+ GitHub organisation used to pull, push images when cache is used. Default: ${_BREEZE_DEFAULT_GITHUB_ORGANISATION:=}.
+
+-g, --github-repo
+ GitHub repository used to pull, push images when cache is used. Default: ${_BREEZE_DEFAULT_GITHUB_REPO:=}.
+
"
}
@@ -1613,8 +1666,8 @@ $(print_star_line)
$(flag_build_docker_images)
$(print_star_line)
- Flags for pushing Docker images (both CI and production)
-$(flag_push_docker_images)
+ Flags for pulling/pushing Docker images (both CI and production)
+$(flag_pull_push_docker_images)
$(print_star_line)
Increase verbosity of the scripts
@@ -1822,7 +1875,7 @@ function run_static_checks {
fi
}
-# Runs Build before a comman - it will check and if needed rebuild necessary image, depending on the
+# Runs Build before a command - it will check and if needed rebuild necessary image, depending on the
# command chosen
function run_build_command {
case "${COMMAND_TO_RUN}" in
@@ -1838,7 +1891,7 @@ function run_build_command {
rebuild_ci_image_if_needed
fi
;;
- build_docs|perform_static_checks|perform_generate_requirements)
+ build_docs|perform_static_checks|perform_generate_requirements|perform_push_image)
prepare_ci_build
rebuild_ci_image_if_needed
;;
@@ -1866,6 +1919,7 @@ function run_build_command {
echo >&2
echo >&2 "ERROR: Unknown command to run ${COMMAND_TO_RUN}"
echo >&2
+ exit 1
;;
esac
}
@@ -1923,13 +1977,6 @@ function run_breeze_command {
run_static_checks "$@"
;;
build_image)
- if [[ ${PUSH_IMAGES} == "true" ]]; then
- if [[ ${PRODUCTION_IMAGE} == "true" ]]; then
- push_prod_images
- else
- push_ci_image
- fi
- fi
;;
cleanup_image)
remove_images
@@ -1937,6 +1984,13 @@ function run_breeze_command {
perform_generate_requirements)
run_generate_requirements
;;
+ perform_push_image)
+ if [[ ${PRODUCTION_IMAGE} == "true" ]]; then
+ push_prod_images
+ else
+ push_ci_image
+ fi
+ ;;
perform_initialize_local_virtualenv)
initialize_virtualenv
;;
diff --git a/breeze-complete b/breeze-complete
index 8afe550..de39a83 100644
--- a/breeze-complete
+++ b/breeze-complete
@@ -79,6 +79,8 @@ EOF
_BREEZE_DEFAULT_DOCKERHUB_USER="apache"
_BREEZE_DEFAULT_DOCKERHUB_REPO="airflow"
+_BREEZE_DEFAULT_GITHUB_ORGANISATION="apache"
+_BREEZE_DEFAULT_GITHUB_REPO="airflow"
_BREEZE_SHORT_OPTIONS="
h p: b: i:
@@ -87,7 +89,7 @@ s x r
l a: t: d:
v y n q f
F P I E: C L
-D: R: u
+D: R: c g: G:
"
_BREEZE_LONG_OPTIONS="
@@ -97,7 +99,7 @@ kind-cluster-start kind-cluster-stop kind-cluster-recreate
skip-mounting-local-sources install-airflow-version: install-airflow-reference: db-reset
verbose assume-yes assume-no assume-quit forward-credentials
force-build-images force-pull-images production-image extras: force-clean-images use-local-cache
-dockerhub-user: dockerhub-repo: push-images
+dockerhub-user: dockerhub-repo: registry-cache github-organisation: github-repo:
postgres-version: mysql-version:
additional-extras: additional-python-deps:
"
@@ -109,6 +111,7 @@ build-image
cleanup-image
exec
generate-requirements
+push-image
initialize-local-virtualenv
setup-autocomplete
stop
@@ -179,6 +182,12 @@ function _get_known_values_breeze() {
-R | --dockerhub-repo)
_BREEZE_KNOWN_VALUES="${_BREEZE_DEFAULT_DOCKERHUB_REPO}"
;;
+ -G | --github-organisation)
+ _BREEZE_KNOWN_VALUES="${_BREEZE_DEFAULT_GITHUB_ORGANISATION}"
+ ;;
+ -g | --github-repo)
+ _BREEZE_KNOWN_VALUES="${_BREEZE_DEFAULT_GITHUB_REPO}"
+ ;;
*)
_BREEZE_KNOWN_VALUES=""
;;
diff --git a/scripts/ci/_utils.sh b/scripts/ci/_utils.sh
index 6c0204c..4ba7d83 100644
--- a/scripts/ci/_utils.sh
+++ b/scripts/ci/_utils.sh
@@ -83,6 +83,11 @@ function initialize_common_environment {
# Default branch name for triggered builds is the one configured in default branch
export BRANCH_NAME=${BRANCH_NAME:=${DEFAULT_BRANCH}}
+ export GITHUB_ORGANISATION=${GITHUB_ORGANISATION:="apache"}
+ export GITHUB_REPO=${GITHUB_REPO:="airflow"}
+ export CACHE_REGISTRY=${CACHE_REGISTRY:="docker.pkg.github.com"}
+ export ENABLE_REGISTRY_CACHE=${ENABLE_REGISTRY_CACHE:="false"}
+
# Default port numbers for forwarded ports
export WEBSERVER_HOST_PORT=${WEBSERVER_HOST_PORT:="28080"}
export POSTGRES_HOST_PORT=${POSTGRES_HOST_PORT:="25433"}
@@ -92,10 +97,6 @@ function initialize_common_environment {
export POSTGRES_VERSION=${POSTGRES_VERSION:="9.6"}
export MYSQL_VERSION=${MYSQL_VERSION:="5.6"}
- # Do not push images by default (push them directly from the build script on Dockerhub or when
- # --push-images flag is specified
- export PUSH_IMAGES=${PUSH_IMAGES:="false"}
-
# Whether base python images should be pulled from cache
export PULL_PYTHON_BASE_IMAGES_FROM_CACHE=${PULL_PYTHON_BASE_IMAGES_FROM_CACHE:="true"}
@@ -1455,11 +1456,14 @@ function prepare_ci_build() {
export AIRFLOW_CI_LOCAL_MANIFEST_IMAGE="local/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}-manifest"
export AIRFLOW_CI_REMOTE_MANIFEST_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}-manifest"
export AIRFLOW_CI_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}"
- if [[ ${CACHE_REGISTRY=""} != "" ]]; then
- echo "${CACHE_REGISTRY_PASSWORD}" | docker login \
- --username "${CACHE_REGISTRY_USERNAME}" \
- --password-stdin \
- "${CACHE_REGISTRY}"
+ if [[ ${ENABLE_REGISTRY_CACHE="false"} == "true" ]]; then
+ if [[ ${CACHE_REGISTRY_PASSWORD:=} != "" ]]; then
+ echo "${CACHE_REGISTRY_PASSWORD}" | docker login \
+ --username "${CACHE_REGISTRY_USERNAME}" \
+ --password-stdin \
+ "${CACHE_REGISTRY}"
+ fi
+ export CACHE_IMAGE_PREFIX=${CACHE_IMAGE_PREFX:=${GITHUB_ORGANISATION}/${GITHUB_REPO}}
export CACHED_AIRFLOW_CI_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/${AIRFLOW_CI_BASE_TAG}"
export CACHED_PYTHON_BASE_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster"
else
@@ -1543,11 +1547,14 @@ function prepare_prod_build() {
export ADDITIONAL_PYTHON_DEPS="${ADDITIONAL_PYTHON_DEPS:=""}"
export AIRFLOW_IMAGE="${AIRFLOW_PROD_IMAGE}"
- if [[ ${CACHE_REGISTRY=""} != "" ]]; then
- echo "${CACHE_REGISTRY_PASSWORD}" | docker login \
- --username "${CACHE_REGISTRY_USERNAME}" \
- --password-stdin \
- "${CACHE_REGISTRY}"
+ if [[ ${ENABLE_REGISTRY_CACHE="false"} == "true" ]]; then
+ if [[ ${CACHE_REGISTRY_PASSWORD:=} != "" ]]; then
+ echo "${CACHE_REGISTRY_PASSWORD}" | docker login \
+ --username "${CACHE_REGISTRY_USERNAME}" \
+ --password-stdin \
+ "${CACHE_REGISTRY}"
+ fi
+ export CACHE_IMAGE_PREFIX=${CACHE_IMAGE_PREFX:=${GITHUB_ORGANISATION}/${GITHUB_REPO}}
export CACHED_AIRFLOW_PROD_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/${AIRFLOW_PROD_BASE_TAG}"
export CACHED_AIRFLOW_PROD_BUILD_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/${AIRFLOW_PROD_BASE_TAG}-build"
export CACHED_PYTHON_BASE_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster"
diff --git a/scripts/ci/ci_fix_ownership.sh b/scripts/ci/ci_fix_ownership.sh
index b47d83a..775cc58 100755
--- a/scripts/ci/ci_fix_ownership.sh
+++ b/scripts/ci/ci_fix_ownership.sh
@@ -19,6 +19,7 @@
#
# Fixes ownership for files created inside container (files owned by root will be owned by host user)
#
+set -x
export PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION:-3.5}
@@ -28,6 +29,14 @@ export PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION:-3.5}
export AIRFLOW_CI_IMAGE=\
${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-ci
+export AIRFLOW_IMAGE=${AIRFLOW_CI_IMAGE}
+export WEBSERVER_HOST_PORT=28080
+HOST_USER_ID="$(id -ur)"
+HOST_GROUP_ID="$(id -gr)"
+
+export HOST_USER_ID
+export HOST_GROUP_ID
+
docker-compose \
-f "${MY_DIR}/docker-compose/base.yml" \
-f "${MY_DIR}/docker-compose/local.yml" \
diff --git a/scripts/ci/in_container/_in_container_utils.sh b/scripts/ci/in_container/_in_container_utils.sh
index be386e0..9b9814a 100644
--- a/scripts/ci/in_container/_in_container_utils.sh
+++ b/scripts/ci/in_container/_in_container_utils.sh
@@ -93,7 +93,7 @@ function in_container_cleanup_pycache() {
#
function in_container_fix_ownership() {
set +o pipefail
- sudo find . -user root -print0 \
+ sudo find "${AIRFLOW_SOURCES}" -print0 -user root \
| sudo xargs --null chown -v "${HOST_USER_ID}.${HOST_GROUP_ID}" --no-dereference >/dev/null 2>&1
set -o pipefail
}