You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2020/10/11 23:35:29 UTC
[airflow] branch v1-10-test updated (96a6f57 -> 3edc89c)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git.
from 96a6f57 Add TaskHandlersMovedRule in upgrade check command (#11265)
new d2ad37e Breeze start-airflow command wasn't able to initialize the db in 1.10.x (#11207)
new 722b1a4 Adds --no-rbac-ui flag for Breeze airflow 1.10 installation (#11315)
new 3c643c2 Better message when Building Image fails or gets cancelled. (#11333)
new e620ff8 Bump cache version for kubernetes tests (#11355)
new 3898a5f Better diagnostics when there are problems with Kerberos (#11353)
new 43e51e4 Use only-if-needed upgrade strategy for PRs (#11363)
new 932b565 Add pypirc initialization (#11386)
new dd3a404 Constraints and PIP packages can be installed from local sources (#11382)
new a7e6f93 Push and schedule duplicates are not cancelled. (#11397)
new f726c38 Fixes automated upgrade to latest constraints. (#11399)
new 73fbd26 Fixes cancelling of too many workflows. (#11403)
new 972a45b Workarounds "unknown blob" issue by introducing retries (#11411)
new dbc9ab9 Add capability of customising PyPI sources (#11385)
new 3edc89c Fixes SHA used for cancel-workflow-action (#11400)
The 14 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.dockerignore | 5 ++
.github/workflows/build-images-workflow-run.yml | 80 +++++++++++------
.github/workflows/ci.yml | 4 +-
.github/workflows/codeql-cancel.yml | 3 +-
.gitignore | 19 +++-
BREEZE.rst | 48 ++++++++--
Dockerfile | 32 +++++--
Dockerfile.ci | 5 +-
IMAGES.rst | 25 +++++-
breeze | 43 ++++++---
breeze-complete | 3 +-
docker-context-files/README.md | 31 +++++++
docs/production-deployment.rst | 106 +++++++++++++++++++++++
scripts/ci/docker-compose/base.yml | 2 +-
scripts/ci/libraries/_build_images.sh | 23 ++++-
scripts/ci/libraries/_initialization.sh | 25 ++++++
scripts/ci/libraries/_push_pull_remove_images.sh | 82 +++++++++++++++---
scripts/in_container/_in_container_utils.sh | 3 +
scripts/in_container/check_environment.sh | 9 +-
scripts/in_container/entrypoint_ci.sh | 11 +++
scripts/in_container/run_generate_constraints.sh | 2 +-
21 files changed, 471 insertions(+), 90 deletions(-)
create mode 100644 docker-context-files/README.md
[airflow] 14/14: Fixes SHA used for cancel-workflow-action (#11400)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 3edc89ce731102427800ca1393fb611a60fe3351
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Sun Oct 11 13:54:00 2020 +0200
Fixes SHA used for cancel-workflow-action (#11400)
The SHA of cancel-workflow-action in #11397 was pointing to previous
(3.1) version of the action. This PR fixes it to point to the
right (3.2) version.
(cherry picked from commit 4de8f85eecec4062210df66909272487196fd559)
---
.github/workflows/build-images-workflow-run.yml | 10 +++++-----
.github/workflows/codeql-cancel.yml | 2 +-
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/.github/workflows/build-images-workflow-run.yml b/.github/workflows/build-images-workflow-run.yml
index ad94cdc..b791fc3 100644
--- a/.github/workflows/build-images-workflow-run.yml
+++ b/.github/workflows/build-images-workflow-run.yml
@@ -65,7 +65,7 @@ jobs:
token: ${{ secrets.GITHUB_TOKEN }}
sourceRunId: ${{ github.event.workflow_run.id }}
- name: "Cancel duplicated 'CI Build' runs"
- uses: potiuk/cancel-workflow-runs@ca4b70a6910d33990e16d95e0c116914cdc0dfd0 # v3_2
+ uses: potiuk/cancel-workflow-runs@f696c622a83e4a63fff74848d3b149074658607d # v3_2
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: duplicates
@@ -84,7 +84,7 @@ jobs:
# in GitHub Actions, we have to use Job names to match Event/Repo/Branch from the
# build-info step there to find the duplicates ¯\_(ツ)_/¯.
- uses: potiuk/cancel-workflow-runs@ca4b70a6910d33990e16d95e0c116914cdc0dfd0 # v3_2
+ uses: potiuk/cancel-workflow-runs@f696c622a83e4a63fff74848d3b149074658607d # v3_2
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
@@ -105,7 +105,7 @@ jobs:
# can cancel all the matching "Build Images" workflow runs in the two following steps.
# Yeah. Adding to the complexity ¯\_(ツ)_/¯.
- uses: potiuk/cancel-workflow-runs@ca4b70a6910d33990e16d95e0c116914cdc0dfd0 # v3_2
+ uses: potiuk/cancel-workflow-runs@f696c622a83e4a63fff74848d3b149074658607d # v3_2
id: cancel-failed
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -140,7 +140,7 @@ jobs:
# it to cancel any jobs that have matching names containing Source Run Id:
# followed by one of the run ids. Yes I know it's super complex ¯\_(ツ)_/¯.
if: env.BUILD_IMAGES == 'true' && steps.source-run-info-failed.outputs.cancelledRuns != '[]'
- uses: potiuk/cancel-workflow-runs@ca4b70a6910d33990e16d95e0c116914cdc0dfd0 # v3_2
+ uses: potiuk/cancel-workflow-runs@f696c622a83e4a63fff74848d3b149074658607d # v3_2
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
@@ -362,7 +362,7 @@ jobs:
needs: [build-images]
steps:
- name: "Canceling the 'CI Build' source workflow in case of failure!"
- uses: potiuk/cancel-workflow-runs@ca4b70a6910d33990e16d95e0c116914cdc0dfd0 # v3_2
+ uses: potiuk/cancel-workflow-runs@f696c622a83e4a63fff74848d3b149074658607d # v3_2
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: self
diff --git a/.github/workflows/codeql-cancel.yml b/.github/workflows/codeql-cancel.yml
index 7dcda4b..2221d7a 100644
--- a/.github/workflows/codeql-cancel.yml
+++ b/.github/workflows/codeql-cancel.yml
@@ -12,7 +12,7 @@ jobs:
if: github.repository == 'apache/airflow' || github.event.workflow_run.event != 'schedule'
steps:
- name: "Cancel duplicated 'CodeQL' runs"
- uses: potiuk/cancel-workflow-runs@ca4b70a6910d33990e16d95e0c116914cdc0dfd0 # v3_2
+ uses: potiuk/cancel-workflow-runs@f696c622a83e4a63fff74848d3b149074658607d # v3_2
id: cancel
with:
token: ${{ secrets.GITHUB_TOKEN }}
[airflow] 08/14: Constraints and PIP packages can be installed from
local sources (#11382)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit dd3a4048ee25af90e2666ca7a4501cb7485ba9b0
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Sat Oct 10 12:58:09 2020 +0200
Constraints and PIP packages can be installed from local sources (#11382)
* Constraints and PIP packages can be installed from local sources
This is the final part of implementing #11171 based on feedback
from enterprise customers we worked with. They want to have
a capability of building the image using binary wheel packages
that are locally available and the official Dockerfile. This means
that besides the official APT sources the Dockerfile build should
not needd GitHub, nor any other external files pulled from outside
including PIP repository.
This change also includes documentation on how to prepare set of
such binaries ready for inspection and review by security teams
in Enterprise environment. Such sets of "known-working-binary-whl"
files can then be separately committed, tracked and scrutinized
in an artifact repository of such an Enterprise.
Fixes: #11171
* Update docs/production-deployment.rst
(cherry picked from commit 04973904c3652fac4a8efc168d2b36f8a9245257)
---
.dockerignore | 4 ++
BREEZE.rst | 48 +++++++++++++----
Dockerfile | 28 ++++++----
IMAGES.rst | 25 +++++++--
breeze | 40 ++++++++++----
breeze-complete | 2 +-
docker-context-files/README.md | 31 +++++++++++
docs/production-deployment.rst | 66 ++++++++++++++++++++++++
scripts/ci/libraries/_build_images.sh | 23 ++++++++-
scripts/ci/libraries/_initialization.sh | 12 +++++
scripts/in_container/run_generate_constraints.sh | 2 +-
11 files changed, 246 insertions(+), 35 deletions(-)
diff --git a/.dockerignore b/.dockerignore
index fb9c80e..8a8bb52 100644
--- a/.dockerignore
+++ b/.dockerignore
@@ -56,6 +56,10 @@
!empty
!.pypirc
+# This folder is for you if you want to add any files to the docker context when you build your own
+# docker image. most of other files and any new folder you add will be excluded by default
+!docker-context-files
+
# Avoid triggering context change on README change (new companies using Airflow)
# So please do not uncomment this line ;)
# !README.md
diff --git a/BREEZE.rst b/BREEZE.rst
index 5b23c84..6bf6f7e 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -1212,9 +1212,6 @@ This is the current syntax for `./breeze <./breeze>`_:
--image-tag TAG
Additional tag in the image.
- --disable-pip-cache
- Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
-
--additional-extras ADDITIONAL_EXTRAS
Additional extras to pass to build images The default is no additional extras.
@@ -1260,6 +1257,19 @@ This is the current syntax for `./breeze <./breeze>`_:
Disables installation of the mysql client which might be problematic if you are building
image in controlled environment. Only valid for production image.
+ --constraints-location
+ Url to the constraints file. In case of the production image it can also be a path to the
+ constraint file placed in 'docker-context-files' folder, in which case it has to be
+ in the form of '/docker-context-files/<NAME_OF_THE_FILE>'
+
+ --disable-pip-cache
+ Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
+
+ --install-local-pip-wheels
+ This flag is only used in production image building. If it is used then instead of
+ installing Airflow from PyPI, the packages are installed from the .whl packages placed
+ in the 'docker-context-files' folder. It implies '--disable-pip-cache'
+
-C, --force-clean-images
Force build images with cache disabled. This will remove the pulled or build images
and start building images from scratch. This might take a long time.
@@ -1700,9 +1710,6 @@ This is the current syntax for `./breeze <./breeze>`_:
--image-tag TAG
Additional tag in the image.
- --disable-pip-cache
- Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
-
--additional-extras ADDITIONAL_EXTRAS
Additional extras to pass to build images The default is no additional extras.
@@ -1748,6 +1755,19 @@ This is the current syntax for `./breeze <./breeze>`_:
Disables installation of the mysql client which might be problematic if you are building
image in controlled environment. Only valid for production image.
+ --constraints-location
+ Url to the constraints file. In case of the production image it can also be a path to the
+ constraint file placed in 'docker-context-files' folder, in which case it has to be
+ in the form of '/docker-context-files/<NAME_OF_THE_FILE>'
+
+ --disable-pip-cache
+ Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
+
+ --install-local-pip-wheels
+ This flag is only used in production image building. If it is used then instead of
+ installing Airflow from PyPI, the packages are installed from the .whl packages placed
+ in the 'docker-context-files' folder. It implies '--disable-pip-cache'
+
-C, --force-clean-images
Force build images with cache disabled. This will remove the pulled or build images
and start building images from scratch. This might take a long time.
@@ -2051,9 +2071,6 @@ This is the current syntax for `./breeze <./breeze>`_:
--image-tag TAG
Additional tag in the image.
- --disable-pip-cache
- Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
-
--additional-extras ADDITIONAL_EXTRAS
Additional extras to pass to build images The default is no additional extras.
@@ -2099,6 +2116,19 @@ This is the current syntax for `./breeze <./breeze>`_:
Disables installation of the mysql client which might be problematic if you are building
image in controlled environment. Only valid for production image.
+ --constraints-location
+ Url to the constraints file. In case of the production image it can also be a path to the
+ constraint file placed in 'docker-context-files' folder, in which case it has to be
+ in the form of '/docker-context-files/<NAME_OF_THE_FILE>'
+
+ --disable-pip-cache
+ Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
+
+ --install-local-pip-wheels
+ This flag is only used in production image building. If it is used then instead of
+ installing Airflow from PyPI, the packages are installed from the .whl packages placed
+ in the 'docker-context-files' folder. It implies '--disable-pip-cache'
+
-C, --force-clean-images
Force build images with cache disabled. This will remove the pulled or build images
and start building images from scratch. This might take a long time.
diff --git a/Dockerfile b/Dockerfile
index cb46c69..f257606 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -140,6 +140,8 @@ ARG INSTALL_MYSQL_CLIENT="true"
ENV INSTALL_MYSQL_CLIENT=${INSTALL_MYSQL_CLIENT}
COPY scripts/docker scripts/docker
+COPY docker-context-files /docker-context-files
+
RUN ./scripts/docker/install_mysql.sh dev
ARG AIRFLOW_REPO=apache/airflow
@@ -153,8 +155,8 @@ ARG ADDITIONAL_AIRFLOW_EXTRAS=""
ENV AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}${ADDITIONAL_AIRFLOW_EXTRAS:+,}${ADDITIONAL_AIRFLOW_EXTRAS}
ARG AIRFLOW_CONSTRAINTS_REFERENCE="constraints-master"
-ARG AIRFLOW_CONSTRAINTS_URL="https://raw.githubusercontent.com/apache/airflow/${AIRFLOW_CONSTRAINTS_REFERENCE}/constraints-${PYTHON_MAJOR_MINOR_VERSION}.txt"
-ENV AIRFLOW_CONSTRAINTS_URL=${AIRFLOW_CONSTRAINTS_URL}
+ARG AIRFLOW_CONSTRAINTS_LOCATION="https://raw.githubusercontent.com/apache/airflow/${AIRFLOW_CONSTRAINTS_REFERENCE}/constraints-${PYTHON_MAJOR_MINOR_VERSION}.txt"
+ENV AIRFLOW_CONSTRAINTS_LOCATION=${AIRFLOW_CONSTRAINTS_LOCATION}
ENV PATH=${PATH}:/root/.local/bin
RUN mkdir -p /root/.local/bin
@@ -170,7 +172,7 @@ RUN if [[ ${AIRFLOW_PRE_CACHED_PIP_PACKAGES} == "true" ]]; then \
fi; \
pip install --user \
"https://github.com/${AIRFLOW_REPO}/archive/${AIRFLOW_BRANCH}.tar.gz#egg=apache-airflow[${AIRFLOW_EXTRAS}]" \
- --constraint "${AIRFLOW_CONSTRAINTS_URL}" && pip uninstall --yes apache-airflow; \
+ --constraint "${AIRFLOW_CONSTRAINTS_LOCATION}" && pip uninstall --yes apache-airflow; \
fi
ARG AIRFLOW_SOURCES_FROM="."
@@ -196,6 +198,9 @@ ENV AIRFLOW_INSTALL_SOURCES=${AIRFLOW_INSTALL_SOURCES}
ARG AIRFLOW_INSTALL_VERSION=""
ENV AIRFLOW_INSTALL_VERSION=${AIRFLOW_INSTALL_VERSION}
+ARG AIRFLOW_LOCAL_PIP_WHEELS=""
+ENV AIRFLOW_LOCAL_PIP_WHEELS=${AIRFLOW_LOCAL_PIP_WHEELS}
+
ARG SLUGIFY_USES_TEXT_UNIDECODE=""
ENV SLUGIFY_USES_TEXT_UNIDECODE=${SLUGIFY_USES_TEXT_UNIDECODE}
@@ -205,12 +210,17 @@ WORKDIR /opt/airflow
RUN if [[ ${INSTALL_MYSQL_CLIENT} != "true" ]]; then \
AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS/mysql,}; \
fi; \
- pip install --user "${AIRFLOW_INSTALL_SOURCES}[${AIRFLOW_EXTRAS}]${AIRFLOW_INSTALL_VERSION}" \
- --constraint "${AIRFLOW_CONSTRAINTS_URL}" && \
- if [ -n "${ADDITIONAL_PYTHON_DEPS}" ]; then pip install --user ${ADDITIONAL_PYTHON_DEPS} \
- --constraint "${AIRFLOW_CONSTRAINTS_URL}"; fi && \
- find /root/.local/ -name '*.pyc' -print0 | xargs -0 rm -r && \
- find /root/.local/ -type d -name '__pycache__' -print0 | xargs -0 rm -r
+ if [[ ${AIRFLOW_LOCAL_PIP_WHEELS} != "true" ]]; then \
+ pip install --user "${AIRFLOW_INSTALL_SOURCES}[${AIRFLOW_EXTRAS}]${AIRFLOW_INSTALL_VERSION}" \
+ --constraint "${AIRFLOW_CONSTRAINTS_LOCATION}"; \
+ if [ -n "${ADDITIONAL_PYTHON_DEPS}" ]; then \
+ pip install --user ${ADDITIONAL_PYTHON_DEPS} --constraint "${AIRFLOW_CONSTRAINTS_LOCATION}"; \
+ fi; \
+ else \
+ pip install --user /docker-context-files/*.whl; \
+ fi \
+ && find /root/.local/ -name '*.pyc' -print0 | xargs -0 rm -r \
+ && find /root/.local/ -type d -name '__pycache__' -print0 | xargs -0 rm -r
RUN AIRFLOW_SITE_PACKAGE="/root/.local/lib/python${PYTHON_MAJOR_MINOR_VERSION}/site-packages/airflow"; \
if [[ -f "${AIRFLOW_SITE_PACKAGE}/www_rbac/package.json" ]]; then \
diff --git a/IMAGES.rst b/IMAGES.rst
index 68467c6..ecfc388 100644
--- a/IMAGES.rst
+++ b/IMAGES.rst
@@ -22,8 +22,13 @@ Airflow docker images
Airflow has two images (build from Dockerfiles):
-* Production image (Dockerfile) - that can be used to build your own production-ready Airflow installation
-* CI image (Dockerfile.ci) - used for running tests and local development
+ * Production image (Dockerfile) - that can be used to build your own production-ready Airflow installation
+ You can read more about building and using the production image in the
+ `Production Deployments <docs/production-deployment.rst>`_ document. The image is built using
+ `Dockerfile <Dockerfile>`_
+
+ * CI image (Dockerfile.ci) - used for running tests and local development. The image is built using
+ `Dockerfile.ci <Dockerfile.ci>`_
Image naming conventions
========================
@@ -332,7 +337,6 @@ based on example in `this comment <https://github.com/apache/airflow/issues/8605
--build-arg ADDITIONAL_RUNTIME_ENV_VARS="ACCEPT_EULA=Y" \
--tag my-image
-
CI image build arguments
........................
@@ -378,6 +382,21 @@ The following build arguments (``--build-arg`` in docker build command) can be u
| | | dependencies from the repository from |
| | | scratch |
+------------------------------------------+------------------------------------------+------------------------------------------+
+| ``AIRFLOW_CONSTRAINTS_LOCATION`` | | If not empty, it will override the |
+| | | source of the constraints with the |
+| | | specified URL or file. Note that the |
+| | | file has to be in docker context so |
+| | | it's best to place such file in |
+| | | one of the folders included in |
+| | | dockerignore |
++------------------------------------------+------------------------------------------+------------------------------------------+
+| ``AIRFLOW_LOCAL_PIP_WHEELS`` | ``false`` | If set to true, Airflow and it's |
+| | | dependencies are installed from locally |
+| | | downloaded .whl files placed in the |
+| | | ``docker-context-files``. Implies |
+| | | ``AIRFLOW_PRE_CACHED_PIP_PACKAGES`` |
+| | | to be false. |
++------------------------------------------+------------------------------------------+------------------------------------------+
| ``AIRFLOW_EXTRAS`` | ``all`` | extras to install |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``AIRFLOW_PRE_CACHED_PIP_PACKAGES`` | ``true`` | Allows to pre-cache airflow PIP packages |
diff --git a/breeze b/breeze
index 1878e31..fc74cac 100755
--- a/breeze
+++ b/breeze
@@ -910,13 +910,6 @@ function breeze::parse_arguments() {
# if not set here, docker cached is determined later, depending on type of image to be build
shift
;;
- -B | --disable-pip-cache)
- echo "Disable PIP cache during build"
- echo
- export AIRFLOW_PRE_CACHED_PIP_PACKAGES="false"
- shift
- ;;
-
-P | --force-pull-images)
echo "Force pulling images before build. Uses pulled images as cache."
echo
@@ -1004,6 +997,23 @@ function breeze::parse_arguments() {
echo "Install MySQL client: ${INSTALL_MYSQL_CLIENT}"
shift
;;
+ --constraints-location)
+ export AIRFLOW_CONSTRAINTS_LOCATION="${2}"
+ echo "Constraints location: ${AIRFLOW_CONSTRAINTS_LOCATION}"
+ shift 2
+ ;;
+ --disable-pip-cache)
+ echo "Disable PIP cache during build"
+ echo
+ export AIRFLOW_PRE_CACHED_PIP_PACKAGES="false"
+ shift
+ ;;
+ --install-local-pip-wheels)
+ export AIRFLOW_LOCAL_PIP_WHEELS="true"
+ export AIRFLOW_PRE_CACHED_PIP_PACKAGES="false"
+ echo "Install from local wheels and disable pip cache"
+ shift
+ ;;
--image-tag)
export IMAGE_TAG="${2}"
echo "Tag to add to the image: ${IMAGE_TAG}"
@@ -2175,9 +2185,6 @@ ${FORMATTED_DEFAULT_PROD_EXTRAS}
--image-tag TAG
Additional tag in the image.
---disable-pip-cache
- Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
-
--additional-extras ADDITIONAL_EXTRAS
Additional extras to pass to build images The default is no additional extras.
@@ -2223,6 +2230,19 @@ Build options:
Disables installation of the mysql client which might be problematic if you are building
image in controlled environment. Only valid for production image.
+--constraints-location
+ Url to the constraints file. In case of the production image it can also be a path to the
+ constraint file placed in 'docker-context-files' folder, in which case it has to be
+ in the form of '/docker-context-files/<NAME_OF_THE_FILE>'
+
+--disable-pip-cache
+ Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
+
+--install-local-pip-wheels
+ This flag is only used in production image building. If it is used then instead of
+ installing Airflow from PyPI, the packages are installed from the .whl packages placed
+ in the 'docker-context-files' folder. It implies '--disable-pip-cache'
+
-C, --force-clean-images
Force build images with cache disabled. This will remove the pulled or build images
and start building images from scratch. This might take a long time.
diff --git a/breeze-complete b/breeze-complete
index 5c58e50..6a62eaf 100644
--- a/breeze-complete
+++ b/breeze-complete
@@ -136,7 +136,7 @@ build-cache-local build-cache-pulled build-cache-disabled disable-pip-cache
dockerhub-user: dockerhub-repo: github-registry github-repository: github-image-id:
postgres-version: mysql-version:
additional-extras: additional-python-deps: additional-dev-deps: additional-runtime-deps: image-tag:
-disable-mysql-client-installation
+disable-mysql-client-installation constraints-location: disable-pip-cache install-local-pip-wheels
additional-extras: additional-python-deps:
dev-apt-deps: additional-dev-apt-deps: dev-apt-command: additional-dev-apt-command: additional-dev-apt-env:
runtime-apt-deps: additional-runtime-apt-deps: runtime-apt-command: additional-runtime-apt-command: additional-runtime-apt-env:
diff --git a/docker-context-files/README.md b/docker-context-files/README.md
new file mode 100644
index 0000000..f53fbfe
--- /dev/null
+++ b/docker-context-files/README.md
@@ -0,0 +1,31 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+This folder is par of the Docker context.
+
+Most of other folders in Airflow are not part of the context in order to make the context smaller.
+
+The Production [Dockerfile](../Dockerfile) copies th [docker-context-files](.) folder to the "build"
+stage of the production image (it is not used in the CI image) and content of the folder is available
+in the `/docker-context-file` folder inside the build image. You can store constraint files and wheel
+packages there that you want to install as PYPI packages and refer to those packages using
+`--constraint-location` flag for constraints or by using `--install-local-pip-wheels` flag.
+
+By default, the content of this folder is .gitignored so that any binaries and files you put here are only
+used for local builds and not committed to the repository.
diff --git a/docs/production-deployment.rst b/docs/production-deployment.rst
index 95e08ed..5e6cad2 100644
--- a/docs/production-deployment.rst
+++ b/docs/production-deployment.rst
@@ -189,6 +189,71 @@ based on example in `this comment <https://github.com/apache/airflow/issues/8605
--build-arg ADDITIONAL_RUNTIME_ENV_VARS="ACCEPT_EULA=Y" \
--tag my-image
+Customizing images in high security restricted environments
+...........................................................
+
+You can also make sure your image is only build using local constraint file and locally downloaded
+wheel files. This is often useful in Enterprise environments where the binary files are verified and
+vetted by the security teams.
+
+This builds below builds the production image in version 3.7 with packages and constraints used from the local
+``docker-context-files`` rather than installed from PyPI or GitHub. It also disables MySQL client
+installation as it is using external installation method.
+
+Note that as a prerequisite - you need to have downloaded wheel files. In the example below we
+first download such constraint file locally and then use ``pip download`` to get the .whl files needed
+but in most likely scenario, those wheel files should be copied from an internal repository of such .whl
+files. Note that ``AIRFLOW_INSTALL_VERSION`` is only there for reference, the apache airflow .whl file
+in the right version is part of the .whl files downloaded.
+
+Note that 'pip download' will only works on Linux host as some of the packages need to be compiled from
+sources and you cannot install them providing ``--platform`` switch. They also need to be downloaded using
+the same python version as the target image.
+
+The ``pip download`` might happen in a separate environment. The files can be committed to a separate
+binary repository and vetted/verified by the security team and used subsequently to build images
+of Airflow when needed on an air-gaped system.
+
+Preparing the constraint files and wheel files:
+
+.. code-block:: bash
+
+ rm docker-context-files/*.whl docker-context-files/*.txt
+
+ curl -Lo "docker-context-files/constraints-1-10.txt" \
+ https://raw.githubusercontent.com/apache/airflow/constraints-1-10/constraints-3.7.txt
+
+ pip download --dest docker-context-files \
+ --constraint docker-context-files/constraints-1-10.txt \
+ apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,ssh,statsd,virtualenv]==1.10.12
+
+
+Building the image (after copying the files downloaded to the "docker-context-files" directory:
+
+.. code-block:: bash
+
+ ./breeze build-image \
+ --production-image --python 3.7 --install-airflow-version=1.10.12 \
+ --disable-mysql-client-installation --disable-pip-cache --install-local-pip-wheels \
+ --constraints-location="/docker-context-files/constraints-1-10.txt"
+
+or
+
+.. code-block:: bash
+
+ docker build . \
+ --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
+ --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \
+ --build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \
+ --build-arg AIRFLOW_INSTALL_VERSION="==1.10.12" \
+ --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1-10" \
+ --build-arg AIRFLOW_SOURCES_FROM="empty" \
+ --build-arg AIRFLOW_SOURCES_TO="/empty" \
+ --build-arg INSTALL_MYSQL_CLIENT="false" \
+ --build-arg AIRFLOW_PRE_CACHED_PIP_PACKAGES="false" \
+ --build-arg AIRFLOW_LOCAL_PIP_WHEELS="true" \
+ --build-arg AIRFLOW_CONSTRAINTS_LOCATION="/docker-context-files/constraints-1-10.txt"
+
Customizing & extending the image together
..........................................
@@ -524,6 +589,7 @@ additional apt dev and runtime dependencies.
--build-arg ADDITIONAL_RUNTIME_APT_DEPS="default-jre-headless"
+
More details about the images
-----------------------------
diff --git a/scripts/ci/libraries/_build_images.sh b/scripts/ci/libraries/_build_images.sh
index 9f391e0..5454116 100644
--- a/scripts/ci/libraries/_build_images.sh
+++ b/scripts/ci/libraries/_build_images.sh
@@ -26,6 +26,16 @@ function build_images::add_build_args_for_remote_install() {
"--build-arg" "AIRFLOW_SOURCES_FROM=empty"
"--build-arg" "AIRFLOW_SOURCES_TO=/empty"
)
+ if [[ ${AIRFLOW_CONSTRAINTS_REFERENCE} != "" ]]; then
+ EXTRA_DOCKER_PROD_BUILD_FLAGS+=(
+ "--build-arg" "AIRFLOW_CONSTRAINTS_REFERENCE=${AIRFLOW_CONSTRAINTS_REFERENCE}"
+ )
+ fi
+ if [[ "${AIRFLOW_CONSTRAINTS_LOCATION}" != "" ]]; then
+ EXTRA_DOCKER_PROD_BUILD_FLAGS+=(
+ "--build-arg" "AIRFLOW_CONSTRAINTS_LOCATION=${AIRFLOW_CONSTRAINTS_LOCATION}"
+ )
+ fi
if [[ ${AIRFLOW_VERSION} =~ [^0-9]*1[^0-9]*10[^0-9]([0-9]*) ]]; then
# All types of references/versions match this regexp for 1.10 series
# for example v1_10_test, 1.10.10, 1.10.9 etc. ${BASH_REMATCH[1]} matches last
@@ -530,6 +540,13 @@ function build_images::build_ci_image() {
EXTRA_DOCKER_CI_BUILD_FLAGS=(
"--build-arg" "AIRFLOW_CONSTRAINTS_REFERENCE=${DEFAULT_CONSTRAINTS_BRANCH}"
)
+
+ if [[ "${AIRFLOW_CONSTRAINTS_LOCATION}" != "" ]]; then
+ EXTRA_DOCKER_CI_BUILD_FLAGS+=(
+ "--build-arg" "AIRFLOW_CONSTRAINTS_LOCATION=${AIRFLOW_CONSTRAINTS_LOCATION}"
+ )
+ fi
+
if [[ -n ${SPIN_PID:=""} ]]; then
kill -HUP "${SPIN_PID}" || true
wait "${SPIN_PID}" || true
@@ -722,13 +739,14 @@ function build_images::build_prod_images() {
--build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
--build-arg AIRFLOW_BRANCH="${AIRFLOW_BRANCH_FOR_PYPI_PRELOADING}" \
--build-arg AIRFLOW_EXTRAS="${AIRFLOW_EXTRAS}" \
- --build-arg AIRFLOW_PRE_CACHED_PIP_PACKAGES="${AIRFLOW_PRE_CACHED_PIP_PACKAGES}" \
--build-arg ADDITIONAL_AIRFLOW_EXTRAS="${ADDITIONAL_AIRFLOW_EXTRAS}" \
--build-arg ADDITIONAL_PYTHON_DEPS="${ADDITIONAL_PYTHON_DEPS}" \
"${additional_dev_args[@]}" \
--build-arg ADDITIONAL_DEV_APT_COMMAND="${ADDITIONAL_DEV_APT_COMMAND}" \
--build-arg ADDITIONAL_DEV_APT_DEPS="${ADDITIONAL_DEV_APT_DEPS}" \
--build-arg ADDITIONAL_DEV_APT_ENV="${ADDITIONAL_DEV_APT_ENV}" \
+ --build-arg AIRFLOW_PRE_CACHED_PIP_PACKAGES="${AIRFLOW_PRE_CACHED_PIP_PACKAGES}" \
+ --build-arg AIRFLOW_LOCAL_PIP_WHEELS="${AIRFLOW_LOCAL_PIP_WHEELS}" \
--build-arg BUILD_ID="${CI_BUILD_ID}" \
--build-arg COMMIT_SHA="${COMMIT_SHA}" \
"${DOCKER_CACHE_PROD_BUILD_DIRECTIVE[@]}" \
@@ -755,10 +773,11 @@ function build_images::build_prod_images() {
--build-arg ADDITIONAL_RUNTIME_APT_COMMAND="${ADDITIONAL_RUNTIME_APT_COMMAND}" \
--build-arg ADDITIONAL_RUNTIME_APT_DEPS="${ADDITIONAL_RUNTIME_APT_DEPS}" \
--build-arg ADDITIONAL_RUNTIME_APT_ENV="${ADDITIONAL_RUNTIME_APT_ENV}" \
+ --build-arg AIRFLOW_PRE_CACHED_PIP_PACKAGES="${AIRFLOW_PRE_CACHED_PIP_PACKAGES}" \
+ --build-arg AIRFLOW_LOCAL_PIP_WHEELS="${AIRFLOW_LOCAL_PIP_WHEELS}" \
--build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
--build-arg AIRFLOW_BRANCH="${AIRFLOW_BRANCH_FOR_PYPI_PRELOADING}" \
--build-arg AIRFLOW_EXTRAS="${AIRFLOW_EXTRAS}" \
- --build-arg AIRFLOW_PRE_CACHED_PIP_PACKAGES="${AIRFLOW_PRE_CACHED_PIP_PACKAGES}" \
--build-arg BUILD_ID="${CI_BUILD_ID}" \
--build-arg COMMIT_SHA="${COMMIT_SHA}" \
"${additional_dev_args[@]}" \
diff --git a/scripts/ci/libraries/_initialization.sh b/scripts/ci/libraries/_initialization.sh
index 2139aa6..1b80364 100644
--- a/scripts/ci/libraries/_initialization.sh
+++ b/scripts/ci/libraries/_initialization.sh
@@ -356,6 +356,14 @@ function initialization::initialize_image_build_variables() {
export INSTALL_MYSQL_CLIENT=${INSTALL_MYSQL_CLIENT:="true"}
# additional tag for the image
export IMAGE_TAG=${IMAGE_TAG:=""}
+
+ # whether installation should be performed from the local wheel packages in "docker-context-files" folder
+ export AIRFLOW_LOCAL_PIP_WHEELS="${AIRFLOW_LOCAL_PIP_WHEELS:="false"}"
+ # reference to CONSTRAINTS. they can be overwritten manually or replaced with AIRFLOW_CONSTRAINTS_LOCATION
+ export AIRFLOW_CONSTRAINTS_REFERENCE="${AIRFLOW_CONSTRAINTS_REFERENCE:=""}"
+ # direct constraints Location - can be URL or path to local file. If empty, it will be calculated
+ # based on which Airflow version is installed and from where
+ export AIRFLOW_CONSTRAINTS_LOCATION="${AIRFLOW_CONSTRAINTS_LOCATION:=""}"
}
# Determine version suffixes used to build backport packages
@@ -673,6 +681,10 @@ function initialization::make_constants_read_only() {
readonly IMAGE_TAG
readonly AIRFLOW_PRE_CACHED_PIP_PACKAGES
+ readonly AIRFLOW_LOCAL_PIP_WHEELS
+ readonly AIRFLOW_CONSTRAINTS_REFERENCE
+ readonly AIRFLOW_CONSTRAINTS_LOCATION
+
# AIRFLOW_EXTRAS are made readonly by the time the image is built (either PROD or CI)
readonly ADDITIONAL_AIRFLOW_EXTRAS
readonly ADDITIONAL_PYTHON_DEPS
diff --git a/scripts/in_container/run_generate_constraints.sh b/scripts/in_container/run_generate_constraints.sh
index d361b0e..dabf698 100755
--- a/scripts/in_container/run_generate_constraints.sh
+++ b/scripts/in_container/run_generate_constraints.sh
@@ -27,7 +27,7 @@ CURRENT_CONSTRAINT_FILE="${CONSTRAINTS_DIR}/constraints-${PYTHON_MAJOR_MINOR_VER
mkdir -pv "${CONSTRAINTS_DIR}"
-curl "${AIRFLOW_CONSTRAINTS_URL}" --output "${LATEST_CONSTRAINT_FILE}"
+curl "${AIRFLOW_CONSTRAINTS_LOCATION}" --output "${LATEST_CONSTRAINT_FILE}"
echo
echo "Freezing constraints to ${CURRENT_CONSTRAINT_FILE}"
[airflow] 02/14: Adds --no-rbac-ui flag for Breeze airflow 1.10
installation (#11315)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 722b1a480a3024f6f2063e260579a94ba5a22cac
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Wed Oct 7 02:00:00 2020 +0200
Adds --no-rbac-ui flag for Breeze airflow 1.10 installation (#11315)
When installing airflow 1.10 via breeze we now enable rbac
by default, but we can disable it with --no-rbac-ui flag.
This is useful to test different variants of 1.10 when testing
release candidataes in connection with the 'start-airflow'
command.
(cherry picked from commit 22c6a843d760d920f329fc97aa55f45d82682ab9)
---
breeze | 3 +--
breeze-complete | 1 +
scripts/ci/docker-compose/base.yml | 2 +-
scripts/ci/libraries/_initialization.sh | 7 +++++++
scripts/in_container/entrypoint_ci.sh | 11 +++++++++++
5 files changed, 21 insertions(+), 3 deletions(-)
diff --git a/breeze b/breeze
index 63471d0..1878e31 100755
--- a/breeze
+++ b/breeze
@@ -63,7 +63,6 @@ export EXTRA_STATIC_CHECK_OPTIONS
# FORWARD_CREDENTIALS
# DB_RESET
# START_AIRFLOW
-# RBAC_UI
# INSTALL_AIRFLOW_VERSION
# INSTALL_AIRFLOW_REFERENCE
# FORCE_BUILD_IMAGES
@@ -121,7 +120,7 @@ function breeze::setup_default_breeze_constants() {
# If set to true, the test connections will be created
export LOAD_DEFAULT_CONNECTIONS="false"
- # If set to true, the sample dags will be created
+ # If set to true, the sample dags will be used
export LOAD_EXAMPLES="false"
# If set to true, RBAC mode is enabled
diff --git a/breeze-complete b/breeze-complete
index 8b1d597..5c58e50 100644
--- a/breeze-complete
+++ b/breeze-complete
@@ -141,6 +141,7 @@ additional-extras: additional-python-deps:
dev-apt-deps: additional-dev-apt-deps: dev-apt-command: additional-dev-apt-command: additional-dev-apt-env:
runtime-apt-deps: additional-runtime-apt-deps: runtime-apt-command: additional-runtime-apt-command: additional-runtime-apt-env:
load-default-connections load-example-dags
+no-rbac-ui
"
_breeze_commands="
diff --git a/scripts/ci/docker-compose/base.yml b/scripts/ci/docker-compose/base.yml
index 706d90c..a0e7728 100644
--- a/scripts/ci/docker-compose/base.yml
+++ b/scripts/ci/docker-compose/base.yml
@@ -52,7 +52,7 @@ services:
- START_AIRFLOW
- LOAD_EXAMPLES
- LOAD_DEFAULT_CONNECTIONS
- - RBAC_UI
+ - DISABLE_RBAC
- ENABLED_SYSTEMS
- RUN_SYSTEM_TESTS
- PYTHON_MAJOR_MINOR_VERSION
diff --git a/scripts/ci/libraries/_initialization.sh b/scripts/ci/libraries/_initialization.sh
index e71ce39..4aa72e1 100644
--- a/scripts/ci/libraries/_initialization.sh
+++ b/scripts/ci/libraries/_initialization.sh
@@ -111,9 +111,15 @@ function initialization::initialize_base_variables() {
# If set to true, the database will be initialized, a user created and webserver and scheduler started
export START_AIRFLOW=${START_AIRFLOW:="false"}
+ # If set to true, the sample dags will be used
export LOAD_EXAMPLES=${LOAD_EXAMPLES:="false"}
+ # If set to true, the test connections will be created
export LOAD_DEFAULT_CONNECTIONS=${LOAD_DEFAULT_CONNECTIONS:="false"}
+
+ # If set to true, RBAC UI will not be used for 1.10 version
+ export DISABLE_RBAC=${DISABLE_RBAC:="false"}
+
# If set the specified file will be used to initialized Airflow after the environment is created,
# otherwise it will use files/airflow-breeze-config/init.sh
export INIT_SCRIPT_FILE=${INIT_SCRIPT_FILE:=""}
@@ -543,6 +549,7 @@ Initialization variables:
INIT_SCRIPT_FILE: ${INIT_SCRIPT_FILE}
LOAD_DEFAULT_CONNECTIONS: ${LOAD_DEFAULT_CONNECTIONS}
LOAD_EXAMPLES: ${LOAD_EXAMPLES}
+ DISABLE_RBAC: ${DISABLE_RBAC}
EOF
diff --git a/scripts/in_container/entrypoint_ci.sh b/scripts/in_container/entrypoint_ci.sh
index e74fe40..7a24875 100755
--- a/scripts/in_container/entrypoint_ci.sh
+++ b/scripts/in_container/entrypoint_ci.sh
@@ -19,6 +19,15 @@ if [[ ${VERBOSE_COMMANDS:="false"} == "true" ]]; then
set -x
fi
+function disable_rbac_if_requested() {
+ if [[ ${DISABLE_RBAC:="false"} == "true" ]]; then
+ export AIRFLOW__WEBSERVER__RBAC="False"
+ else
+ export AIRFLOW__WEBSERVER__RBAC="True"
+ fi
+}
+
+
# shellcheck source=scripts/in_container/_in_container_script_init.sh
. /opt/airflow/scripts/in_container/_in_container_script_init.sh
@@ -100,6 +109,8 @@ unset AIRFLOW__CORE__UNIT_TEST_MODE
mkdir -pv "${AIRFLOW_HOME}/logs/"
cp -f "${IN_CONTAINER_DIR}/airflow_ci.cfg" "${AIRFLOW_HOME}/unittests.cfg"
+disable_rbac_if_requested
+
set +e
"${IN_CONTAINER_DIR}/check_environment.sh"
ENVIRONMENT_EXIT_CODE=$?
[airflow] 03/14: Better message when Building Image fails or gets
cancelled. (#11333)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 3c643c259ef4adda9ec8004f4817da6ab455fd76
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Thu Oct 8 13:09:34 2020 +0200
Better message when Building Image fails or gets cancelled. (#11333)
(cherry picked from commit 9dc32a3d8a3164dced57b5ed5a488acfe6799b31)
---
.github/workflows/build-images-workflow-run.yml | 35 ++++++++++++++++++-------
.github/workflows/codeql-cancel.yml | 2 +-
2 files changed, 27 insertions(+), 10 deletions(-)
diff --git a/.github/workflows/build-images-workflow-run.yml b/.github/workflows/build-images-workflow-run.yml
index 5984a51..a30947d 100644
--- a/.github/workflows/build-images-workflow-run.yml
+++ b/.github/workflows/build-images-workflow-run.yml
@@ -59,7 +59,7 @@ jobs:
if: github.repository == 'apache/airflow' || github.event.workflow_run.event != 'schedule'
steps:
- name: "Cancel duplicated 'CI Build' runs"
- uses: potiuk/cancel-workflow-runs@v3
+ uses: potiuk/cancel-workflow-runs@ca4b70a6910d33990e16d95e0c116914cdc0dfd0 #v3_1
id: cancel
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -78,7 +78,7 @@ jobs:
# in GitHub Actions, we have to use Job names to match Event/Repo/Branch from the
# build-info step there to find the duplicates ¯\_(ツ)_/¯.
- uses: potiuk/cancel-workflow-runs@v3
+ uses: potiuk/cancel-workflow-runs@ca4b70a6910d33990e16d95e0c116914cdc0dfd0 #v3_1
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
@@ -96,7 +96,7 @@ jobs:
# can cancel all the matching "Build Images" workflow runs in the two following steps.
# Yeah. Adding to the complexity ¯\_(ツ)_/¯.
- uses: potiuk/cancel-workflow-runs@v3
+ uses: potiuk/cancel-workflow-runs@ca4b70a6910d33990e16d95e0c116914cdc0dfd0 #v3_1
id: cancel-failed
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -133,7 +133,7 @@ jobs:
# followed by one of the run ids. Yes I know it's super complex ¯\_(ツ)_/¯.
if: env.BUILD_IMAGES == 'true' && steps.cancel-failed.outputs.cancelledRuns != '[]'
- uses: potiuk/cancel-workflow-runs@v3
+ uses: potiuk/cancel-workflow-runs@ca4b70a6910d33990e16d95e0c116914cdc0dfd0 #v3_1
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
@@ -270,7 +270,7 @@ jobs:
echo "::set-output name=proceed::false"
fi
- name: Initiate Github Checks for Building image
- uses: LouisBrunner/checks-action@v1.1.0
+ uses: LouisBrunner/checks-action@9f02872da71b6f558c6a6f190f925dde5e4d8798 #v1.1.0
id: build-image-check
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -318,7 +318,7 @@ jobs:
run: ./scripts/ci/images/ci_push_production_images.sh
if: matrix.image-type == 'PROD' && steps.defaults.outputs.proceed == 'true'
- name: Update Github Checks for Building image with status
- uses: LouisBrunner/checks-action@v1.1.0
+ uses: LouisBrunner/checks-action@9f02872da71b6f558c6a6f190f925dde5e4d8798 #v1.1.0
if: always() && steps.defaults.outputs.proceed == 'true'
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -333,16 +333,33 @@ jobs:
[Image Build](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})
for details" }
+ cancel-on-build-cancel:
+ name: "Cancel 'CI Build' jobs on build image cancelling."
+ runs-on: ubuntu-latest
+ if: cancelled()
+ needs: [build-images]
+ steps:
+ - name: "Canceling the 'CI Build' source workflow in case of failure!"
+ uses: potiuk/cancel-workflow-runs@cancel_message #v3
+ with:
+ token: ${{ secrets.GITHUB_TOKEN }}
+ cancelMode: self
+ notifyPRCancel: true
+ notifyPRCancelMessage: "Building image for the PR has been cancelled"
+ sourceRunId: ${{ github.event.workflow_run.id }}
+
cancel-on-build-failure:
- name: "Cancel 'CI Build' jobs on build image failure"
+ name: "Cancel 'CI Build' jobs on build image failing."
runs-on: ubuntu-latest
- if: cancelled() || failure()
+ if: failure()
needs: [build-images]
steps:
- name: "Canceling the 'CI Build' source workflow in case of failure!"
- uses: potiuk/cancel-workflow-runs@v3
+ uses: potiuk/cancel-workflow-runs@ca4b70a6910d33990e16d95e0c116914cdc0dfd0 #v3_1
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: self
notifyPRCancel: true
+ notifyPRCancelMessage: |
+ Building images for the PR has failed. Follow the the workflow link to check the reason.
sourceRunId: ${{ github.event.workflow_run.id }}
diff --git a/.github/workflows/codeql-cancel.yml b/.github/workflows/codeql-cancel.yml
index 216fed1..6834898 100644
--- a/.github/workflows/codeql-cancel.yml
+++ b/.github/workflows/codeql-cancel.yml
@@ -12,7 +12,7 @@ jobs:
if: github.repository == 'apache/airflow' || github.event.workflow_run.event != 'schedule'
steps:
- name: "Cancel duplicated 'CodeQL' runs"
- uses: potiuk/cancel-workflow-runs@v3
+ uses: potiuk/cancel-workflow-runs@ca4b70a6910d33990e16d95e0c116914cdc0dfd0 #v3_1
id: cancel
with:
token: ${{ secrets.GITHUB_TOKEN }}
[airflow] 04/14: Bump cache version for kubernetes tests (#11355)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit e620ff85b37c0843d2b850fed0f7efd99833f4b6
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Thu Oct 8 19:10:46 2020 +0200
Bump cache version for kubernetes tests (#11355)
Seems that the k8s cache for virtualenv got broken during the
recent problems. This commits bumps the cache version to make
it afresh
(cherry picked from commit 666e81ab4a468047b9f6869b9eaad6e92b5bc7dd)
---
.github/workflows/ci.yml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index c433d82..85f8e27 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -561,10 +561,10 @@ jobs:
- name: "Cache virtualenv for kubernetes testing"
uses: actions/cache@v2
env:
- cache-name: cache-kubernetes-tests-virtualenv-v3
+ cache-name: cache-kubernetes-tests-virtualenv-v4
with:
path: .build/.kubernetes_venv
- key: "${{ env.cache-name }}-${{ github.job }}-${{ hashFiles('setup.py') }}-v1"
+ key: "${{ env.cache-name }}-${{ github.job }}-${{ hashFiles('setup.py') }}"
- name: "Kubernetes Tests"
run: ./scripts/ci/kubernetes/ci_run_kubernetes_tests.sh
- name: "Upload KinD logs"
[airflow] 12/14: Workarounds "unknown blob" issue by introducing
retries (#11411)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 972a45bac6d83e06a1bf5c517f0700a85a0f4f92
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Sun Oct 11 06:02:46 2020 +0200
Workarounds "unknown blob" issue by introducing retries (#11411)
We have started to experience "unknown_blob" errors intermittently
recently with GitHub Docker registry. We might eventually need
to migrate to GCR (which eventually is going to replace the
Docker Registry for GitHub:
The ticket is opened to the Apache Infrastructure to enable
access to the GCR and to make some statements about Access
Rights management for GCR https://issues.apache.org/jira/projects/INFRA/issues/INFRA-20959
Also a ticket to GitHub Support has been raised about it
https://support.github.com/ticket/personal/0/861667 as we
cannot delete our public images in Docker registry.
But until this happens, the workaround might help us
to handle the situations where we got intermittent errors
while pushing to the registry. This seems to be a common
error, when NGINX proxy is used to proxy Github Registry so
it is likely that retrying will workaround the issue.
(cherry picked from commit f9dddd5d3cdb06bb68c6d3caf2c3d4aba72416ff)
---
scripts/ci/libraries/_push_pull_remove_images.sh | 82 ++++++++++++++++++++----
1 file changed, 69 insertions(+), 13 deletions(-)
diff --git a/scripts/ci/libraries/_push_pull_remove_images.sh b/scripts/ci/libraries/_push_pull_remove_images.sh
index a3e5800..5810303 100644
--- a/scripts/ci/libraries/_push_pull_remove_images.sh
+++ b/scripts/ci/libraries/_push_pull_remove_images.sh
@@ -16,6 +16,34 @@
# specific language governing permissions and limitations
# under the License.
+
+# Tries to push the image several times in case we receive an intermittent error on push
+# $1 - tag to push
+function push_pull_remove_images::push_image_with_retries() {
+ for try_num in 1 2 3 4
+ do
+ set +e
+ echo
+ echo "Trying to push the image ${1}. Number of try: ${try_num}"
+ docker push "${1}"
+ local res=$?
+ set -e
+ if [[ ${res} != "0" ]]; then
+ >&2 echo
+ >&2 echo "Error ${res} when pushing image on ${try_num} try"
+ >&2 echo
+ continue
+ else
+ return 0
+ fi
+ done
+ >&2 echo
+ >&2 echo "Error ${res} when pushing image on ${try_num} try. Giving up!"
+ >&2 echo
+ return 1
+}
+
+
# Pulls image in case it is needed (either has never been pulled or pulling was forced
# Should be run with set +e
# Parameters:
@@ -125,12 +153,12 @@ function push_pull_remove_images::pull_prod_images_if_needed() {
# Pushes Ci images and the manifest to the registry in DockerHub.
function push_pull_remove_images::push_ci_images_to_dockerhub() {
- docker push "${AIRFLOW_CI_IMAGE}"
+ push_pull_remove_images::push_image_with_retries "${AIRFLOW_CI_IMAGE}"
docker tag "${AIRFLOW_CI_LOCAL_MANIFEST_IMAGE}" "${AIRFLOW_CI_REMOTE_MANIFEST_IMAGE}"
- docker push "${AIRFLOW_CI_REMOTE_MANIFEST_IMAGE}"
+ push_pull_remove_images::push_image_with_retries "${AIRFLOW_CI_REMOTE_MANIFEST_IMAGE}"
if [[ -n ${DEFAULT_CI_IMAGE=} ]]; then
# Only push default image to DockerHub registry if it is defined
- docker push "${DEFAULT_CI_IMAGE}"
+ push_pull_remove_images::push_image_with_retries "${DEFAULT_CI_IMAGE}"
fi
}
@@ -142,19 +170,33 @@ function push_pull_remove_images::push_ci_images_to_github() {
# "latest" - in case of push builds
AIRFLOW_CI_TAGGED_IMAGE="${GITHUB_REGISTRY_AIRFLOW_CI_IMAGE}:${GITHUB_REGISTRY_PUSH_IMAGE_TAG}"
docker tag "${AIRFLOW_CI_IMAGE}" "${AIRFLOW_CI_TAGGED_IMAGE}"
- docker push "${AIRFLOW_CI_TAGGED_IMAGE}"
+ push_pull_remove_images::push_image_with_retries "${AIRFLOW_CI_TAGGED_IMAGE}"
if [[ -n ${GITHUB_SHA=} ]]; then
# Also push image to GitHub registry with commit SHA
AIRFLOW_CI_SHA_IMAGE="${GITHUB_REGISTRY_AIRFLOW_CI_IMAGE}:${COMMIT_SHA}"
docker tag "${AIRFLOW_CI_IMAGE}" "${AIRFLOW_CI_SHA_IMAGE}"
- docker push "${AIRFLOW_CI_SHA_IMAGE}"
+ push_pull_remove_images::push_image_with_retries "${AIRFLOW_CI_SHA_IMAGE}"
fi
PYTHON_TAG_SUFFIX=""
if [[ ${GITHUB_REGISTRY_PUSH_IMAGE_TAG} != "latest" ]]; then
PYTHON_TAG_SUFFIX="-${GITHUB_REGISTRY_PUSH_IMAGE_TAG}"
fi
docker tag "${PYTHON_BASE_IMAGE}" "${GITHUB_REGISTRY_PYTHON_BASE_IMAGE}${PYTHON_TAG_SUFFIX}"
- docker push "${GITHUB_REGISTRY_PYTHON_BASE_IMAGE}${PYTHON_TAG_SUFFIX}"
+ set +e
+ push_pull_remove_images::push_image_with_retries "${GITHUB_REGISTRY_PYTHON_BASE_IMAGE}${PYTHON_TAG_SUFFIX}"
+ local result=$?
+ set -e
+ if [[ ${result} != "0" ]]; then
+ >&2 echo
+ >&2 echo "There was an unexpected error when pushing images to the GitHub Registry"
+ >&2 echo
+ >&2 echo "If you see 'unknown blob' or similar kind of error it means that it was a transient error"
+ >&2 echo "And it will likely be gone next time"
+ >&2 echo
+ >&2 echo "Please rebase your change or 'git commit --amend; git push --force' and try again"
+ >&2 echo
+ exit "${result}"
+ fi
}
@@ -170,16 +212,15 @@ function push_pull_remove_images::push_ci_images() {
# Pushes PROD image to registry in DockerHub
function push_pull_remove_images::push_prod_images_to_dockerhub () {
# Prod image
- docker push "${AIRFLOW_PROD_IMAGE}"
+ push_pull_remove_images::push_image_with_retries "${AIRFLOW_PROD_IMAGE}"
if [[ -n ${DEFAULT_PROD_IMAGE=} ]]; then
- docker push "${DEFAULT_PROD_IMAGE}"
+ push_pull_remove_images::push_image_with_retries "${DEFAULT_PROD_IMAGE}"
fi
# Prod build image
- docker push "${AIRFLOW_PROD_BUILD_IMAGE}"
+ push_pull_remove_images::push_image_with_retries "${AIRFLOW_PROD_BUILD_IMAGE}"
}
-
# Pushes PROD image to and their tags to registry in GitHub
function push_pull_remove_images::push_prod_images_to_github () {
# Push image to GitHub registry with chosen push tag
@@ -188,17 +229,32 @@ function push_pull_remove_images::push_prod_images_to_github () {
# "latest" - in case of push builds
AIRFLOW_PROD_TAGGED_IMAGE="${GITHUB_REGISTRY_AIRFLOW_PROD_IMAGE}:${GITHUB_REGISTRY_PUSH_IMAGE_TAG}"
docker tag "${AIRFLOW_PROD_IMAGE}" "${AIRFLOW_PROD_TAGGED_IMAGE}"
- docker push "${GITHUB_REGISTRY_AIRFLOW_PROD_IMAGE}:${GITHUB_REGISTRY_PUSH_IMAGE_TAG}"
+ push_pull_remove_images::push_image_with_retries "${GITHUB_REGISTRY_AIRFLOW_PROD_IMAGE}:${GITHUB_REGISTRY_PUSH_IMAGE_TAG}"
if [[ -n ${COMMIT_SHA=} ]]; then
# Also push image to GitHub registry with commit SHA
AIRFLOW_PROD_SHA_IMAGE="${GITHUB_REGISTRY_AIRFLOW_PROD_IMAGE}:${COMMIT_SHA}"
docker tag "${AIRFLOW_PROD_IMAGE}" "${AIRFLOW_PROD_SHA_IMAGE}"
- docker push "${AIRFLOW_PROD_SHA_IMAGE}"
+ push_pull_remove_images::push_image_with_retries "${AIRFLOW_PROD_SHA_IMAGE}"
fi
# Also push prod build image
AIRFLOW_PROD_BUILD_TAGGED_IMAGE="${GITHUB_REGISTRY_AIRFLOW_PROD_BUILD_IMAGE}:${GITHUB_REGISTRY_PUSH_IMAGE_TAG}"
docker tag "${AIRFLOW_PROD_BUILD_IMAGE}" "${AIRFLOW_PROD_BUILD_TAGGED_IMAGE}"
- docker push "${AIRFLOW_PROD_BUILD_TAGGED_IMAGE}"
+ set +e
+ push_pull_remove_images::push_image_with_retries "${AIRFLOW_PROD_BUILD_TAGGED_IMAGE}"
+ local result=$?
+ set -e
+ if [[ ${result} != "0" ]]; then
+ >&2 echo
+ >&2 echo "There was an unexpected error when pushing images to the GitHub Registry"
+ >&2 echo
+ >&2 echo "If you see 'unknown blob' or similar kind of error it means that it was a transient error"
+ >&2 echo "And it will likely be gone next time"
+ >&2 echo
+ >&2 echo "Please rebase your change or 'git commit --amend; git push --force' and try again"
+ >&2 echo
+ exit "${result}"
+ fi
+
}
[airflow] 09/14: Push and schedule duplicates are not cancelled.
(#11397)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit a7e6f931b322fbd5ba8cf2f2b929739316878fd0
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Sat Oct 10 13:51:58 2020 +0200
Push and schedule duplicates are not cancelled. (#11397)
The push and schedule builds should not be cancelled even if
they are duplicates. By seing which of the master merges
failed, we have better visibility on which merge caused
a problem and we can trace it's origin faster even if the builds
will take longer overall.
Scheduled builds also serve it's purpose and they should
be always run to completion.
(cherry picked from commit 401a579dd16b3c0106f4708bcfe25c8ae635b3ef)
---
.github/workflows/build-images-workflow-run.yml | 53 ++++++++++++++-----------
.github/workflows/codeql-cancel.yml | 3 +-
2 files changed, 32 insertions(+), 24 deletions(-)
diff --git a/.github/workflows/build-images-workflow-run.yml b/.github/workflows/build-images-workflow-run.yml
index a30947d..58804f3 100644
--- a/.github/workflows/build-images-workflow-run.yml
+++ b/.github/workflows/build-images-workflow-run.yml
@@ -47,25 +47,31 @@ jobs:
name: "Cancel workflow runs"
runs-on: ubuntu-latest
outputs:
- sourceHeadRepo: ${{ steps.cancel.outputs.sourceHeadRepo }}
- sourceHeadBranch: ${{ steps.cancel.outputs.sourceHeadBranch }}
- sourceHeadSha: ${{ steps.cancel.outputs.sourceHeadSha }}
- mergeCommitSha: ${{ steps.cancel.outputs.mergeCommitSha }}
- targetCommitSha: ${{ steps.cancel.outputs.targetCommitSha }}
- sourceEvent: ${{ steps.cancel.outputs.sourceEvent }}
+ sourceHeadRepo: ${{ steps.source-run-info.outputs.sourceHeadRepo }}
+ sourceHeadBranch: ${{ steps.source-run-info.outputs.sourceHeadBranch }}
+ sourceHeadSha: ${{ steps.source-run-info.outputs.sourceHeadSha }}
+ mergeCommitSha: ${{ steps.source-run-info.outputs.mergeCommitSha }}
+ targetCommitSha: ${{ steps.source-run-info.outputs.targetCommitSha }}
+ sourceEvent: ${{ steps.source-run-info.outputs.sourceEvent }}
cacheDirective: ${{ steps.cache-directive.outputs.docker-cache }}
buildImages: ${{ steps.build-images.outputs.buildImages }}
upgradeToLatestConstraints: ${{ steps.upgrade-constraints.upgradeToLatestConstraints }}
if: github.repository == 'apache/airflow' || github.event.workflow_run.event != 'schedule'
steps:
+ - name: "Get information about the origin 'CI Build' run"
+ uses: potiuk/get-workflow-origin@c657bb36aef4a7402bbe9b2e09a820320f8ff447 # v1
+ id: source-run-info
+ with:
+ token: ${{ secrets.GITHUB_TOKEN }}
+ sourceRunId: ${{ github.event.workflow_run.id }}
- name: "Cancel duplicated 'CI Build' runs"
- uses: potiuk/cancel-workflow-runs@ca4b70a6910d33990e16d95e0c116914cdc0dfd0 #v3_1
- id: cancel
+ uses: potiuk/cancel-workflow-runs@ca4b70a6910d33990e16d95e0c116914cdc0dfd0 # v3_2
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: duplicates
sourceRunId: ${{ github.event.workflow_run.id }}
notifyPRCancel: true
+ skipEventTypes: '["schedule", "push"]'
- name: "Output BUILD_IMAGES"
id: build-images
run: |
@@ -78,16 +84,19 @@ jobs:
# in GitHub Actions, we have to use Job names to match Event/Repo/Branch from the
# build-info step there to find the duplicates ¯\_(ツ)_/¯.
- uses: potiuk/cancel-workflow-runs@ca4b70a6910d33990e16d95e0c116914cdc0dfd0 #v3_1
+ uses: potiuk/cancel-workflow-runs@ca4b70a6910d33990e16d95e0c116914cdc0dfd0 # v3_2
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
notifyPRCancel: true
+ skipEventTypes: '["schedule", "push"]'
jobNameRegexps: >
- [".*Event: ${{ steps.cancel.outputs.sourceEvent }}
- Repo: ${{ steps.cancel.outputs.sourceHeadRepo }}
- Branch: ${{ steps.cancel.outputs.sourceHeadBranch }}.*"]
- if: env.BUILD_IMAGES == 'true'
+ [".*Event: ${{ steps.source-run-info.outputs.sourceEvent }}
+ Repo: ${{ steps.source-run-info.outputs.sourceHeadRepo }}
+ Branch: ${{ steps.source-run-info.outputs.sourceHeadBranch }}.*"]
+ if: >
+ env.BUILD_IMAGES == 'true' && steps.source-run-info.outputs.sourceEvent != 'schedule'
+ && steps.source-run-info.outputs.sourceEvent != 'push'
- name: "Cancel all 'CI Build' runs where some jobs failed"
# We find any of the "CI Build" workflow runs, where any of the important jobs
@@ -96,13 +105,14 @@ jobs:
# can cancel all the matching "Build Images" workflow runs in the two following steps.
# Yeah. Adding to the complexity ¯\_(ツ)_/¯.
- uses: potiuk/cancel-workflow-runs@ca4b70a6910d33990e16d95e0c116914cdc0dfd0 #v3_1
+ uses: potiuk/cancel-workflow-runs@ca4b70a6910d33990e16d95e0c116914cdc0dfd0 # v3_2
id: cancel-failed
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: failedJobs
sourceRunId: ${{ github.event.workflow_run.id }}
notifyPRCancel: true
+ skipEventTypes: '["schedule", "push"]'
jobNameRegexps: >
["^Static checks.*", "^Build docs$", "^Spell check docs$", "^Backport packages$",
"^Checks: Helm tests$", "^Test OpenAPI*"]
@@ -113,27 +123,24 @@ jobs:
# above - we want to cancel also the corresponding "Build Images" runs. Again we have
# to match the jobs using job name rather than use proper API because that feature
# is currently missing in GitHub Actions ¯\_(ツ)_/¯.
-
id: extract-cancelled-failed-runs
- if: steps.cancel-failed.outputs.cancelledRuns != '[]'
+ if: steps.source-run-info-failed.outputs.cancelledRuns != '[]'
run: |
REGEXP="Source Run id: "
SEPARATOR=""
- for run_id in $(echo "${{ steps.cancel-failed.outputs.cancelledRuns }}" | jq '.[]')
+ for run_id in $(echo "${{ steps.source-run-info-failed.outputs.cancelledRuns }}" | jq '.[]')
do
REGEXP="${REGEXP}${SEPARATOR}(${run_id})"
SEPARATOR="|"
done
echo "::set-output name=matching-regexp::[\"${REGEXP}\"]"
- name: "Cancel triggered 'Build Images' runs for the cancelled failed runs"
-
# In case we do have some cancelled jobs in the "cancel-failed" step above
# We take the extracted regexp array prepared in the previous step and we use
# it to cancel any jobs that have matching names containing Source Run Id:
# followed by one of the run ids. Yes I know it's super complex ¯\_(ツ)_/¯.
-
- if: env.BUILD_IMAGES == 'true' && steps.cancel-failed.outputs.cancelledRuns != '[]'
- uses: potiuk/cancel-workflow-runs@ca4b70a6910d33990e16d95e0c116914cdc0dfd0 #v3_1
+ if: env.BUILD_IMAGES == 'true' && steps.source-run-info-failed.outputs.cancelledRuns != '[]'
+ uses: potiuk/cancel-workflow-runs@ca4b70a6910d33990e16d95e0c116914cdc0dfd0 # v3_2
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
@@ -142,7 +149,7 @@ jobs:
- name: "Set Docker Cache Directive"
id: cache-directive
run: |
- if [[ ${{ steps.cancel.outputs.sourceEvent }} == 'schedule' ]]; then
+ if [[ ${{ steps.source-run-info.outputs.sourceEvent }} == 'schedule' ]]; then
echo "::set-output name=docker-cache::disabled"
else
echo "::set-output name=docker-cache::pulled"
@@ -355,7 +362,7 @@ jobs:
needs: [build-images]
steps:
- name: "Canceling the 'CI Build' source workflow in case of failure!"
- uses: potiuk/cancel-workflow-runs@ca4b70a6910d33990e16d95e0c116914cdc0dfd0 #v3_1
+ uses: potiuk/cancel-workflow-runs@ca4b70a6910d33990e16d95e0c116914cdc0dfd0 # v3_2
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: self
diff --git a/.github/workflows/codeql-cancel.yml b/.github/workflows/codeql-cancel.yml
index 6834898..7dcda4b 100644
--- a/.github/workflows/codeql-cancel.yml
+++ b/.github/workflows/codeql-cancel.yml
@@ -12,10 +12,11 @@ jobs:
if: github.repository == 'apache/airflow' || github.event.workflow_run.event != 'schedule'
steps:
- name: "Cancel duplicated 'CodeQL' runs"
- uses: potiuk/cancel-workflow-runs@ca4b70a6910d33990e16d95e0c116914cdc0dfd0 #v3_1
+ uses: potiuk/cancel-workflow-runs@ca4b70a6910d33990e16d95e0c116914cdc0dfd0 # v3_2
id: cancel
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: duplicates
sourceRunId: ${{ github.event.workflow_run.id }}
notifyPRCancel: true
+ skipEventTypes: '["schedule", "push"]'
[airflow] 05/14: Better diagnostics when there are problems with
Kerberos (#11353)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 3898a5feaff5a396b301301a30da6c6dd197492e
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Thu Oct 8 21:08:11 2020 +0200
Better diagnostics when there are problems with Kerberos (#11353)
(cherry picked from commit f5b7bbcb929d6b7cec6b2b3868fbff2db60797de)
---
scripts/in_container/_in_container_utils.sh | 3 +++
1 file changed, 3 insertions(+)
diff --git a/scripts/in_container/_in_container_utils.sh b/scripts/in_container/_in_container_utils.sh
index dd4b7ce..7ef0cd1 100644
--- a/scripts/in_container/_in_container_utils.sh
+++ b/scripts/in_container/_in_container_utils.sh
@@ -204,6 +204,9 @@ function setup_kerberos() {
RES_3=$?
if [[ ${RES_1} != 0 || ${RES_2} != 0 || ${RES_3} != 0 ]]; then
+ echo
+ echo "Error when setting up Kerberos: ${RES_1} ${RES_2} ${RES_3}}!"
+ echo
exit 1
else
echo
[airflow] 01/14: Breeze start-airflow command wasn't able to
initialize the db in 1.10.x (#11207)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit d2ad37e5ce89029582f3b2cac4a3055703f6d393
Author: mucio <mu...@mucio.net>
AuthorDate: Tue Oct 6 10:40:32 2020 +0200
Breeze start-airflow command wasn't able to initialize the db in 1.10.x (#11207)
(cherry picked from commit 03e0ff24b16c63ba041a47be433aa83cf6bb744a)
---
scripts/in_container/check_environment.sh | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/scripts/in_container/check_environment.sh b/scripts/in_container/check_environment.sh
index 0a040a9..793dfe1 100755
--- a/scripts/in_container/check_environment.sh
+++ b/scripts/in_container/check_environment.sh
@@ -111,10 +111,11 @@ function startairflow_if_requested() {
. "$( dirname "${BASH_SOURCE[0]}" )/configure_environment.sh"
- # initialize db
- airflow initdb
- if [[ ${RBAC_UI} == "true" ]]; then
- # For rbac UI create the admin user if it's a new run
+ # initialize db and create the admin user if it's a new run
+ if [[ ${RUN_AIRFLOW_1_10} == "true" ]]; then
+ airflow initdb
+ airflow create_user -u admin -p admin -f Thor -l Adminstra -r Admin -e dummy@dummy.email || true
+ else
airflow create_user -u admin -p admin -f Thor -l Adminstra -r Admin -e dummy@dummy.email
fi
[airflow] 10/14: Fixes automated upgrade to latest constraints.
(#11399)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit f726c38d2285e61e554a5f5b2a493e7b3a20d301
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Sat Oct 10 15:09:10 2020 +0200
Fixes automated upgrade to latest constraints. (#11399)
Wrong if query in the GitHub action caused upgrade to latest
constraints did not work for a while.
(cherry picked from commit a34f5ee76d536fcb18a7283bf93976358fb3c5c8)
---
.github/workflows/build-images-workflow-run.yml | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/.github/workflows/build-images-workflow-run.yml b/.github/workflows/build-images-workflow-run.yml
index 58804f3..fe95dce 100644
--- a/.github/workflows/build-images-workflow-run.yml
+++ b/.github/workflows/build-images-workflow-run.yml
@@ -55,7 +55,7 @@ jobs:
sourceEvent: ${{ steps.source-run-info.outputs.sourceEvent }}
cacheDirective: ${{ steps.cache-directive.outputs.docker-cache }}
buildImages: ${{ steps.build-images.outputs.buildImages }}
- upgradeToLatestConstraints: ${{ steps.upgrade-constraints.upgradeToLatestConstraints }}
+ upgradeToLatestConstraints: ${{ steps.upgrade-constraints.outputs.upgradeToLatestConstraints }}
if: github.repository == 'apache/airflow' || github.event.workflow_run.event != 'schedule'
steps:
- name: "Get information about the origin 'CI Build' run"
@@ -157,8 +157,8 @@ jobs:
- name: "Set upgrade to latest constraints"
id: upgrade-constraints
run: |
- if [[ ${{ needs.cancel-workflow-runs.outputs.sourceEvent == 'push' ||
- needs.cancel-workflow-runs.outputs.sourceEvent == 'scheduled' }} == 'true' ]]; then
+ if [[ ${{ steps.cancel.outputs.sourceEvent == 'push' ||
+ steps.cancel.outputs.sourceEvent == 'scheduled' }} == 'true' ]]; then
echo "::set-output name=upgradeToLatestConstraints::${{ github.sha }}"
else
echo "::set-output name=upgradeToLatestConstraints::false"
[airflow] 13/14: Add capability of customising PyPI sources (#11385)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit dbc9ab95ae733cced3f285b9472c28cbf5ef3fcf
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Sun Oct 11 06:19:57 2020 +0200
Add capability of customising PyPI sources (#11385)
* Add capability of customising PyPI sources
This change adds capability of customising installation of PyPI
modules via custom .pypirc file. This might allow to install
dependencies from in-house, vetted registry of PyPI
(cherry picked from commit 45d33dbd432fd010f6ff2b698c682c31ac436c24)
---
Dockerfile | 4 ++++
docs/production-deployment.rst | 40 ++++++++++++++++++++++++++++++++++++++++
2 files changed, 44 insertions(+)
diff --git a/Dockerfile b/Dockerfile
index f257606..7cc7f94 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -164,6 +164,8 @@ RUN mkdir -p /root/.local/bin
ARG AIRFLOW_PRE_CACHED_PIP_PACKAGES="true"
ENV AIRFLOW_PRE_CACHED_PIP_PACKAGES=${AIRFLOW_PRE_CACHED_PIP_PACKAGES}
+COPY .pypirc /root/.pypirc
+
# In case of Production build image segment we want to pre-install master version of airflow
# dependencies from github so that we do not have to always reinstall it from the scratch.
RUN if [[ ${AIRFLOW_PRE_CACHED_PIP_PACKAGES} == "true" ]]; then \
@@ -385,6 +387,8 @@ RUN chmod a+x /entrypoint /clean-logs
# See https://github.com/apache/airflow/issues/9248
RUN chmod g=u /etc/passwd
+COPY .pypirc ${AIRFLOW_USER_HOME_DIR}/.pypirc
+
ENV PATH="${AIRFLOW_USER_HOME_DIR}/.local/bin:${PATH}"
ENV GUNICORN_CMD_ARGS="--worker-tmp-dir /dev/shm"
diff --git a/docs/production-deployment.rst b/docs/production-deployment.rst
index 5e6cad2..7c4bfab 100644
--- a/docs/production-deployment.rst
+++ b/docs/production-deployment.rst
@@ -262,6 +262,14 @@ You can combine both - customizing & extending the image. You can build the imag
``customize`` method (either with docker command or with ``breeze`` and then you can ``extend``
the resulting image using ``FROM:`` any dependencies you want.
+Customizing PYPI installation
+.............................
+
+You can customize PYPI sources used during image build by modifying .pypirc file that should be
+placed in the root of Airflow Directory. This .pypirc will never be committed to the repository
+and will not be present in the final production image. It is added and used only in the build
+segment of the image so it is never copied to the final image.
+
External sources for dependencies
---------------------------------
@@ -595,3 +603,35 @@ More details about the images
You can read more details about the images - the context, their parameters and internal structure in the
`IMAGES.rst <https://github.com/apache/airflow/blob/master/IMAGES.rst>`_ document.
+
+.. _production-deployment:kerberos:
+
+Kerberos-authenticated workers
+==============================
+
+Apache Airflow has a built-in mechanism for authenticating the operation with a KDC (Key Distribution Center).
+Airflow has a separate command ``airflow kerberos`` that acts as token refresher. It uses the pre-configured
+Kerberos Keytab to authenticate in the KDC to obtain a valid token, and then refreshing valid token
+at regular intervals within the current token expiry window.
+
+Each request for refresh uses a configured principal, and only keytab valid for the principal specified
+is capable of retrieving the authentication token.
+
+The best practice to implement proper security mechanism in this case is to make sure that worker
+workloads have no access to the Keytab but only have access to the periodically refreshed, temporary
+authentication tokens. This can be achieved in docker environment by running the ``airflow kerberos``
+command and the worker command in separate containers - where only the ``airflow kerberos`` token has
+access to the Keytab file (preferably configured as secret resource). Those two containers should share
+a volume where the temporary token should be written by the ``airflow kerberos`` and read by the workers.
+
+In the Kubernetes environment, this can be realized by the concept of side-car, where both Kerberos
+token refresher and worker are part of the same Pod. Only the Kerberos side-car has access to
+Keytab secret and both containers in the same Pod share the volume, where temporary token is written by
+the side-care container and read by the worker container.
+
+This concept is implemented in the development version of the Helm Chart that is part of Airflow source code.
+
+
+.. spelling::
+
+ pypirc
[airflow] 07/14: Add pypirc initialization (#11386)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 932b565a4e3e6305307b63498f747fc1546778eb
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Fri Oct 9 22:55:03 2020 +0200
Add pypirc initialization (#11386)
This PR needs to be merged first in order to handle the #11385
which requires .pypirc to be created before dockerfile gets build.
This means that the script change needs to be merged to master
first in this PR.
(cherry picked from commit e198077f3e54db1d0846c9583c009f7c0ae75209)
---
.dockerignore | 1 +
.gitignore | 19 ++++++++++++++++++-
scripts/ci/libraries/_initialization.sh | 6 ++++++
3 files changed, 25 insertions(+), 1 deletion(-)
diff --git a/.dockerignore b/.dockerignore
index 7d29561..fb9c80e 100644
--- a/.dockerignore
+++ b/.dockerignore
@@ -54,6 +54,7 @@
!NOTICE
!.github
!empty
+!.pypirc
# Avoid triggering context change on README change (new companies using Airflow)
# So please do not uncomment this line ;)
diff --git a/.gitignore b/.gitignore
index 4f6b451..27b6f0c 100644
--- a/.gitignore
+++ b/.gitignore
@@ -10,7 +10,6 @@ secrets.py
airflow.db
unittests.db
-
# Airflow temporary artifacts
airflow/git_version
airflow/www/static/coverage/
@@ -149,6 +148,8 @@ tramp
# Spark
rat-results.txt
+# Git stuff
+.gitattributes
# Kubernetes generated templated files
*.generated
*.tar.gz
@@ -185,8 +186,24 @@ dmypy.json
/.kube
/.inputrc
log.txt*
+/backport_packages/CHANGELOG.txt
+
+# Local .terraform directories
+**/.terraform/*
+
+# .tfstate files
+*.tfstate
+*.tfstate.*
+
+# Terraform variables
+*.tfvars
Chart.lock
# Chart dependencies
**/charts/*.tgz
+
+# Might be generated when you build wheels
+pip-wheel-metadata
+
+.pypirc
diff --git a/scripts/ci/libraries/_initialization.sh b/scripts/ci/libraries/_initialization.sh
index 4aa72e1..2139aa6 100644
--- a/scripts/ci/libraries/_initialization.sh
+++ b/scripts/ci/libraries/_initialization.sh
@@ -34,6 +34,12 @@ function initialization::create_directories() {
export FILES_DIR="${AIRFLOW_SOURCES}/files"
readonly FILES_DIR
+ # Create an empty .pypirc file that you can customise. It is .gitignored so it will never
+ # land in the repository - it is only added to the "build image" of production image
+ # So you can keep your credentials safe as long as you do not push the build image.
+ # The final image does not contain it.
+ touch "${AIRFLOW_SOURCES}/.pypirc"
+
# Directory where all the build cache is stored - we keep there status of all the docker images
# As well as hashes of the important files, but also we generate build scripts there that are
# Used to execute the commands for breeze
[airflow] 06/14: Use only-if-needed upgrade strategy for PRs
(#11363)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 43e51e48c202d827303553ef8f878e688358bb11
Author: Kaxil Naik <ka...@gmail.com>
AuthorDate: Fri Oct 9 08:57:51 2020 +0100
Use only-if-needed upgrade strategy for PRs (#11363)
Currently, upgrading dependencies in setup.py still runs with previous versions of the package for the PR which fails.
This will change to upgrade only the package that is required for the PRs
(cherry picked from commit 7f674c685d8b95787c52af15e359de7053a8bf67)
---
Dockerfile.ci | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/Dockerfile.ci b/Dockerfile.ci
index 02a9a53..122c5e2 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -294,15 +294,14 @@ ENV UPGRADE_TO_LATEST_CONSTRAINTS=${UPGRADE_TO_LATEST_CONSTRAINTS}
# The goal of this line is to install the dependencies from the most current setup.py from sources
# This will be usually incremental small set of packages in CI optimized build, so it will be very fast
# In non-CI optimized build this will install all dependencies before installing sources.
-# Usually we will install versions constrained to the current constraints file
+# Usually we will install versions based on the dependencies in setup.py and upgraded only if needed.
# But in cron job we will install latest versions matching setup.py to see if there is no breaking change
# and push the constraints if everything is successful
RUN \
if [[ "${UPGRADE_TO_LATEST_CONSTRAINTS}" != "false" ]]; then \
pip install -e ".[${AIRFLOW_EXTRAS}]" --upgrade --upgrade-strategy eager; \
else \
- pip install -e ".[${AIRFLOW_EXTRAS}]" \
- --constraint "${AIRFLOW_CONSTRAINTS_URL}" ; \
+ pip install -e ".[${AIRFLOW_EXTRAS}]" --upgrade --upgrade-strategy only-if-needed; \
fi
# Copy all the www/ files we need to compile assets. Done as two separate COPY
[airflow] 11/14: Fixes cancelling of too many workflows. (#11403)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 73fbd26a3978412e6cdfadb1e676aafd01e2a167
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Sat Oct 10 18:33:06 2020 +0200
Fixes cancelling of too many workflows. (#11403)
A problem was introduced in #11397 where a bit too many "Build Image"
jobs is being cancelled by subsequent Build Image run. For now it
cancels all the Build Image jobs that are running :(.
(cherry picked from commit 076fe88a1dadc9e9cde4674e79c3dc9e1881dffb)
---
.github/workflows/build-images-workflow-run.yml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/.github/workflows/build-images-workflow-run.yml b/.github/workflows/build-images-workflow-run.yml
index fe95dce..ad94cdc 100644
--- a/.github/workflows/build-images-workflow-run.yml
+++ b/.github/workflows/build-images-workflow-run.yml
@@ -124,11 +124,11 @@ jobs:
# to match the jobs using job name rather than use proper API because that feature
# is currently missing in GitHub Actions ¯\_(ツ)_/¯.
id: extract-cancelled-failed-runs
- if: steps.source-run-info-failed.outputs.cancelledRuns != '[]'
+ if: steps.cancel-failed.outputs.cancelledRuns != '[]'
run: |
REGEXP="Source Run id: "
SEPARATOR=""
- for run_id in $(echo "${{ steps.source-run-info-failed.outputs.cancelledRuns }}" | jq '.[]')
+ for run_id in $(echo "${{ steps.cancel-failed.outputs.cancelledRuns }}" | jq '.[]')
do
REGEXP="${REGEXP}${SEPARATOR}(${run_id})"
SEPARATOR="|"