You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2022/06/15 09:42:52 UTC
[airflow] branch main updated: First attempt to have CI-controlled process of releasing PROD image (#24433)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new b79790d8e5 First attempt to have CI-controlled process of releasing PROD image (#24433)
b79790d8e5 is described below
commit b79790d8e5de18782a3179a8a55457eafa529a88
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Wed Jun 15 11:42:41 2022 +0200
First attempt to have CI-controlled process of releasing PROD image (#24433)
---
.github/workflows/release_dockerhub_image.yml | 125 +++++++++++++++++++++
dev/MANUALLY_BUILDING_IMAGES.md | 96 ++++++++++++++++
dev/README_RELEASE_AIRFLOW.md | 109 ++++--------------
.../commands/release_management_commands.py | 7 +-
dev/images/release_prod_image.png | Bin 0 -> 112569 bytes
5 files changed, 245 insertions(+), 92 deletions(-)
diff --git a/.github/workflows/release_dockerhub_image.yml b/.github/workflows/release_dockerhub_image.yml
new file mode 100644
index 0000000000..bd9fdaec1e
--- /dev/null
+++ b/.github/workflows/release_dockerhub_image.yml
@@ -0,0 +1,125 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: "Release PROD image"
+on: # yamllint disable-line rule:truthy
+ workflow_dispatch:
+ inputs:
+ airflowVersion:
+ description: 'Airflow version'
+ required: true
+ skipLatest:
+ description: 'Skip Latest: Set to true if not latest.'
+ default: ''
+ required: false
+jobs:
+ build-info:
+ timeout-minutes: 10
+ name: "Build Info"
+ runs-on: ${{ github.repository == 'apache/airflow' && 'self-hosted' || 'ubuntu-20.04' }}
+ outputs:
+ pythonVersions: ${{ steps.selective-checks.outputs.python-versions }}
+ allPythonVersions: ${{ steps.selective-checks.outputs.all-python-versions }}
+ defaultPythonVersion: ${{ steps.selective-checks.outputs.default-python-version }}
+ skipLatest: ${{ github.event.inputs.skipLatest == '' && ' ' || '--skip-latest' }}
+ limitPlatform: ${{ github.repository == 'apache/airflow' && ' ' || '--limit-platform linux/amd64' }}
+ env:
+ GITHUB_CONTEXT: ${{ toJson(github) }}
+ steps:
+ - name: Cleanup repo
+ run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
+ uses: actions/checkout@v2
+ with:
+ persist-credentials: false
+ submodules: recursive
+ - name: Selective checks
+ id: selective-checks
+ run: ./scripts/ci/selective_ci_checks.sh
+ release-images:
+ timeout-minutes: 120
+ name: "Release images"
+ runs-on: ${{ github.repository == 'apache/airflow' && 'self-hosted' || 'ubuntu-20.04' }}
+ needs: [build-info]
+ strategy:
+ fail-fast: false
+ matrix:
+ python-version: ${{ fromJson(needs.build-info.outputs.pythonVersions) }}
+ env:
+ RUNS_ON: ${{ github.repository == 'apache/airflow' && 'self-hosted' || 'ubuntu-20.04' }}
+ if: contains(fromJSON('[
+ "ashb",
+ "ephraimbuddy",
+ "jedcunningham",
+ "kaxil",
+ "potiuk",
+ ]'), github.event.sender.login)
+ steps:
+ - name: Cleanup repo
+ run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
+ uses: actions/checkout@v2
+ with:
+ persist-credentials: false
+ - name: "Setup python"
+ uses: actions/setup-python@v2
+ with:
+ python-version: ${{ needs.build-info.outputs.defaultPythonVersion }}
+ cache: 'pip'
+ cache-dependency-path: ./dev/breeze/setup*
+ - run: ./scripts/ci/install_breeze.sh
+ - name: "Free space"
+ run: breeze free-space
+ - name: Build CI image for PROD build ${{ needs.build-info.outputs.defaultPythonVersion }}
+ run: breeze build-image
+ env:
+ PYTHON_MAJOR_MINOR_VERSION: ${{ needs.build-info.outputs.defaultPythonVersion }}
+ - name: "Cleanup dist and context file"
+ run: rm -fv ./dist/* ./docker-context-files/*
+ - name: "Start ARM instance"
+ run: ./scripts/ci/images/ci_start_arm_instance_and_connect_to_docker.sh
+ if: github.repository == 'apache/airflow'
+ - name: "Login to docker"
+ run: >
+ echo ${{ secrets.DOCKERHUB_TOKEN }} |
+ docker login --password-stdin --username ${{ secrets.DOCKERHUB_USER }}
+ - name: "Release regular images"
+ run: >
+ breeze release-prod-images
+ --dockerhub-repo ${{ github.repository }}
+ --airflow-version ${{ github.event.inputs.airflowVersion }}
+ ${{ needs.build-info.outputs.skipLatest }}
+ ${{ needs.build-info.outputs.limitPlatform }}
+ --limit-python ${{ matrix.python-version }}
+ - name: "Release slim images"
+ run: >
+ breeze release-prod-images
+ --dockerhub-repo ${{ github.repository }}
+ --airflow-version ${{ github.event.inputs.airflowVersion }}
+ ${{ needs.build-info.outputs.skipLatest }}
+ ${{ needs.build-info.outputs.limitPlatform }}
+ --limit-python ${{ matrix.python-version }} --slim-images
+ - name: "Docker logout"
+ run: docker logout
+ if: always()
+ - name: "Stop ARM instance"
+ run: ./scripts/ci/images/ci_stop_arm_instance.sh
+ if: always() && github.repository == 'apache/airflow'
+ - name: "Fix ownership"
+ run: breeze fix-ownership
+ if: always()
diff --git a/dev/MANUALLY_BUILDING_IMAGES.md b/dev/MANUALLY_BUILDING_IMAGES.md
new file mode 100644
index 0000000000..99cf589d69
--- /dev/null
+++ b/dev/MANUALLY_BUILDING_IMAGES.md
@@ -0,0 +1,96 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+
+- [Building docker images](#building-docker-images)
+- [Setting environment with emulation](#setting-environment-with-emulation)
+- [Setting up cache refreshing with hardware ARM/AMD support](#setting-up-cache-refreshing-with-hardware-armamd-support)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+## Building docker images
+
+In order to build images on local hardware, you need to have the buildx plugin installed to run the build.
+Also, you need to have regctl installed from https://github.com/regclient/regclient in order to tag
+the multi-platform images in DockerHub. The script to build images will refuse to work if
+you do not have those two installed.
+
+You also need to have the right permissions to push the images, so you should run
+`docker login` before and authenticate with your DockerHub token.
+
+## Setting environment with emulation
+
+According to the [official installation instructions](https://docs.docker.com/buildx/working-with-buildx/#build-multi-platform-images)
+this can be achieved via:
+
+```shell
+docker run --privileged --rm tonistiigi/binfmt --install all
+```
+
+More information can be found [here](https://docs.docker.com/engine/reference/commandline/buildx_create/).
+
+However, emulation is very slow - more than 10x slower than hardware-backed builds.
+
+## Setting up cache refreshing with hardware ARM/AMD support
+
+If you plan to build a number of images, it's probably better to set up a hardware remote builder
+for your ARM or AMD builds (depending which platform you build images on - the "other" platform should be
+remote).
+
+This can be achieved by settings build as described in
+[this blog post](https://www.docker.com/blog/speed-up-building-with-docker-buildx-and-graviton2-ec2/) and
+adding it to docker buildx `airflow_cache` builder.
+
+This usually can be done with those two commands:
+
+```bash
+docker buildx create --name airflow_cache # your local builder
+docker buildx create --name airflow_cache --append HOST:PORT # your remote builder
+```
+
+One of the ways to have HOST:PORT is to login to the remote machine via SSH and forward the port to
+the docker engine running on the remote machine.
+
+When everything is fine you should see both local and remote builder configured and reporting status:
+
+```bash
+docker buildx ls
+
+ airflow_cache docker-container
+ airflow_cache0 unix:///var/run/docker.sock
+ airflow_cache1 tcp://127.0.0.1:2375
+```
+
+Preparing regular images:
+
+```shell script
+breeze release-prod-images --airflow-version "${VERSION}"
+```
+
+Preparing slim images:
+
+```shell script
+breeze release-prod-images --airflow-version "${VERSION}" --slim-images
+```
+
+This will wipe Breeze cache and docker-context-files in order to make sure the build is "clean". It
+also performs image verification after pushing the images.
diff --git a/dev/README_RELEASE_AIRFLOW.md b/dev/README_RELEASE_AIRFLOW.md
index 7e6c16289e..5a785ff7df 100644
--- a/dev/README_RELEASE_AIRFLOW.md
+++ b/dev/README_RELEASE_AIRFLOW.md
@@ -28,9 +28,6 @@
- [[\Optional\] Prepare new release branches and cache](#%5Coptional%5C-prepare-new-release-branches-and-cache)
- [Prepare PyPI convenience "snapshot" packages](#prepare-pypi-convenience-snapshot-packages)
- [Prepare production Docker Image](#prepare-production-docker-image)
- - [Prerequisites](#prerequisites)
- - [Setting environment with emulation](#setting-environment-with-emulation)
- - [Setting up cache refreshing with hardware ARM/AMD support](#setting-up-cache-refreshing-with-hardware-armamd-support)
- [Prepare issue for testing status of rc](#prepare-issue-for-testing-status-of-rc)
- [Prepare Vote email on the Apache Airflow release candidate](#prepare-vote-email-on-the-apache-airflow-release-candidate)
- [Verify the release candidate by PMCs](#verify-the-release-candidate-by-pmcs)
@@ -494,76 +491,23 @@ is not supposed to be used by and advertised to the end-users who do not read th
Production Docker images should be manually prepared and pushed by the release manager or another committer
who has access to Airflow's DockerHub. Note that we started releasing a multi-platform build, so you need
-to have an environment prepared to build multi-platform images. You can achieve it with either emulation
-(very slow) or if you have two types of hardware (AMD64 and ARM64) you can configure Hardware builders.
+to have an environment prepared to build multi-platform images. You can achieve it with:
-## Prerequisites
+* GitHub Actions Manual Job (easiest)
+* Emulation (very slow)
+* Hardware builders if you have both AMD64 and ARM64 hardware locally
-You need to have buildx plugin installed to run the build. Also, you need to have regctl
-installed from https://github.com/regclient/regclient in order to tag the multi-platform images in
-DockerHub. The script to build images will refuse to work if you do not have those two installed.
+Building the image is triggered by running the `Release PROD image` workflow via
+[GitHub Actions](https://github.com/apache/airflow/actions).
-You also need to have the right permissions to push the images, so you should run
-`docker login` before and authenticate with your DockerHub token.
+When you trigger it you need to pass:
-## Setting environment with emulation
+* Airflow Version
+* Optional "true" in skip latest field if you do not want to retag the latest image
-According to the [official installation instructions](https://docs.docker.com/buildx/working-with-buildx/#build-multi-platform-images)
-this can be achieved via:
-
-```shell
-docker run --privileged --rm tonistiigi/binfmt --install all
-```
-
-More information can be found [here](https://docs.docker.com/engine/reference/commandline/buildx_create/)
-
-However, emulation is very slow - more than 10x slower than hardware-backed builds.
-
-## Setting up cache refreshing with hardware ARM/AMD support
-
-If you plan to build a number of images, probably better solution is to set up a hardware remote builder
-for your ARM or AMD builds (depending which platform you build images on - the "other" platform should be
-remote.
-
-This can be achieved by settings build as described in
-[this guideline](https://www.docker.com/blog/speed-up-building-with-docker-buildx-and-graviton2-ec2/) and
-adding it to docker buildx `airflow_cache` builder.
-
-This usually can be done with those two commands:
-
-```bash
-docker buildx create --name airflow_cache # your local builder
-docker buildx create --name airflow_cache --append HOST:PORT # your remote builder
-```
-
-One of the ways to have HOST:PORT is to login to the remote machine via SSH and forward the port to
-the docker engine running on the remote machine.
-
-When everything is fine you should see both local and remote builder configured and reporting status:
-
-```bash
-docker buildx ls
-
- airflow_cache docker-container
- airflow_cache0 unix:///var/run/docker.sock
- airflow_cache1 tcp://127.0.0.1:2375
-```
-
-Preparing regular images:
-
-```shell script
-breeze release-prod-images --airflow-version "${VERSION}"
-```
-
-Preparing slim images:
-
-```shell script
-breeze release-prod-images --airflow-version "${VERSION}" --slim-images
-```
-
-This will wipe Breeze cache and docker-context-files in order to make sure the build is "clean". It
-also performs image verification after pushing the images.
+
+The manual building is described in [MANUALLY_BUILDING_IMAGES.md](MANUALLY_BUILDING_IMAGES.md).
## Prepare issue for testing status of rc
@@ -1014,33 +958,22 @@ At this point we release an official package:
## Manually prepare production Docker Image
-Note that this scripts prepares multi-platform image, so you need to fulfill prerequisites as
-described above in the preparation of RC images.
+Building the image is triggered by running the `Release PROD image` workflow via
+[GitHub Actions](https://github.com/apache/airflow/actions).
+
+When you trigger it you need to pass:
+
+* Airflow Version
+* Optional "true" in skip latest field if you do not want to retag the latest image
+
+
Note that by default the `latest` images tagged are aliased to the just released image which is the usual
way we release. For example when you are releasing 2.3.N image and 2.3 is our latest branch the new image is
marked as "latest".
In case we are releasing (which almost never happens so far) a critical bugfix release in one of
-the older branches, you should add the `--skip-latest` flag.
-
-Preparing regular images:
-
-```shell script
-breeze release-prod-images --airflow-version "${VERSION}"
-```
-
-Preparing slim images:
-
-```shell script
-breeze release-prod-images --airflow-version "${VERSION}" --slim-images
-```
-
-Preparing a release that is not in the latest branch:
-
-```shell script
-breeze release-prod-images --airflow-version "${VERSION}" --slim-images --skip-latest
-```
+the older branches, you should set the "skip" field to true.
## Verify production images
diff --git a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
index ce80e6a193..839b966fae 100644
--- a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
@@ -650,12 +650,11 @@ def release_prod_images(
["docker", 'buildx', 'inspect', 'airflow_cache'], check=False, dry_run=dry_run, verbose=verbose
)
if result_inspect_builder.returncode != 0:
- get_console().print("[error]Regctl must be installed and on PATH to release the images[/]")
+ get_console().print("[error]Airflow Cache builder must be configured to release the images[/]")
get_console().print()
get_console().print(
- "See https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md"
- "#setting-up-cache-refreshing-with-hardware-armamd-support for "
- "instructions on setting it up."
+ "See https://github.com/apache/airflow/blob/main/dev/MANUALLY_BUILDING_IMAGES.md"
+ " for instructions on setting it up."
)
sys.exit(1)
result_regctl = run_command(["regctl", 'version'], check=False, dry_run=dry_run, verbose=verbose)
diff --git a/dev/images/release_prod_image.png b/dev/images/release_prod_image.png
new file mode 100644
index 0000000000..78f941a43b
Binary files /dev/null and b/dev/images/release_prod_image.png differ