You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2021/06/28 12:35:31 UTC
[airflow] branch main updated: Add preparation of images as part of
RC preparation process (#16674)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 98c12d4 Add preparation of images as part of RC preparation process (#16674)
98c12d4 is described below
commit 98c12d49f37f6879e3e9fd926853f57a15ab761b
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Mon Jun 28 14:35:12 2021 +0200
Add preparation of images as part of RC preparation process (#16674)
---
CI.rst | 16 ----------------
dev/README_RELEASE_AIRFLOW.md | 20 +++++++++++++++++---
scripts/ci/images/ci_build_dockerhub.sh | 5 ++++-
scripts/ci/libraries/_build_images.sh | 4 ++--
4 files changed, 23 insertions(+), 22 deletions(-)
diff --git a/CI.rst b/CI.rst
index af5e8aa..226d745 100644
--- a/CI.rst
+++ b/CI.rst
@@ -573,22 +573,6 @@ Those runs and their corresponding ``Build Images`` runs are only executed in ma
repository, they are not executed in forks - we want to be nice to the contributors and not use their
free build minutes on GitHub Actions.
-Sometimes (bugs in DockerHub or prolonged periods when the scheduled builds are failing)
-the automated build for nightly main is not executed for a long time. Such builds can be manually
-prepared and pushed by a maintainer who has the rights to push images to DockerHub (committers need
-to file JIRA ticket to Apache Infra in order to get an access).
-
-.. code-block:: bash
-
- export BRANCH=main
- export DOCKER_REPO=docker.io/apache/airflow
- for python_version in "3.6" "3.7" "3.8"
- (
- export DOCKER_TAG=${BRANCH}-python${python_version}
- ./scripts/ci/images/ci_build_dockerhub.sh
- )
-
-
Workflows
=========
diff --git a/dev/README_RELEASE_AIRFLOW.md b/dev/README_RELEASE_AIRFLOW.md
index c0f62cc..43779fc 100644
--- a/dev/README_RELEASE_AIRFLOW.md
+++ b/dev/README_RELEASE_AIRFLOW.md
@@ -22,6 +22,7 @@
- [Prepare the Apache Airflow Package RC](#prepare-the-apache-airflow-package-rc)
- [Build RC artifacts](#build-rc-artifacts)
+ - [Manually prepare production Docker Image](#manually-prepare-production-docker-image)
- [[\Optional\] Create new release branch](#%5Coptional%5C-create-new-release-branch)
- [Prepare PyPI convenience "snapshot" packages](#prepare-pypi-convenience-snapshot-packages)
- [Prepare production Docker Image](#prepare-production-docker-image)
@@ -37,7 +38,7 @@
- [Publish release to SVN](#publish-release-to-svn)
- [Prepare PyPI "release" packages](#prepare-pypi-release-packages)
- [Update CHANGELOG.md](#update-changelogmd)
- - [Manually prepare production Docker Image](#manually-prepare-production-docker-image)
+ - [Manually prepare production Docker Image](#manually-prepare-production-docker-image-1)
- [Publish documentation](#publish-documentation)
- [Notify developers of release](#notify-developers-of-release)
- [Update Announcements page](#update-announcements-page)
@@ -56,8 +57,9 @@ The Release Candidate artifacts we vote upon should be the exact ones we vote ag
```shell script
# Set Version
- export VERSION=2.0.2rc3
+ export VERSION=2.1.2rc3
export VERSION_SUFFIX=rc3
+ export VERSION_CONSTRAINT_BRANCH=2-1
export VERSION_WITHOUT_RC=${VERSION/rc?/}
# Set AIRFLOW_REPO_ROOT to the path of your git repo
@@ -105,7 +107,7 @@ The Release Candidate artifacts we vote upon should be the exact ones we vote ag
- Tag & Push the latest constraints files. This pushes constraints with rc suffix (this is expected)!
```shell script
- git checkout constraints-2-0
+ git checkout constraints-${VERSION_CONSTRAINT_BRANCH}
git tag -s "constraints-${VERSION}"
git push origin "constraints-${VERSION}"
```
@@ -127,6 +129,18 @@ The Release Candidate artifacts we vote upon should be the exact ones we vote ag
svn commit -m "Add artifacts for Airflow ${VERSION}"
```
+
+## Manually prepare production Docker Image
+
+
+```shell script
+./scripts/ci/tools/prepare_prod_docker_images.sh ${VERSION}
+```
+
+This will wipe Breeze cache and docker-context-files in order to make sure the build is "clean". It
+also performs image verification before pushing the images.
+
+
## [\Optional\] Create new release branch
When you just released the `X.Y.0` version (first release of new minor version) you need to create release
diff --git a/scripts/ci/images/ci_build_dockerhub.sh b/scripts/ci/images/ci_build_dockerhub.sh
index 495295a..b464cf2 100755
--- a/scripts/ci/images/ci_build_dockerhub.sh
+++ b/scripts/ci/images/ci_build_dockerhub.sh
@@ -44,13 +44,16 @@ rm -rf "${AIRFLOW_SOURCES}/docker-context-files/*"
build_images::prepare_prod_build
build_images::build_prod_images
verify_image::verify_prod_image "${AIRFLOW_PROD_IMAGE}"
+echo
+echo "Pushing airflow image as apache/airflow:${INSTALL_AIRFLOW_VERSION}-python${PYTHON_MAJOR_MINOR_VERSION}"
+echo
# Re-tag the image to be published in "apache/airflow"
docker tag "apache/airflow-ci:${INSTALL_AIRFLOW_VERSION}-python${PYTHON_MAJOR_MINOR_VERSION}" \
"apache/airflow:${INSTALL_AIRFLOW_VERSION}-python${PYTHON_MAJOR_MINOR_VERSION}"
docker push "apache/airflow:${INSTALL_AIRFLOW_VERSION}-python${PYTHON_MAJOR_MINOR_VERSION}"
if [[ ${PYTHON_MAJOR_MINOR_VERSION} == "${DEFAULT_PYTHON_MAJOR_MINOR_VERSION}" ]]; then
echo
- echo "Pushing default airflow image"
+ echo "Pushing default airflow image as apache/airflow:${INSTALL_AIRFLOW_VERSION}"
echo
# In case of default Python version we also push ":version" tag
docker tag "apache/airflow:${INSTALL_AIRFLOW_VERSION}-python${PYTHON_MAJOR_MINOR_VERSION}" \
diff --git a/scripts/ci/libraries/_build_images.sh b/scripts/ci/libraries/_build_images.sh
index 598d944..e10f0b6 100644
--- a/scripts/ci/libraries/_build_images.sh
+++ b/scripts/ci/libraries/_build_images.sh
@@ -743,8 +743,8 @@ function build_images::prepare_prod_build() {
build_images::add_build_args_for_remote_install
elif [[ -n "${INSTALL_AIRFLOW_VERSION=}" ]]; then
# When --install-airflow-version is used then the image is build using released PIP package
- # For PROD image only numeric versions are allowed
- if [[ ! ${INSTALL_AIRFLOW_VERSION} =~ ^[0-9\.]*$ ]]; then
+ # For PROD image only numeric versions are allowed and RC candidates
+ if [[ ! ${INSTALL_AIRFLOW_VERSION} =~ ^[0-9\.]+(rc[0-9]+)?$ ]]; then
echo
echo "${COLOR_RED}ERROR: Bad value for install-airflow-version: '${INSTALL_AIRFLOW_VERSION}'. Only numerical versions allowed for PROD image here'!${COLOR_RESET}"
echo