You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jedcunningham (via GitHub)" <gi...@apache.org> on 2023/08/10 21:51:55 UTC
[GitHub] [airflow] jedcunningham commented on a diff in pull request #33144: Add update-constraints command that allows to modify released ones
jedcunningham commented on code in PR #33144:
URL: https://github.com/apache/airflow/pull/33144#discussion_r1290671429
##########
dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md:
##########
@@ -0,0 +1,334 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+
+- [Purpose of the document](#purpose-of-the-document)
+- [Automated image cache and constraints refreshing in CI](#automated-image-cache-and-constraints-refreshing-in-ci)
+- [Manually refreshing the image cache](#manually-refreshing-the-image-cache)
+ - [Why we need to update image cache manually](#why-we-need-to-update-image-cache-manually)
+ - [Prerequisites](#prerequisites)
+ - [How to refresh the image cache](#how-to-refresh-the-image-cache)
+ - [Is it safe to refresh the image cache?](#is-it-safe-to-refresh-the-image-cache)
+ - [What the command does](#what-the-command-does)
+- [Manually generating constraint files](#manually-generating-constraint-files)
+ - [Why we need to generate constraint files manually](#why-we-need-to-generate-constraint-files-manually)
+ - [How to generate constraint files](#how-to-generate-constraint-files)
+ - [Is it safe to generate constraints manually?](#is-it-safe-to-generate-constraints-manually)
+- [Manually updating already tagged constraint files](#manually-updating-already-tagged-constraint-files)
+ - [Why we need to update constraint files manually (very rarely)](#why-we-need-to-update-constraint-files-manually-very-rarely)
+ - [How to update the constraints](#how-to-update-the-constraints)
+ - [Is it safe to update constraints manually?](#is-it-safe-to-update-constraints-manually)
+ - [How the command works under-the-hood ?](#how-the-command-works-under-the-hood-)
+ - [Examples of running the command](#examples-of-running-the-command)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+# Purpose of the document
+
+This document contains explanation of a few manual procedures we might use at certain times, to update
+our CI and constraints manually when the automation of our CI is not enough. There are some edge cases
+and events that might trigger the need of refreshing the information stored in our GitHub Repository.
+
+We are storing two things we are storing in our GitHub Registry that are needed for both - our contributors
+and users:
+
+* `CI and PROD image cache` - used by our CI jobs to speed up building of images while CI jobs are running
+* `Constraints files` - used by both, CI jobs (to fix the versions of dependencies used by CI jobs in regular
+ PRs) and used by our users to reproducibly install released airflow versions.
+
+Normally, both are updated and refreshed automatically via [CI system](../CI.rst). However, there are some
+cases where we need to update them manually. This document describes how to do it.
+
+# Automated image cache and constraints refreshing in CI
+
+Our [CI system](../CI.rst) is build in the way that it self-maintains. Regular scheduled builds and
+merges to `main` branch builds (also known as `canary` builds) have separate maintenance step that
+take care about refreshing the cache that is used to speed up our builds and to speed up
+rebuilding of [Breeze](../BREEZE.rst) images for development purpose. This is all happening automatically, usually:
+
+* The latest [constraints](../CONTRIBUTING.rst#pinned-constraint-files) are pushed to appropriate branch after all tests succeed in the
Review Comment:
```suggestion
* The latest [constraints](../CONTRIBUTING.rst#pinned-constraint-files) are pushed to appropriate branch after all tests succeed in the
```
##########
dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md:
##########
@@ -0,0 +1,334 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+
+- [Purpose of the document](#purpose-of-the-document)
+- [Automated image cache and constraints refreshing in CI](#automated-image-cache-and-constraints-refreshing-in-ci)
+- [Manually refreshing the image cache](#manually-refreshing-the-image-cache)
+ - [Why we need to update image cache manually](#why-we-need-to-update-image-cache-manually)
+ - [Prerequisites](#prerequisites)
+ - [How to refresh the image cache](#how-to-refresh-the-image-cache)
+ - [Is it safe to refresh the image cache?](#is-it-safe-to-refresh-the-image-cache)
+ - [What the command does](#what-the-command-does)
+- [Manually generating constraint files](#manually-generating-constraint-files)
+ - [Why we need to generate constraint files manually](#why-we-need-to-generate-constraint-files-manually)
+ - [How to generate constraint files](#how-to-generate-constraint-files)
+ - [Is it safe to generate constraints manually?](#is-it-safe-to-generate-constraints-manually)
+- [Manually updating already tagged constraint files](#manually-updating-already-tagged-constraint-files)
+ - [Why we need to update constraint files manually (very rarely)](#why-we-need-to-update-constraint-files-manually-very-rarely)
+ - [How to update the constraints](#how-to-update-the-constraints)
+ - [Is it safe to update constraints manually?](#is-it-safe-to-update-constraints-manually)
+ - [How the command works under-the-hood ?](#how-the-command-works-under-the-hood-)
+ - [Examples of running the command](#examples-of-running-the-command)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+# Purpose of the document
+
+This document contains explanation of a few manual procedures we might use at certain times, to update
+our CI and constraints manually when the automation of our CI is not enough. There are some edge cases
+and events that might trigger the need of refreshing the information stored in our GitHub Repository.
+
+We are storing two things we are storing in our GitHub Registry that are needed for both - our contributors
Review Comment:
```suggestion
We are storing two things in our GitHub Registry that are needed for both our contributors
```
##########
dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md:
##########
@@ -0,0 +1,334 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+
+- [Purpose of the document](#purpose-of-the-document)
+- [Automated image cache and constraints refreshing in CI](#automated-image-cache-and-constraints-refreshing-in-ci)
+- [Manually refreshing the image cache](#manually-refreshing-the-image-cache)
+ - [Why we need to update image cache manually](#why-we-need-to-update-image-cache-manually)
+ - [Prerequisites](#prerequisites)
+ - [How to refresh the image cache](#how-to-refresh-the-image-cache)
+ - [Is it safe to refresh the image cache?](#is-it-safe-to-refresh-the-image-cache)
+ - [What the command does](#what-the-command-does)
+- [Manually generating constraint files](#manually-generating-constraint-files)
+ - [Why we need to generate constraint files manually](#why-we-need-to-generate-constraint-files-manually)
+ - [How to generate constraint files](#how-to-generate-constraint-files)
+ - [Is it safe to generate constraints manually?](#is-it-safe-to-generate-constraints-manually)
+- [Manually updating already tagged constraint files](#manually-updating-already-tagged-constraint-files)
+ - [Why we need to update constraint files manually (very rarely)](#why-we-need-to-update-constraint-files-manually-very-rarely)
+ - [How to update the constraints](#how-to-update-the-constraints)
+ - [Is it safe to update constraints manually?](#is-it-safe-to-update-constraints-manually)
+ - [How the command works under-the-hood ?](#how-the-command-works-under-the-hood-)
+ - [Examples of running the command](#examples-of-running-the-command)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+# Purpose of the document
+
+This document contains explanation of a few manual procedures we might use at certain times, to update
+our CI and constraints manually when the automation of our CI is not enough. There are some edge cases
+and events that might trigger the need of refreshing the information stored in our GitHub Repository.
+
+We are storing two things we are storing in our GitHub Registry that are needed for both - our contributors
+and users:
+
+* `CI and PROD image cache` - used by our CI jobs to speed up building of images while CI jobs are running
+* `Constraints files` - used by both, CI jobs (to fix the versions of dependencies used by CI jobs in regular
+ PRs) and used by our users to reproducibly install released airflow versions.
+
+Normally, both are updated and refreshed automatically via [CI system](../CI.rst). However, there are some
+cases where we need to update them manually. This document describes how to do it.
+
+# Automated image cache and constraints refreshing in CI
+
+Our [CI system](../CI.rst) is build in the way that it self-maintains. Regular scheduled builds and
+merges to `main` branch builds (also known as `canary` builds) have separate maintenance step that
+take care about refreshing the cache that is used to speed up our builds and to speed up
+rebuilding of [Breeze](../BREEZE.rst) images for development purpose. This is all happening automatically, usually:
+
+* The latest [constraints](../CONTRIBUTING.rst#pinned-constraint-files) are pushed to appropriate branch after all tests succeed in the
+ `canary` build.
+
+* The [images](../IMAGES.rst) in `ghcr.io` registry are refreshed early at the beginning of the `canary` build. This
+ is done twice during the canary build:
+ * By the `Push Early Image Cache` job that is run at the beginning of the `canary` build. This cover the
+ case when there are new dependencies added or Dockerfile/scripts change. Thanks to that step, subsequent
+ PRs will be faster when they use the new Dockerfile/script. Those jobs **might fail** occasionally,
+ if the latest PR added some conflicting dependencies with current constraints. This is not a problem
+ and when it happens, it will be fixed by the next step.
+ * By the `Push Image Cache` job that is run at the end of the `canary` build. This covers the case when
+ cache is also refreshed after than `main` build succeeds after the new constraints are pushed. This
+ step makes sure that constraints are committed and pushed just before the cache is refreshed, so
+ there is no problem with conflicting dependencies.
+
+# Manually refreshing the image cache
+
+## Why we need to update image cache manually
+
+Sometimes, when we have a problem with our CI running and flakiness of GitHub Actions runners or our
+tests, the refresh might not be triggered. This has been mitigated by "Push Early Image Cache" job added in
+our CI, but there are other reasons you might want to refresh the cache. Sometimes we want to refresh the
+image cache in `vX_Y_test` branch (following our convention of branch names `vX_Y_test` branch is the branch
+used to release all `X.Y.*` versions of airflow) before we attempt to push a change there.
+There are no PRs happening in this branch, so manual refresh before we make a PR might speed up the PR build.
+Or sometimes we just refreshed the constraints (see below) and we want the cache to include those.
+
+## Prerequisites
+
+Note that in order to refresh images you have to not only have `buildx` command installed for docker,
+but you should also make sure that you have the buildkit builder configured and set. Since we also build
+multi-platform images (for both AMD and ARM), you need to have support for qemu or hardware ARM/AMD builders
+configured. The chapters below explain both options.
+
+### Setting up cache refreshing with emulation
+
+According to the [official installation instructions](https://docs.docker.com/buildx/working-with-buildx/#build-multi-platform-images)
+this can be achieved via:
+
+```bash
+docker run --privileged --rm tonistiigi/binfmt --install all
+```
+
+More information can be found [here](https://docs.docker.com/engine/reference/commandline/buildx_create/)
+
+However, emulation is very slow - more than 10x slower than hardware-backed builds.
+
+### Setting up cache refreshing with hardware ARM/AMD support
+
+If you plan to build a number of images, probably better solution is to set up a hardware remote builder
+for your ARM or AMD builds (depending which platform you build images on - the "other" platform should be
+remote.
+
+This can be achieved by settings build as described in
+[this guideline](https://www.docker.com/blog/speed-up-building-with-docker-buildx-and-graviton2-ec2/) and
+adding it to docker buildx `airflow_cache` builder.
+
+This usually can be done with those two commands:
+
+```bash
+docker buildx create --name airflow_cache # your local builder
+docker buildx create --name airflow_cache --append HOST:PORT # your remote builder
+```
+
+One of the ways to have HOST:PORT is to login to the remote machine via SSH and forward the port to
+the docker engine running on the remote machine.
+
+When everything is fine you should see both local and remote builder configured and reporting status:
+
+```bash
+docker buildx ls
+
+ airflow_cache docker-container
+ airflow_cache0 unix:///var/run/docker.sock
+ airflow_cache1 tcp://127.0.0.1:2375
+```
+
+## How to refresh the image cache
+
+The images can be rebuilt and refreshed after the constraints are pushed. Refreshing image for all
+python version is a simple as running the [refresh_images.sh](refresh_images.sh) script which will
+rebuild all the images in parallel and push them to the registry.
+
+Note that you need to run `docker login ghcr.io` before you run the script and you need to be
+a committer in order to be able to push the cache to the registry.
+
+```bash
+./dev/refresh_images.sh
+```
+
+## Is it safe to refresh the image cache?
+
+Yes. Image cache is only used to speed up the build process in CI. The worst thing that can happen if
+the image cache is broken is that the PR builds of our will run slower - usually, for regular PRs building
+the images from scratch takes about 15 minutes. With the image cache it takes about 1 minute if there are no
+dependency changes. So if the image cache is broken, the worst thing that will happen is that the PR builds
+will run longer "Wait for CI Image" step and "Wait for PROD image" will simply wait a bit longer.
+
+Eventually the cache will heal itself. When the `main` build succeeds with all the tests, the cache is
+automatically updated. Actually it's even faster in new CI process of ours, the cache is refreshed
+very quickly after there is a merge of a new PR to the main ("Push Early Image Cache" jobs), so
+cache refreshing and self-healing should be generally rather quick.
+
+## What the command does
+
+The command does the following:
+
+* builds the CI image using the builders configured using buildx and pushes the cache
+ to the `apache/airflow` registry (`--prepare-buildx-cache` flag). It builds all images in parallel for
+ both AMD and ARM architectures.
+* prepares packages and airflow packages in `dist` folder using the latest sources
+* moves the packages to the `docker-context-files` folder so that they are available when building the
+ PROD images
+* builds the PROD image using the builders configured and packages prepared using buildx and pushes the cache
+ to the `apache/airflow` registry (`--prepare-buildx-cache` flag). It builds all images in parallel for
+ both AMD and ARM architectures.
+
+# Manually generating constraint files
+
+## Why we need to generate constraint files manually
+
+Sometimes we want to generate constraint files if - for whatever reason - we cannot or do not want to wait
+until `main` or `vY_Z_test` branch tests succeed. The constraints are only refreshed by CI when all the tests
+pass, and this is a good thing, however there are some cases where we cannot solve some intermittent problem
+with tests, but we KNOW that the tip of the branch is good and we want to release a new airflow version or
+we want to move the PRs of contributors to start using the new constraints. This should be done with caution
+and you need to be sure what you are doing, but you can always do it manually if you want.
+
+## How to generate constraint files
+
+```bash
+breeze ci-image build --run-in-parallel --upgrade-to-newer-dependencies --answer yes
+breeze release-management generate-constraints --airflow-constraints-mode constraints --run-in-parallel --answer yes
+breeze release-management generate-constraints --airflow-constraints-mode constraints-source-providers --run-in-parallel --answer yes
+breeze release-management generate-constraints --airflow-constraints-mode constraints-no-providers --run-in-parallel --answer yes
+
+AIRFLOW_SOURCES=$(pwd)
+```
+
+The constraints will be generated in `files/constraints-PYTHON_VERSION/constraints-*.txt` files. You need to
+check out the right 'constraints-' branch in a separate repository, and then you can copy, commit and push the
+generated files.
+
+You need to be a committer, and you have to be authenticated in the apache/airflow repository for your
+git commands to be able to push the new constraints
+
+```bash
+cd <AIRFLOW_WITH_CONSTRAINTS-MAIN_DIRECTORY>
+git pull
+cp ${AIRFLOW_SOURCES}/files/constraints-*/constraints*.txt .
+git diff
+git add .
+git commit -m "Your commit message here" --no-verify
+git push
+```
+
+## Is it safe to generate constraints manually?
+
+The slight risk is that if there is a constraint problem that impacts regular PRs and tests then it might
+make all PRs "red" until the constraint is fixed. However, if this is the case then usually we should fix
+the problem by fixing the tests or dependencies and the automated CI process should be able to self-heal.
+The main build does not use constraints and it will attempt to upgrade (or downgradea) the dependencies to
Review Comment:
```suggestion
The main build does not use constraints and it will attempt to upgrade (or downgrade) the dependencies to
```
##########
dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md:
##########
@@ -0,0 +1,334 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+
+- [Purpose of the document](#purpose-of-the-document)
+- [Automated image cache and constraints refreshing in CI](#automated-image-cache-and-constraints-refreshing-in-ci)
+- [Manually refreshing the image cache](#manually-refreshing-the-image-cache)
+ - [Why we need to update image cache manually](#why-we-need-to-update-image-cache-manually)
+ - [Prerequisites](#prerequisites)
+ - [How to refresh the image cache](#how-to-refresh-the-image-cache)
+ - [Is it safe to refresh the image cache?](#is-it-safe-to-refresh-the-image-cache)
+ - [What the command does](#what-the-command-does)
+- [Manually generating constraint files](#manually-generating-constraint-files)
+ - [Why we need to generate constraint files manually](#why-we-need-to-generate-constraint-files-manually)
+ - [How to generate constraint files](#how-to-generate-constraint-files)
+ - [Is it safe to generate constraints manually?](#is-it-safe-to-generate-constraints-manually)
+- [Manually updating already tagged constraint files](#manually-updating-already-tagged-constraint-files)
+ - [Why we need to update constraint files manually (very rarely)](#why-we-need-to-update-constraint-files-manually-very-rarely)
+ - [How to update the constraints](#how-to-update-the-constraints)
+ - [Is it safe to update constraints manually?](#is-it-safe-to-update-constraints-manually)
+ - [How the command works under-the-hood ?](#how-the-command-works-under-the-hood-)
+ - [Examples of running the command](#examples-of-running-the-command)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+# Purpose of the document
+
+This document contains explanation of a few manual procedures we might use at certain times, to update
+our CI and constraints manually when the automation of our CI is not enough. There are some edge cases
+and events that might trigger the need of refreshing the information stored in our GitHub Repository.
+
+We are storing two things we are storing in our GitHub Registry that are needed for both - our contributors
+and users:
+
+* `CI and PROD image cache` - used by our CI jobs to speed up building of images while CI jobs are running
+* `Constraints files` - used by both, CI jobs (to fix the versions of dependencies used by CI jobs in regular
+ PRs) and used by our users to reproducibly install released airflow versions.
+
+Normally, both are updated and refreshed automatically via [CI system](../CI.rst). However, there are some
+cases where we need to update them manually. This document describes how to do it.
+
+# Automated image cache and constraints refreshing in CI
+
+Our [CI system](../CI.rst) is build in the way that it self-maintains. Regular scheduled builds and
+merges to `main` branch builds (also known as `canary` builds) have separate maintenance step that
+take care about refreshing the cache that is used to speed up our builds and to speed up
+rebuilding of [Breeze](../BREEZE.rst) images for development purpose. This is all happening automatically, usually:
+
+* The latest [constraints](../CONTRIBUTING.rst#pinned-constraint-files) are pushed to appropriate branch after all tests succeed in the
+ `canary` build.
+
+* The [images](../IMAGES.rst) in `ghcr.io` registry are refreshed early at the beginning of the `canary` build. This
+ is done twice during the canary build:
+ * By the `Push Early Image Cache` job that is run at the beginning of the `canary` build. This cover the
+ case when there are new dependencies added or Dockerfile/scripts change. Thanks to that step, subsequent
+ PRs will be faster when they use the new Dockerfile/script. Those jobs **might fail** occasionally,
+ if the latest PR added some conflicting dependencies with current constraints. This is not a problem
+ and when it happens, it will be fixed by the next step.
+ * By the `Push Image Cache` job that is run at the end of the `canary` build. This covers the case when
+ cache is also refreshed after than `main` build succeeds after the new constraints are pushed. This
+ step makes sure that constraints are committed and pushed just before the cache is refreshed, so
+ there is no problem with conflicting dependencies.
+
+# Manually refreshing the image cache
+
+## Why we need to update image cache manually
+
+Sometimes, when we have a problem with our CI running and flakiness of GitHub Actions runners or our
+tests, the refresh might not be triggered. This has been mitigated by "Push Early Image Cache" job added in
+our CI, but there are other reasons you might want to refresh the cache. Sometimes we want to refresh the
+image cache in `vX_Y_test` branch (following our convention of branch names `vX_Y_test` branch is the branch
+used to release all `X.Y.*` versions of airflow) before we attempt to push a change there.
+There are no PRs happening in this branch, so manual refresh before we make a PR might speed up the PR build.
+Or sometimes we just refreshed the constraints (see below) and we want the cache to include those.
+
+## Prerequisites
+
+Note that in order to refresh images you have to not only have `buildx` command installed for docker,
+but you should also make sure that you have the buildkit builder configured and set. Since we also build
+multi-platform images (for both AMD and ARM), you need to have support for qemu or hardware ARM/AMD builders
+configured. The chapters below explain both options.
+
+### Setting up cache refreshing with emulation
+
+According to the [official installation instructions](https://docs.docker.com/buildx/working-with-buildx/#build-multi-platform-images)
+this can be achieved via:
+
+```bash
+docker run --privileged --rm tonistiigi/binfmt --install all
+```
+
+More information can be found [here](https://docs.docker.com/engine/reference/commandline/buildx_create/)
+
+However, emulation is very slow - more than 10x slower than hardware-backed builds.
+
+### Setting up cache refreshing with hardware ARM/AMD support
+
+If you plan to build a number of images, probably better solution is to set up a hardware remote builder
+for your ARM or AMD builds (depending which platform you build images on - the "other" platform should be
+remote.
+
+This can be achieved by settings build as described in
+[this guideline](https://www.docker.com/blog/speed-up-building-with-docker-buildx-and-graviton2-ec2/) and
+adding it to docker buildx `airflow_cache` builder.
+
+This usually can be done with those two commands:
+
+```bash
+docker buildx create --name airflow_cache # your local builder
+docker buildx create --name airflow_cache --append HOST:PORT # your remote builder
+```
+
+One of the ways to have HOST:PORT is to login to the remote machine via SSH and forward the port to
+the docker engine running on the remote machine.
+
+When everything is fine you should see both local and remote builder configured and reporting status:
+
+```bash
+docker buildx ls
+
+ airflow_cache docker-container
+ airflow_cache0 unix:///var/run/docker.sock
+ airflow_cache1 tcp://127.0.0.1:2375
+```
+
+## How to refresh the image cache
+
+The images can be rebuilt and refreshed after the constraints are pushed. Refreshing image for all
+python version is a simple as running the [refresh_images.sh](refresh_images.sh) script which will
+rebuild all the images in parallel and push them to the registry.
+
+Note that you need to run `docker login ghcr.io` before you run the script and you need to be
+a committer in order to be able to push the cache to the registry.
+
+```bash
+./dev/refresh_images.sh
+```
+
+## Is it safe to refresh the image cache?
+
+Yes. Image cache is only used to speed up the build process in CI. The worst thing that can happen if
+the image cache is broken is that the PR builds of our will run slower - usually, for regular PRs building
+the images from scratch takes about 15 minutes. With the image cache it takes about 1 minute if there are no
+dependency changes. So if the image cache is broken, the worst thing that will happen is that the PR builds
+will run longer "Wait for CI Image" step and "Wait for PROD image" will simply wait a bit longer.
+
+Eventually the cache will heal itself. When the `main` build succeeds with all the tests, the cache is
+automatically updated. Actually it's even faster in new CI process of ours, the cache is refreshed
+very quickly after there is a merge of a new PR to the main ("Push Early Image Cache" jobs), so
+cache refreshing and self-healing should be generally rather quick.
+
+## What the command does
+
+The command does the following:
+
+* builds the CI image using the builders configured using buildx and pushes the cache
+ to the `apache/airflow` registry (`--prepare-buildx-cache` flag). It builds all images in parallel for
+ both AMD and ARM architectures.
+* prepares packages and airflow packages in `dist` folder using the latest sources
+* moves the packages to the `docker-context-files` folder so that they are available when building the
+ PROD images
+* builds the PROD image using the builders configured and packages prepared using buildx and pushes the cache
+ to the `apache/airflow` registry (`--prepare-buildx-cache` flag). It builds all images in parallel for
+ both AMD and ARM architectures.
+
+# Manually generating constraint files
+
+## Why we need to generate constraint files manually
+
+Sometimes we want to generate constraint files if - for whatever reason - we cannot or do not want to wait
+until `main` or `vY_Z_test` branch tests succeed. The constraints are only refreshed by CI when all the tests
+pass, and this is a good thing, however there are some cases where we cannot solve some intermittent problem
+with tests, but we KNOW that the tip of the branch is good and we want to release a new airflow version or
+we want to move the PRs of contributors to start using the new constraints. This should be done with caution
+and you need to be sure what you are doing, but you can always do it manually if you want.
+
+## How to generate constraint files
+
+```bash
+breeze ci-image build --run-in-parallel --upgrade-to-newer-dependencies --answer yes
+breeze release-management generate-constraints --airflow-constraints-mode constraints --run-in-parallel --answer yes
+breeze release-management generate-constraints --airflow-constraints-mode constraints-source-providers --run-in-parallel --answer yes
+breeze release-management generate-constraints --airflow-constraints-mode constraints-no-providers --run-in-parallel --answer yes
+
+AIRFLOW_SOURCES=$(pwd)
+```
+
+The constraints will be generated in `files/constraints-PYTHON_VERSION/constraints-*.txt` files. You need to
+check out the right 'constraints-' branch in a separate repository, and then you can copy, commit and push the
+generated files.
+
+You need to be a committer, and you have to be authenticated in the apache/airflow repository for your
+git commands to be able to push the new constraints
+
+```bash
+cd <AIRFLOW_WITH_CONSTRAINTS-MAIN_DIRECTORY>
+git pull
+cp ${AIRFLOW_SOURCES}/files/constraints-*/constraints*.txt .
+git diff
+git add .
+git commit -m "Your commit message here" --no-verify
+git push
+```
+
+## Is it safe to generate constraints manually?
+
+The slight risk is that if there is a constraint problem that impacts regular PRs and tests then it might
+make all PRs "red" until the constraint is fixed. However, if this is the case then usually we should fix
+the problem by fixing the tests or dependencies and the automated CI process should be able to self-heal.
+The main build does not use constraints and it will attempt to upgrade (or downgradea) the dependencies to
+the latest version matching the dependency specification we have in setup.cfg/setup.py/provider.yaml files.
+Also the constraints are pushed without `--force` so there is no risk of destroying anything.
+The history is kept in Git, so you can always revert to the previous version if needed.
+
+# Manually updating already tagged constraint files
+
+## Why we need to update constraint files manually (very rarely)
+
+Sometimes - very rarely - we need to fix historical constraint files when Airflow fails to install with the
+constraints that were used in the past. This happened already several times and usually only happens when
+there is a backwards-incompatible change in the build environment in Python installation toolchain
+(pip, setuptools, wheel, Cython etc.). The Python build environment is not controllable by us - by default
+pip uses `build isolation` which means that it will install the latest version of the build tools. Those
+tools versions are chosen by `pip` separately for each package. However. this might mean that new versions of
Review Comment:
```suggestion
tools versions are chosen by `pip` separately for each package. However, this might mean that new versions of
```
##########
dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md:
##########
@@ -0,0 +1,334 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+
+- [Purpose of the document](#purpose-of-the-document)
+- [Automated image cache and constraints refreshing in CI](#automated-image-cache-and-constraints-refreshing-in-ci)
+- [Manually refreshing the image cache](#manually-refreshing-the-image-cache)
+ - [Why we need to update image cache manually](#why-we-need-to-update-image-cache-manually)
+ - [Prerequisites](#prerequisites)
+ - [How to refresh the image cache](#how-to-refresh-the-image-cache)
+ - [Is it safe to refresh the image cache?](#is-it-safe-to-refresh-the-image-cache)
+ - [What the command does](#what-the-command-does)
+- [Manually generating constraint files](#manually-generating-constraint-files)
+ - [Why we need to generate constraint files manually](#why-we-need-to-generate-constraint-files-manually)
+ - [How to generate constraint files](#how-to-generate-constraint-files)
+ - [Is it safe to generate constraints manually?](#is-it-safe-to-generate-constraints-manually)
+- [Manually updating already tagged constraint files](#manually-updating-already-tagged-constraint-files)
+ - [Why we need to update constraint files manually (very rarely)](#why-we-need-to-update-constraint-files-manually-very-rarely)
+ - [How to update the constraints](#how-to-update-the-constraints)
+ - [Is it safe to update constraints manually?](#is-it-safe-to-update-constraints-manually)
+ - [How the command works under-the-hood ?](#how-the-command-works-under-the-hood-)
+ - [Examples of running the command](#examples-of-running-the-command)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+# Purpose of the document
+
+This document contains explanation of a few manual procedures we might use at certain times, to update
+our CI and constraints manually when the automation of our CI is not enough. There are some edge cases
+and events that might trigger the need of refreshing the information stored in our GitHub Repository.
+
+We are storing two things we are storing in our GitHub Registry that are needed for both - our contributors
+and users:
+
+* `CI and PROD image cache` - used by our CI jobs to speed up building of images while CI jobs are running
+* `Constraints files` - used by both, CI jobs (to fix the versions of dependencies used by CI jobs in regular
+ PRs) and used by our users to reproducibly install released airflow versions.
+
+Normally, both are updated and refreshed automatically via [CI system](../CI.rst). However, there are some
+cases where we need to update them manually. This document describes how to do it.
+
+# Automated image cache and constraints refreshing in CI
+
+Our [CI system](../CI.rst) is build in the way that it self-maintains. Regular scheduled builds and
+merges to `main` branch builds (also known as `canary` builds) have separate maintenance step that
+take care about refreshing the cache that is used to speed up our builds and to speed up
+rebuilding of [Breeze](../BREEZE.rst) images for development purpose. This is all happening automatically, usually:
+
+* The latest [constraints](../CONTRIBUTING.rst#pinned-constraint-files) are pushed to appropriate branch after all tests succeed in the
+ `canary` build.
+
+* The [images](../IMAGES.rst) in `ghcr.io` registry are refreshed early at the beginning of the `canary` build. This
+ is done twice during the canary build:
+ * By the `Push Early Image Cache` job that is run at the beginning of the `canary` build. This cover the
+ case when there are new dependencies added or Dockerfile/scripts change. Thanks to that step, subsequent
+ PRs will be faster when they use the new Dockerfile/script. Those jobs **might fail** occasionally,
+ if the latest PR added some conflicting dependencies with current constraints. This is not a problem
+ and when it happens, it will be fixed by the next step.
+ * By the `Push Image Cache` job that is run at the end of the `canary` build. This covers the case when
+ cache is also refreshed after than `main` build succeeds after the new constraints are pushed. This
+ step makes sure that constraints are committed and pushed just before the cache is refreshed, so
+ there is no problem with conflicting dependencies.
+
+# Manually refreshing the image cache
+
+## Why we need to update image cache manually
+
+Sometimes, when we have a problem with our CI running and flakiness of GitHub Actions runners or our
+tests, the refresh might not be triggered. This has been mitigated by "Push Early Image Cache" job added in
+our CI, but there are other reasons you might want to refresh the cache. Sometimes we want to refresh the
+image cache in `vX_Y_test` branch (following our convention of branch names `vX_Y_test` branch is the branch
+used to release all `X.Y.*` versions of airflow) before we attempt to push a change there.
+There are no PRs happening in this branch, so manual refresh before we make a PR might speed up the PR build.
+Or sometimes we just refreshed the constraints (see below) and we want the cache to include those.
+
+## Prerequisites
+
+Note that in order to refresh images you have to not only have `buildx` command installed for docker,
+but you should also make sure that you have the buildkit builder configured and set. Since we also build
+multi-platform images (for both AMD and ARM), you need to have support for qemu or hardware ARM/AMD builders
+configured. The chapters below explain both options.
+
+### Setting up cache refreshing with emulation
+
+According to the [official installation instructions](https://docs.docker.com/buildx/working-with-buildx/#build-multi-platform-images)
+this can be achieved via:
+
+```bash
+docker run --privileged --rm tonistiigi/binfmt --install all
+```
+
+More information can be found [here](https://docs.docker.com/engine/reference/commandline/buildx_create/)
+
+However, emulation is very slow - more than 10x slower than hardware-backed builds.
+
+### Setting up cache refreshing with hardware ARM/AMD support
+
+If you plan to build a number of images, probably better solution is to set up a hardware remote builder
+for your ARM or AMD builds (depending which platform you build images on - the "other" platform should be
+remote.
+
+This can be achieved by settings build as described in
+[this guideline](https://www.docker.com/blog/speed-up-building-with-docker-buildx-and-graviton2-ec2/) and
+adding it to docker buildx `airflow_cache` builder.
+
+This usually can be done with those two commands:
+
+```bash
+docker buildx create --name airflow_cache # your local builder
+docker buildx create --name airflow_cache --append HOST:PORT # your remote builder
+```
+
+One of the ways to have HOST:PORT is to login to the remote machine via SSH and forward the port to
+the docker engine running on the remote machine.
+
+When everything is fine you should see both local and remote builder configured and reporting status:
+
+```bash
+docker buildx ls
+
+ airflow_cache docker-container
+ airflow_cache0 unix:///var/run/docker.sock
+ airflow_cache1 tcp://127.0.0.1:2375
+```
+
+## How to refresh the image cache
+
+The images can be rebuilt and refreshed after the constraints are pushed. Refreshing image for all
+python version is a simple as running the [refresh_images.sh](refresh_images.sh) script which will
Review Comment:
```suggestion
python version is as simple as running the [refresh_images.sh](refresh_images.sh) script which will
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org