You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2023/11/26 01:06:06 UTC
(airflow) branch optimize-image-wait-verify updated (5db4097cab -> af556b4b19)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch optimize-image-wait-verify
in repository https://gitbox.apache.org/repos/asf/airflow.git
omit 5db4097cab Optimize CI/PROD image waiting and verification in CI workflow
new af556b4b19 Optimize CI/PROD image waiting and verification in CI workflow
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (5db4097cab)
\
N -- N -- N refs/heads/optimize-image-wait-verify (af556b4b19)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
CI_DIAGRAMS.md | 1 +
1 file changed, 1 insertion(+)
(airflow) 01/01: Optimize CI/PROD image waiting and verification in CI workflow
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch optimize-image-wait-verify
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit af556b4b199b832374c2bffc76b32cece8fc1683
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sat Nov 25 14:05:23 2023 +0100
Optimize CI/PROD image waiting and verification in CI workflow
Currently both "wait-for-ci-images" and "preview-constraints"
jobs are waiting for images to be built - which means that they
both take a running worker slot (public runner) just to do
the waiting while the image is being built. Also "verify-image"
job is run as part of "wait-for-image" which adds additional
delay between being downloaded and dependent jobs starting.
This PR optimizes it quite a bit:
* preview-constraints job now depends on "wait-for-ci-images".
This means that only one slot will be busy while waiting
for images.
* both CI and PROD `verify-image` commands in breeze got
--run-in-parallel set of flags that allow the verification
to happen for all images in parallel.
* Image verification is added as separate step in jobs that
already need to pull the images to do other stuff. For
CI Image it's "Preview constraints" and for PROD image
it is "Test Docker compose job". The fact that they are
not run as part of "wait for image" jobs allows us to
start the other jobs faster but also to not let failure
in image verification block other tests from running.
* In case of the "in-workflow-build" the "wait-for-ci-images"
does not have to be run at all, because there wait-for-ci-images
depends on in-workflow build-ci-images job - so if that job
completes, we know image is built already and we do not have to
wait for it separately - so far we had to run it in order to
add `--verify` flag to verify the images. With separate job we
can run i in parallel to all the other waiting jobs.
* Also names and dependencies between jobs are updated, including
CI documentation describing diagrams of how CI workflows work.
The diagrams are cleaned-up/verified and updated. The separate
diagram for scheduled build has been removed as it was essentially
the same as "canary build". A paragraph description for every
type of workflow was added to add more context to the diagrams.
---
.github/workflows/ci.yml | 338 ++++++++---------
CI.rst | 87 ++---
CI_DIAGRAMS.md | 402 +++++++++++----------
.../airflow_breeze/commands/ci_image_commands.py | 123 ++++++-
.../commands/ci_image_commands_config.py | 11 +
.../commands/production_image_commands.py | 131 +++++--
.../commands/production_image_commands_config.py | 11 +
dev/breeze/src/airflow_breeze/utils/run_tests.py | 2 +-
docker_tests/test_ci_image.py | 34 +-
images/breeze/output_ci-image_verify.svg | 68 +++-
images/breeze/output_ci-image_verify.txt | 2 +-
images/breeze/output_prod-image_verify.svg | 68 +++-
images/breeze/output_prod-image_verify.txt | 2 +-
13 files changed, 791 insertions(+), 488 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 11256ab1de..4a99031e4a 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -44,7 +44,7 @@ env:
CONSTRAINTS_GITHUB_REPOSITORY: >-
${{ secrets.CONSTRAINTS_GITHUB_REPOSITORY != '' &&
secrets.CONSTRAINTS_GITHUB_REPOSITORY || 'apache/airflow' }}
- # In builds from forks, this token is read-only. For scheduler/direct push it is WRITE one
+ # In builds from forks, this token is read-only. For scheduled/direct push it is WRITE one
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
IMAGE_TAG: "${{ github.event.pull_request.head.sha || github.sha }}"
USE_SUDO: "true"
@@ -320,148 +320,12 @@ jobs:
PYTHON_VERSIONS: ${{needs.build-info.outputs.all-python-versions-list-as-string}}
DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
BUILD_TIMEOUT_MINUTES: 70
-
- generate-constraints:
- permissions:
- contents: read
- timeout-minutes: 70
- name: >
- Preview constraints
- ${{needs.build-info.outputs.all-python-versions-list-as-string}}
- runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
- needs: [build-info, build-ci-images]
- env:
- RUNS_ON: "${{ needs.build-info.outputs.runs-on }}"
- PYTHON_VERSIONS: ${{needs.build-info.outputs.all-python-versions-list-as-string}}
- DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
- if: needs.build-info.outputs.ci-image-build == 'true'
- steps:
- - name: Cleanup repo
- run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
- - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
- uses: actions/checkout@v4
- with:
- persist-credentials: false
- - name: "Install Breeze"
- uses: ./.github/actions/breeze
- - name: Pull CI images ${{ env.PYTHON_VERSIONS }}:${{ env.IMAGE_TAG }}
- run: breeze ci-image pull --run-in-parallel --tag-as-latest --wait-for-image
- - name: "Source constraints"
- shell: bash
- run: >
- breeze release-management generate-constraints --run-in-parallel
- --airflow-constraints-mode constraints-source-providers
- - name: "No providers constraints"
- shell: bash
- timeout-minutes: 25
- run: >
- breeze release-management generate-constraints --run-in-parallel
- --airflow-constraints-mode constraints-no-providers
- - name: "PyPI constraints"
- shell: bash
- timeout-minutes: 25
- run: >
- breeze release-management generate-constraints --run-in-parallel
- --airflow-constraints-mode constraints
- - name: "Dependency upgrade summary"
- shell: bash
- run: |
- for PYTHON_VERSION in ${{ env.PYTHON_VERSIONS }}; do
- echo "Summarizing Python $PYTHON_VERSION"
- cat "files/constraints-${PYTHON_VERSION}"/*.md >> $GITHUB_STEP_SUMMARY || true
- done
- - name: "Upload constraint artifacts"
- uses: actions/upload-artifact@v3
- with:
- name: constraints
- path: ./files/constraints-*/constraints-*.txt
- retention-days: 7
-
- build-prod-images:
- timeout-minutes: 80
- name: >
- ${{needs.build-info.outputs.build-job-description}} PROD images
- ${{needs.build-info.outputs.all-python-versions-list-as-string}}
- runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
- needs: [build-info, build-ci-images]
- env:
- DEFAULT_BRANCH: ${{ needs.build-info.outputs.default-branch }}
- DEFAULT_CONSTRAINTS_BRANCH: ${{ needs.build-info.outputs.default-constraints-branch }}
- RUNS_ON: "${{needs.build-info.outputs.runs-on}}"
- BACKEND: sqlite
- DOCKER_CACHE: ${{ needs.build-info.outputs.cache-directive }}
- VERSION_SUFFIX_FOR_PYPI: "dev0"
- DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
- # Force more parallelism for build even on public images
- PARALLELISM: 6
- steps:
- - name: Cleanup repo
- run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
- if: needs.build-info.outputs.in-workflow-build == 'true'
- - uses: actions/checkout@v4
- with:
- ref: ${{ needs.build-info.outputs.targetCommitSha }}
- persist-credentials: false
- if: needs.build-info.outputs.in-workflow-build == 'true'
- - name: "Install Breeze"
- uses: ./.github/actions/breeze
- if: needs.build-info.outputs.in-workflow-build == 'true'
- - name: >
- Build PROD Images
- ${{needs.build-info.outputs.all-python-versions-list-as-string}}:${{env.IMAGE_TAG}}
- uses: ./.github/actions/build-prod-images
- if: needs.build-info.outputs.in-workflow-build == 'true'
- with:
- build-provider-packages: ${{ needs.build-info.outputs.default-branch == 'main' }}
+ - name: Verify CI images ${{ env.PYTHON_VERSIONS }}:${{ env.IMAGE_TAG }}
+ run: breeze ci-image verify --run-in-parallel
env:
- UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgrade-to-newer-dependencies }}
- DOCKER_CACHE: ${{ needs.build-info.outputs.cache-directive }}
PYTHON_VERSIONS: ${{needs.build-info.outputs.all-python-versions-list-as-string}}
- DEBUG_RESOURCES: ${{ needs.build-info.outputs.debug-resources }}
-
- build-prod-images-bullseye:
- timeout-minutes: 80
- name: >
- ${{needs.build-info.outputs.build-job-description}} Bullseye PROD images
- ${{needs.build-info.outputs.all-python-versions-list-as-string}}
- runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
- needs: [build-info, build-ci-images]
- if: needs.build-info.outputs.canary-run == 'true'
- env:
- DEFAULT_BRANCH: ${{ needs.build-info.outputs.default-branch }}
- DEFAULT_CONSTRAINTS_BRANCH: ${{ needs.build-info.outputs.default-constraints-branch }}
- RUNS_ON: "${{needs.build-info.outputs.runs-on}}"
- BACKEND: sqlite
- DOCKER_CACHE: ${{ needs.build-info.outputs.cache-directive }}
- VERSION_SUFFIX_FOR_PYPI: "dev0"
- DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
- # Force more parallelism for build even on public images
- PARALLELISM: 6
- steps:
- - name: Cleanup repo
- run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
- - uses: actions/checkout@v3
- with:
- ref: ${{ needs.build-info.outputs.targetCommitSha }}
- persist-credentials: false
- submodules: recursive
- - name: "Install Breeze"
- uses: ./.github/actions/breeze
- - name: >
- Build Bullseye PROD Images
- ${{needs.build-info.outputs.all-python-versions-list-as-string}}:${{env.IMAGE_TAG}}
- uses: ./.github/actions/build-prod-images
- with:
- build-provider-packages: ${{ needs.build-info.outputs.default-branch == 'main' }}
- env:
- UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgrade-to-newer-dependencies }}
- DOCKER_CACHE: ${{ needs.build-info.outputs.cache-directive }}
- PYTHON_VERSIONS: ${{needs.build-info.outputs.all-python-versions-list-as-string}}
- DEBUG_RESOURCES: ${{ needs.build-info.outputs.debug-resources }}
- DEBIAN_VERSION: "bullseye"
- # Do not override the "bookworm" image - just push a new bullseye image
- # TODO: improve caching for that build
- IMAGE_TAG: "bullseye-${{ github.event.pull_request.head.sha || github.sha }}"
+ DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
+ if: needs.build-info.outputs.in-workflow-build == 'true'
run-breeze-tests:
timeout-minutes: 10
@@ -565,6 +429,7 @@ jobs:
with:
fetch-depth: 2
persist-credentials: false
+
wait-for-ci-images:
timeout-minutes: 120
name: "Wait for CI images"
@@ -579,18 +444,84 @@ jobs:
steps:
- name: Cleanup repo
run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
+ if: needs.build-info.outputs.in-workflow-build == 'false'
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v4
with:
persist-credentials: false
+ if: needs.build-info.outputs.in-workflow-build == 'false'
- name: "Install Breeze"
uses: ./.github/actions/breeze
+ if: needs.build-info.outputs.in-workflow-build == 'false'
- name: Wait for CI images ${{ env.PYTHON_VERSIONS }}:${{ env.IMAGE_TAG }}
id: wait-for-images
- run: breeze ci-image pull --run-in-parallel --verify --wait-for-image --tag-as-latest
+ run: breeze ci-image pull --run-in-parallel --wait-for-image --tag-as-latest
env:
PYTHON_VERSIONS: ${{ needs.build-info.outputs.python-versions-list-as-string }}
DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
+ if: needs.build-info.outputs.in-workflow-build == 'false'
+
+ generate-constraints:
+ permissions:
+ contents: read
+ timeout-minutes: 70
+ name: >
+ Verify CI Images & Generate constraints
+ ${{needs.build-info.outputs.all-python-versions-list-as-string}}
+ runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
+ needs: [build-info, wait-for-ci-images]
+ env:
+ RUNS_ON: "${{ needs.build-info.outputs.runs-on }}"
+ PYTHON_VERSIONS: ${{needs.build-info.outputs.all-python-versions-list-as-string}}
+ DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
+ if: needs.build-info.outputs.ci-image-build == 'true'
+ steps:
+ - name: Cleanup repo
+ run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
+ uses: actions/checkout@v4
+ with:
+ persist-credentials: false
+ - name: "Install Breeze"
+ uses: ./.github/actions/breeze
+ - name: Pull CI images ${{ env.PYTHON_VERSIONS }}:${{ env.IMAGE_TAG }}
+ run: breeze ci-image pull --run-in-parallel --tag-as-latest
+ - name: Verify CI images ${{ env.PYTHON_VERSIONS }}:${{ env.IMAGE_TAG }}
+ run: breeze ci-image verify --run-in-parallel
+ env:
+ PYTHON_VERSIONS: ${{needs.build-info.outputs.all-python-versions-list-as-string}}
+ DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
+ - name: "Source constraints"
+ shell: bash
+ run: >
+ breeze release-management generate-constraints --run-in-parallel
+ --airflow-constraints-mode constraints-source-providers
+ - name: "No providers constraints"
+ shell: bash
+ timeout-minutes: 25
+ run: >
+ breeze release-management generate-constraints --run-in-parallel
+ --airflow-constraints-mode constraints-no-providers
+ - name: "PyPI constraints"
+ shell: bash
+ timeout-minutes: 25
+ run: >
+ breeze release-management generate-constraints --run-in-parallel
+ --airflow-constraints-mode constraints
+ - name: "Dependency upgrade summary"
+ shell: bash
+ run: |
+ for PYTHON_VERSION in ${{ env.PYTHON_VERSIONS }}; do
+ echo "Summarizing Python $PYTHON_VERSION"
+ cat "files/constraints-${PYTHON_VERSION}"/*.md >> $GITHUB_STEP_SUMMARY || true
+ done
+ - name: "Upload constraint artifacts"
+ uses: actions/upload-artifact@v3
+ with:
+ name: constraints
+ path: ./files/constraints-*/constraints-*.txt
+ retention-days: 7
+
static-checks:
timeout-minutes: 45
@@ -999,9 +930,7 @@ jobs:
JOB_ID: "helm-tests"
USE_XDIST: "true"
if: >
- needs.build-info.outputs.needs-helm-tests == 'true' &&
- (github.repository == 'apache/airflow' || github.event_name != 'schedule') &&
- needs.build-info.outputs.default-branch == 'main'
+ needs.build-info.outputs.needs-helm-tests == 'true' && needs.build-info.outputs.default-branch == 'main'
steps:
- name: Cleanup repo
run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
@@ -1644,10 +1573,93 @@ jobs:
echo
echo Total number of unique warnings $(cat ./artifacts/test-warnings*/* | sort | uniq | wc -l)
+ build-prod-images:
+ timeout-minutes: 80
+ name: >
+ ${{needs.build-info.outputs.build-job-description}} PROD images
+ ${{needs.build-info.outputs.all-python-versions-list-as-string}}
+ runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
+ needs: [build-info, build-ci-images]
+ env:
+ DEFAULT_BRANCH: ${{ needs.build-info.outputs.default-branch }}
+ DEFAULT_CONSTRAINTS_BRANCH: ${{ needs.build-info.outputs.default-constraints-branch }}
+ RUNS_ON: "${{needs.build-info.outputs.runs-on}}"
+ BACKEND: sqlite
+ VERSION_SUFFIX_FOR_PYPI: "dev0"
+ DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
+ # Force more parallelism for build even on public images
+ PARALLELISM: 6
+ steps:
+ - name: Cleanup repo
+ run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
+ if: needs.build-info.outputs.in-workflow-build == 'true'
+ - uses: actions/checkout@v4
+ with:
+ ref: ${{ needs.build-info.outputs.targetCommitSha }}
+ persist-credentials: false
+ if: needs.build-info.outputs.in-workflow-build == 'true'
+ - name: "Install Breeze"
+ uses: ./.github/actions/breeze
+ if: needs.build-info.outputs.in-workflow-build == 'true'
+ - name: >
+ Build PROD Images
+ ${{needs.build-info.outputs.all-python-versions-list-as-string}}:${{env.IMAGE_TAG}}
+ uses: ./.github/actions/build-prod-images
+ if: needs.build-info.outputs.in-workflow-build == 'true'
+ with:
+ build-provider-packages: ${{ needs.build-info.outputs.default-branch == 'main' }}
+ env:
+ UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgrade-to-newer-dependencies }}
+ DOCKER_CACHE: ${{ needs.build-info.outputs.cache-directive }}
+ PYTHON_VERSIONS: ${{needs.build-info.outputs.all-python-versions-list-as-string}}
+ DEBUG_RESOURCES: ${{ needs.build-info.outputs.debug-resources }}
+
+ build-prod-images-bullseye:
+ timeout-minutes: 80
+ name: >
+ Build Bullseye PROD images
+ ${{needs.build-info.outputs.all-python-versions-list-as-string}}
+ runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
+ needs: [build-info, build-ci-images]
+ if: needs.build-info.outputs.canary-run == 'true'
+ env:
+ DEFAULT_BRANCH: ${{ needs.build-info.outputs.default-branch }}
+ DEFAULT_CONSTRAINTS_BRANCH: ${{ needs.build-info.outputs.default-constraints-branch }}
+ RUNS_ON: "${{needs.build-info.outputs.runs-on}}"
+ BACKEND: sqlite
+ VERSION_SUFFIX_FOR_PYPI: "dev0"
+ DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
+ # Force more parallelism for build even on public images
+ PARALLELISM: 6
+ steps:
+ - name: Cleanup repo
+ run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
+ - uses: actions/checkout@v3
+ with:
+ ref: ${{ needs.build-info.outputs.targetCommitSha }}
+ persist-credentials: false
+ submodules: recursive
+ - name: "Install Breeze"
+ uses: ./.github/actions/breeze
+ - name: >
+ Build Bullseye PROD Images
+ ${{needs.build-info.outputs.all-python-versions-list-as-string}}:${{env.IMAGE_TAG}}
+ uses: ./.github/actions/build-prod-images
+ with:
+ build-provider-packages: ${{ needs.build-info.outputs.default-branch == 'main' }}
+ env:
+ UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgrade-to-newer-dependencies }}
+ DOCKER_CACHE: ${{ needs.build-info.outputs.cache-directive }}
+ PYTHON_VERSIONS: ${{needs.build-info.outputs.all-python-versions-list-as-string}}
+ DEBUG_RESOURCES: ${{ needs.build-info.outputs.debug-resources }}
+ DEBIAN_VERSION: "bullseye"
+ # Do not override the "bookworm" image - just push a new bullseye image
+ # TODO: improve caching for that build
+ IMAGE_TAG: "bullseye-${{ github.event.pull_request.head.sha || github.sha }}"
wait-for-prod-images:
timeout-minutes: 80
- name: "Wait for & verify PROD images"
+ name: "Wait for PROD images"
runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
needs: [build-info, wait-for-ci-images, build-prod-images]
if: needs.build-info.outputs.prod-image-build == 'true'
@@ -1660,12 +1672,15 @@ jobs:
steps:
- name: Cleanup repo
run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
+ if: needs.build-info.outputs.in-workflow-build == 'false'
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v4
with:
persist-credentials: false
+ if: needs.build-info.outputs.in-workflow-build == 'false'
- name: "Install Breeze"
uses: ./.github/actions/breeze
+ if: needs.build-info.outputs.in-workflow-build == 'false'
- name: Wait for PROD images ${{ env.PYTHON_VERSIONS }}:${{ env.IMAGE_TAG }}
# We wait for the images to be available either from "build-images.yml' run as pull_request_target
# or from build-prod-images above.
@@ -1675,18 +1690,11 @@ jobs:
env:
PYTHON_VERSIONS: ${{ needs.build-info.outputs.python-versions-list-as-string }}
DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
- - name: Verify PROD images ${{ env.PYTHON_VERSIONS }}:${{ env.IMAGE_TAG }}
- # We pull images again (which is a NOOP) but this time we also verify the images
- # Having it as a separate step allows us to see if the problem was with waiting or verification
- run: breeze prod-image pull --verify --run-in-parallel
- env:
- PYTHON_VERSIONS: ${{ needs.build-info.outputs.python-versions-list-as-string }}
- DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
-
+ if: needs.build-info.outputs.in-workflow-build == 'false'
test-docker-compose-quick-start:
timeout-minutes: 60
- name: "Test docker-compose quick start"
+ name: "Verify PROD image and docker-compose quick start"
runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
needs: [build-info, wait-for-prod-images]
if: needs.build-info.outputs.prod-image-build == 'true'
@@ -1700,12 +1708,17 @@ jobs:
with:
fetch-depth: 2
persist-credentials: false
- - name: >
- Prepare breeze & PROD image:
- ${{needs.build-info.outputs.default-python-version}}:${{env.IMAGE_TAG}}
- uses: ./.github/actions/prepare_breeze_and_image
- with:
- pull-image-type: 'PROD'
+ - name: Pull PROD images ${{ env.PYTHON_VERSIONS }}:${{ env.IMAGE_TAG }}
+ run: breeze prod-image pull --run-in-parallel --tag-as-latest
+ env:
+ PYTHON_VERSIONS: ${{ needs.build-info.outputs.python-versions-list-as-string }}
+ # Force more parallelism for pull even on public images
+ PARALLELISM: 6
+ - name: Verify PROD images ${{ env.PYTHON_VERSIONS }}:${{ env.IMAGE_TAG }}
+ run: breeze prod-image verify --run-in-parallel
+ env:
+ PYTHON_VERSIONS: ${{needs.build-info.outputs.all-python-versions-list-as-string}}
+ DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
- name: "Test docker-compose quick start"
run: breeze testing docker-compose-tests
@@ -1741,6 +1754,8 @@ jobs:
run: breeze prod-image pull --run-in-parallel --tag-as-latest
env:
PYTHON_VERSIONS: ${{ needs.build-info.outputs.python-versions-list-as-string }}
+ # Force more parallelism for pull even on public images
+ PARALLELISM: 6
- name: "Cache bin folder with tools for kubernetes testing"
uses: actions/cache@v3
with:
@@ -1796,7 +1811,6 @@ jobs:
- tests-no-db
- tests-integration-postgres
- tests-integration-mysql
- # Skip when generate constraints fails
- generate-constraints
env:
RUNS_ON: "${{needs.build-info.outputs.runs-on}}"
diff --git a/CI.rst b/CI.rst
index 867686d8e7..b464817f6e 100644
--- a/CI.rst
+++ b/CI.rst
@@ -117,22 +117,6 @@ have to be percent-encoded when you access them via UI (/ = %2F)
| (DockerHub) | | Python maintainer release new versions of those image |
| | | with security fixes every few weeks in DockerHub. |
+--------------+----------------------------------------------------------+----------------------------------------------------------+
-| Airflow | airflow/<BRANCH>/python:<X.Y>-slim-bookworm | Version of python base image used in Airflow Builds |
-| python base | | We keep the "latest" version only to mark last "good" |
-| image | | python base that went through testing and was pushed. |
-+--------------+----------------------------------------------------------+----------------------------------------------------------+
-| PROD Build | airflow/<BRANCH>/prod-build/python<X.Y>:latest | Production Build image - this is the "build" stage of |
-| image | | production image. It contains build-essentials and all |
-| | | necessary apt packages to build/install PIP packages. |
-| | | We keep the "latest" version only to speed up builds. |
-+--------------+----------------------------------------------------------+----------------------------------------------------------+
-| Manifest | airflow/<BRANCH>/ci-manifest/python<X.Y>:latest | CI manifest image - this is the image used to optimize |
-| CI image | | pulls and builds for Breeze development environment |
-| | | They store hash indicating whether the image will be |
-| | | faster to build or pull. |
-| | | We keep the "latest" version only to help breeze to |
-| | | check if new image should be pulled. |
-+--------------+----------------------------------------------------------+----------------------------------------------------------+
| CI image | airflow/<BRANCH>/ci/python<X.Y>:latest | CI image - this is the image used for most of the tests. |
| | or | Contains all provider dependencies and tools useful |
| | airflow/<BRANCH>/ci/python<X.Y>:<COMMIT_SHA> | For testing. This image is used in Breeze. |
@@ -239,49 +223,39 @@ Regular PR builds run in a "stable" environment:
* no ARM image builds are build in the regular PRs
* lower probability of flaky tests for non-committer PRs (public runners and less parallelism)
+Maintainers can also run the "Pull Request run" from the "apache/airflow" repository by pushing
+to a branch in the "apache/airflow" repository. This is useful when you want to test a PR that
+changes the CI/CD infrastructure itself (for example changes to the CI/CD scripts or changes to
+the CI/CD workflows). In this case the PR is run in the context of the "apache/airflow" repository
+and has WRITE access to the GitHub Container Registry.
+
Canary run
----------
-Those runs are results of direct pushes done by the committers - basically merging of a Pull Request
-by the committers. Those runs execute in the context of the Apache Airflow Code Repository and have also
-write permission for GitHub resources (container registry, code repository).
+This is the flow that happens when a pull request is merged to the "main" branch or pushed to any of
+the "v2-*-test" branches. The "Canary" run attempts to upgrade dependencies to the latest versions
+and quickly pushes a preview of cache the CI/PROD images to the GitHub Registry - so that pull requests
+can quickly use the new cache - this is useful when Dockerfile or installation scripts change because such
+cache will already have the latest Dockerfile and scripts pushed even if some tests will fail.
+When successful, the run updates the constraints files in the "constraints-main" branch with the latest
+constraints and pushes both cache and latest CI/PROD images to the GitHub Registry.
-The main purpose for the run is to check if the code after merge still holds all the assertions - like
-whether it still builds, all tests are green. This is a "Canary" build that helps us to detect early
-problems with dependencies, image building, full matrix of tests in case they passed through selective checks.
+When "Canary" build fails, it's often a sign that some of our dependencies released a new version that
+is not compatible with current tests or Airflow code, Also it might mean that a breaking change has been
+merged to "main". Both cases should be addressed quickly by the maintainers. The "broken main" by our code
+should be fixed quickly, while the "broken dependencies" can take a bit of time to fix as until the tests
+succeeds, constraints will not be updated, which means that regular PRs will continue using the old version
+of dependencies that already passed one of the previous "Canary" runs.
-This is needed because some of the conflicting changes from multiple PRs might cause build and test failures
-after merge even if they do not fail in isolation. Also those runs are already reviewed and confirmed by the
-committers so they can be used to do some housekeeping:
-
-- pushing most recent image build in the PR to the GitHub Container Registry (for caching) including recent
- Dockerfile changes and setup.py/setup.cfg changes (Early Cache)
-- test that image in ``breeze`` command builds quickly
-- run full matrix of tests to detect any tests that will be mistakenly missed in ``selective checks``
-- upgrading to latest constraints and pushing those constraints if all tests succeed
-- refresh latest Python base images in case new patch-level is released
-
-The housekeeping is important - Python base images are refreshed with varying frequency (once every few months
-usually but sometimes several times per week) with the latest security and bug fixes.
Scheduled runs
--------------
-Those runs are results of (nightly) triggered job - only for ``main`` branch. The
-main purpose of the job is to check if there was no impact of external dependency changes on the Apache
-Airflow code (for example transitive dependencies released that fail the build). It also checks if the
-Docker images can be built from the scratch (again - to see if some dependencies have not changed - for
-example downloaded package releases etc.
-
-All runs consist of the same jobs, but the jobs behave slightly differently or they are skipped in different
-run categories. Here is a summary of the run categories with regards of the jobs they are running.
-Those jobs often have matrix run strategy which runs several different variations of the jobs
-(with different Backend type / Python version, type of the tests to run for example). The following chapter
-describes the workflows that execute for each run.
-
-Those runs and their corresponding ``Build Images`` runs are only executed in main ``apache/airflow``
-repository, they are not executed in forks - we want to be nice to the contributors and not use their
-free build minutes on GitHub Actions.
+This is the flow that happens when a scheduled run is triggered. The "scheduled" workflow is aimed to
+run regularly (overnight). Scheduled run is generally the same as "Canary" run, with the difference
+that the image is build always from the scratch and not from the cache. This way we can check that no
+"system" dependencies in debian base image have changed and that the build is still reproducible.
+No separate diagram is needed for scheduled run as it is identical to that of "Canary" run.
Workflows
=========
@@ -364,10 +338,12 @@ This workflow is a regular workflow that performs all checks of Airflow code.
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
| Build CI images | Builds images in-workflow (not in the ``build images``) | - | Yes | Yes (1) | Yes (4) |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
-| Generate constraints | Generates constraints that were updated in this build | Yes (2) | Yes (2) | Yes (2) | Yes (2) |
+| Verify CI/generate constraints | Verify CI image and generate constraints for the build | Yes (2) | Yes (2) | Yes (2) | Yes (2) |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
| Build PROD images | Builds images in-workflow (not in the ``build images``) | - | Yes | Yes (1) | Yes (4) |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
+| Build Bullseye PROD images | Builds images based on Bullseye debian | - | Yes | Yes | Yes |
++---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
| Run breeze tests | Run unit tests for Breeze | Yes | Yes | Yes | Yes |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
| Test OpenAPI client gen | Tests if OpenAPIClient continues to generate | Yes | Yes | Yes | Yes |
@@ -412,7 +388,7 @@ This workflow is a regular workflow that performs all checks of Airflow code.
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
| Wait for PROD Images | Waits for and verify PROD Images | Yes (2) | Yes (2) | Yes (2) | Yes (2) |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
-| Test docker-compose | Tests if quick-start docker compose works | Yes | Yes | Yes | Yes |
+| Verify PROD/test compose | Verify PROD image and tests quick-start Docker Compose | Yes | Yes | Yes | Yes |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
| Tests Kubernetes | Run Kubernetes test | Yes | Yes | Yes | - |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
@@ -429,9 +405,10 @@ This workflow is a regular workflow that performs all checks of Airflow code.
``(2)`` The jobs wait for CI images to be available. It only actually runs when build image is needed (in
case of simpler PRs that do not change dependencies or source code, images are not build)
-``(3)`` PROD and CI cache & images are pushed as "latest" to GitHub Container registry and constraints are
-upgraded only if all tests are successful. The images are rebuilt in this step using constraints pushed
-in the previous step. Constraints are only actually pushed in the ``canary`` runs.
+``(3)`` PROD and CI cache & images are pushed as "cache" (both AMD and ARM) and "latest" (only AMD)
+to GitHub Container registry and constraints are upgraded only if all tests are successful.
+The images are rebuilt in this step using constraints pushed in the previous step.
+Constraints are only actually pushed in the ``canary/scheduled`` runs.
``(4)`` In main, PROD image uses locally build providers using "latest" version of the provider code. In the
non-main version of the build, the latest released providers from PyPI are used.
diff --git a/CI_DIAGRAMS.md b/CI_DIAGRAMS.md
index 8bc1baabaf..a921f4187a 100644
--- a/CI_DIAGRAMS.md
+++ b/CI_DIAGRAMS.md
@@ -19,10 +19,21 @@
# CI Sequence diagrams
-You can see here the sequence diagrams of the flow happening during the CI Jobs.
+You can see here the sequence diagrams of the flow happening during the CI Jobs. More detailed description
+for the CI flows can be found in the [CI.rst](CI.rst) document.
## Pull request flow from fork
+This is the flow that happens when a pull request is created from a fork - which is the most frequent
+pull request flow that happens in Airflow. The "pull_request" workflow does not have write access
+to the GitHub Registry, so it cannot push the CI/PROD images there. Instead, we push the images
+from the "pull_request_target" workflow, which has write access to the GitHub Registry. Note that
+this workflow always uses scripts and workflows from the "target" branch of the "apache/airflow"
+repository, so the user submitting such pull request cannot override our build scripts and inject malicious
+code into the workflow that has potentially write access to the GitHub Registry (and can override cache).
+
+Security is the main reason why we have two workflows for pull requests and such complex workflows.
+
```mermaid
sequenceDiagram
Note over Airflow Repo: pull request
@@ -33,12 +44,17 @@ sequenceDiagram
activate Tests
Tests -->> Build Images: Trigger 'pull_request_target'
activate Build Images
- Note over Build Images: Build info<br>Decide which Python
- Note over Tests: Build info<br>Decide on tests<br>Decide on Matrix (selective)
+ Note over Tests: Build info
+ Note over Tests: Selective checks<br>Decide what to do
+ Note over Build Images: Build info
+ Note over Build Images: Selective checks<br>Decide what to do
Note over Tests: Skip Build<br>(Runs in 'Build Images')<br>CI Images
Note over Tests: Skip Build<br>(Runs in 'Build Images')<br>PROD Images
par
+ GitHub Registry ->> Build Images: Use cache from registry
Note over Build Images: Build CI Images<br>[COMMIT_SHA]<br>Use latest constraints<br>or upgrade if setup changed
+ Build Images ->> GitHub Registry: Push CI Images<br>[COMMIT_SHA]
+ Build Images ->> Artifacts: Upload source constraints
and
Note over Tests: OpenAPI client gen
and
@@ -52,14 +68,19 @@ sequenceDiagram
Note over Tests: Run basic <br>static checks
end
end
- Build Images ->> GitHub Registry: Push CI Images<br>[COMMIT_SHA]
loop Wait for CI images
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
end
- Note over Tests: Verify CI Images<br>[COMMIT_SHA]
par
- GitHub Registry ->> Build Images: Pull PROD Images<br>[latest]
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Verify CI Images<br>[COMMIT_SHA]
+ Note over Tests: Generate constraints<br>source,pypi,no-providers
+ Tests ->> Artifacts: Upload source,pypi,no-providers constraints
+ and
+ Artifacts ->> Build Images: Download source constraints
+ GitHub Registry ->> Build Images: Use cache from registry
Note over Build Images: Build PROD Images<br>[COMMIT_SHA]
+ Build Images ->> GitHub Registry: Push PROD Images<br>[COMMIT_SHA]
and
opt
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
@@ -68,52 +89,67 @@ sequenceDiagram
and
opt
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Build docs / Spellcheck docs
+ Note over Tests: Build docs
end
and
opt
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Test Pytest collection<br>[COMMIT_SHA]
- par
- opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Unit Tests<br>Python/DB matrix
- end
- and
- opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Integration Tests
- end
- and
- opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Quarantined Tests
- end
- end
+ Note over Tests: Spellcheck docs
+ end
+ and
+ opt
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Unit Tests<br>Python/DB matrix
end
and
opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Test provider <br>packages build
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Unit Tests<br>Python/Non-DB matrix
end
and
opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Test airflow <br>packages build
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Integration Tests
end
and
opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Helm tests
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Quarantined Tests
+ end
+ and
+ opt
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Build/test provider packages<br>wheel, sdist, old airflow
+ end
+ and
+ opt
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Test airflow <br>release commands
+ end
+ and
+ opt
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Helm tests
end
end
- Note over Tests: Summarize Warnings
- Build Images ->> GitHub Registry: Push PROD Images<br>[COMMIT_SHA]
- deactivate Build Images
- loop Wait for PROD images
- GitHub Registry ->> Tests: Pull PROD Images<br>[COMMIT_SHA]
+ par
+ Note over Tests: Summarize Warnings
+ and
+ opt
+ Artifacts ->> Tests: Download source,pypi,no-providers constraints
+ Note over Tests: Display constraints diff
+ end
+ and
+ opt
+ loop Wait for PROD images
+ GitHub Registry ->> Tests: Pull PROD Images<br>[COMMIT_SHA]
+ end
+ end
+ and
+ opt
+ Note over Tests: Build ARM CI images
+ end
end
- Note over Tests: Verify PROD Image<br>[COMMIT_SHA]
par
opt
GitHub Registry ->> Tests: Pull PROD Images<br>[COMMIT_SHA]
@@ -122,14 +158,10 @@ sequenceDiagram
and
opt
GitHub Registry ->> Tests: Pull PROD Images<br>[COMMIT_SHA]
+ Note over Tests: Verify PROD Images<br>[COMMIT_SHA]
Note over Tests: Run docker-compose <br>tests
end
end
- opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Generate constraints
- end
- Note over Tests: Build ARM CI images
Tests -->> Airflow Repo: Status update
deactivate Airflow Repo
deactivate Tests
@@ -137,27 +169,44 @@ sequenceDiagram
## Pull request flow from "apache/airflow" repo
+The difference between this flow and the previous one is that the CI/PROD images are built in the
+CI workflow and pushed to the GitHub Registry from there. This cannot be done in case of fork
+pull request, because Pull Request from forks cannot have "write" access to GitHub Registry. All the steps
+except "Build Info" from the "Build Images" workflows are skipped in this case.
+
+THis workflow can be used by maintainers in case they have a Pull Request that changes the scripts and
+CI workflows used to build images, because in this case the "Build Images" workflow will use them
+from the Pull Request. This is safe, because the Pull Request is from the "apache/airflow" repository
+and only maintainers can push to that repository and create Pull Requests from it.
+
```mermaid
sequenceDiagram
Note over Airflow Repo: pull request
- Note over Tests: pull_request<br>[Read Token]
- Note over Build Images: pull_request_target<br>[Write Token]
+ Note over Tests: pull_request<br>[Write Token]
+ Note over Build Images: pull_request_target<br>[Unused Token]
activate Airflow Repo
Airflow Repo -->> Tests: Trigger 'pull_request'
activate Tests
Tests -->> Build Images: Trigger 'pull_request_target'
activate Build Images
+ Note over Tests: Build info
+ Note over Tests: Selective checks<br>Decide what to do
Note over Build Images: Build info
+ Note over Build Images: Selective checks<br>Decide what to do
Note over Build Images: Skip Build<br>(Runs in 'Tests')<br>CI Images
Note over Build Images: Skip Build<br>(Runs in 'Tests')<br>PROD Images
deactivate Build Images
- Note over Tests: Build info<br>Decide on tests<br>Decide on Matrix (selective)
+ Note over Tests: Build info
+ Note over Tests: Selective checks<br>Decide what to do
par
+ GitHub Registry ->> Tests: Use cache from registry
Note over Tests: Build CI Images<br>[COMMIT_SHA]<br>Use latest constraints<br>or upgrade if setup changed
+ Tests ->> GitHub Registry: Push CI Images<br>[COMMIT_SHA]
+ Tests ->> Artifacts: Upload source constraints
and
Note over Tests: OpenAPI client gen
and
- Note over Tests: Test UI
+ Note over Tests: React WWW tests
and
Note over Tests: Test examples<br>PROD image building
and
@@ -167,11 +216,17 @@ sequenceDiagram
Note over Tests: Run basic <br>static checks
end
end
- Tests ->> GitHub Registry: Push CI Images<br>[COMMIT_SHA]
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Verify CI Image<br>[COMMIT_SHA]
+ Note over Tests: Skip waiting for CI images
par
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Verify CI Images<br>[COMMIT_SHA]
+ Note over Tests: Generate constraints<br>source,pypi,no-providers
+ Tests ->> Artifacts: Upload source,pypi,no-providers constraints
+ and
+ Artifacts ->> Tests: Download source constraints
+ GitHub Registry ->> Tests: Use cache from registry
Note over Tests: Build PROD Images<br>[COMMIT_SHA]
+ Tests ->> GitHub Registry: Push PROD Images<br>[COMMIT_SHA]
and
opt
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
@@ -180,50 +235,60 @@ sequenceDiagram
and
opt
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Build docs/ Spellcheck docs
+ Note over Tests: Build docs
end
and
opt
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Test Pytest collection<br>[COMMIT_SHA]
- par
- opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Unit Tests<br>Python/DB matrix/No DB
- end
- and
- opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Integration Tests
- end
- and
- opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Quarantined Tests
- end
- end
+ Note over Tests: Spellcheck docs
+ end
+ and
+ opt
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Unit Tests<br>Python/DB matrix
end
and
opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Test provider <br>packages build
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Unit Tests<br>Python/Non-DB matrix
end
and
opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Test airflow <br>packages build
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Integration Tests
end
and
opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Helm tests
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Quarantined Tests
+ end
+ and
+ opt
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Build/test provider packages<br>wheel, sdist, old airflow
+ end
+ and
+ opt
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Test airflow <br>release commands
+ end
+ and
+ opt
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Helm tests
end
end
- Note over Tests: Summarize Warnings
- Tests ->> GitHub Registry: Push PROD Images<br>[COMMIT_SHA]
- GitHub Registry ->> Tests: Pull PROD Image<br>[COMIT_SHA]
- Note over Tests: Verify PROD Image<br>[COMMIT_SHA]
+ Note over Tests: Skip waiting for PROD images
par
+ Note over Tests: Summarize Warnings
+ and
+ opt
+ Artifacts ->> Tests: Download source,pypi,no-providers constraints
+ Note over Tests: Display constraints diff
+ end
+ and
+ Note over Tests: Build ARM CI images
+ and
opt
GitHub Registry ->> Tests: Pull PROD Images<br>[COMMIT_SHA]
Note over Tests: Run Kubernetes <br>tests
@@ -231,14 +296,10 @@ sequenceDiagram
and
opt
GitHub Registry ->> Tests: Pull PROD Images<br>[COMMIT_SHA]
+ Note over Tests: Verify PROD Images<br>[COMMIT_SHA]
Note over Tests: Run docker-compose <br>tests
end
end
- opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Generate constraints
- end
- Note over Tests: Build ARM CI images
Tests -->> Airflow Repo: Status update
deactivate Airflow Repo
deactivate Tests
@@ -246,6 +307,14 @@ sequenceDiagram
## Merge "Canary" run
+This is the flow that happens when a pull request is merged to the "main" branch or pushed to any of
+the "v2-*-test" branches. The "Canary" run attempts to upgrade dependencies to the latest versions
+and quickly pushes an early cache the CI/PROD images to the GitHub Registry - so that pull requests
+can quickly use the new cache - this is useful when Dockerfile or installation scripts change because such
+cache will already have the latest Dockerfile and scripts pushed even if some tests will fail.
+When successful, the run updates the constraints files in the "constraints-main" branch with the latest
+constraints and pushes both cache and latest CI/PROD images to the GitHub Registry.
+
```mermaid
sequenceDiagram
Note over Airflow Repo: push/merge
@@ -253,13 +322,20 @@ sequenceDiagram
activate Airflow Repo
Airflow Repo -->> Tests: Trigger 'push'
activate Tests
- Note over Tests: Build info<br>All tests<br>Full matrix
+ Note over Tests: Build info
+ Note over Tests: Selective checks<br>Decide what to do
par
+ GitHub Registry ->> Tests: Use cache from registry<br>(Not for scheduled run)
Note over Tests: Build CI Images<br>[COMMIT_SHA]<br>Always upgrade deps
+ Tests ->> GitHub Registry: Push CI Images<br>[COMMIT_SHA]
+ Tests ->> Artifacts: Upload source constraints
and
+ GitHub Registry ->> Tests: Use cache from registry<br>(Not for scheduled run)
Note over Tests: Check that image builds quickly
and
- Note over Tests: Push early cache and images
+ GitHub Registry ->> Tests: Use cache from registry<br>(Not for scheduled run)
+ Note over Tests: Push early CI Image cache
+ Tests ->> GitHub Registry: Push CI cache Images
and
Note over Tests: OpenAPI client gen
and
@@ -269,147 +345,91 @@ sequenceDiagram
and
Note over Tests: Test git clone on Windows
end
- Tests ->> GitHub Registry: Push CI Images<br>[COMMIT_SHA]
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Verify CI Image<br>[COMMIT_SHA]
+ Note over Tests: Skip waiting for CI images
par
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Verify CI Images<br>[COMMIT_SHA]
+ Note over Tests: Generate constraints<br>source,pypi,no-providers
+ Tests ->> Artifacts: Upload source,pypi,no-providers constraints
+ and
+ Artifacts ->> Tests: Download source constraints
+ GitHub Registry ->> Tests: Use cache from registry
Note over Tests: Build PROD Images<br>[COMMIT_SHA]
+ Tests ->> GitHub Registry: Push PROD Images<br>[COMMIT_SHA]
+ and
+ Artifacts ->> Tests: Download source constraints
+ Note over Tests: Build Bullseye PROD Images<br>[COMMIT_SHA]
and
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
Note over Tests: Run static checks
and
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Build docs / Spellcheck docs
+ Note over Tests: Build docs
and
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Test Pytest collection<br>[COMMIT_SHA]
- par
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Unit Tests<br>Python/DB matrix
- and
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Integration Tests
- and
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Quarantined Tests
- end
- and
- opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Test provider <br>packages build
- end
+ Note over Tests: Spellcheck docs
and
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Test airflow <br>packages build
- and
- opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Helm tests
- end
- end
- Note over Tests: Summarize Warnings
- Tests ->> GitHub Registry: Push PROD Images<br>[COMMIT_SHA]
- GitHub Registry ->> Tests: Pull PROD Image<br>[COMMIT_SHA]
- Note over Tests: Verify PROD Image<br>[COMMIT_SHA]
- par
- GitHub Registry ->> Tests: Pull PROD Image<br>[COMMIT_SHA]
- Note over Tests: Run Kubernetes <br>tests
- and
- GitHub Registry ->> Tests: Pull PROD Image<br>[COMMIT_SHA]
- Note over Tests: Run docker-compose <br>tests
- end
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Generate constraints
- Tests ->> Airflow Repo: Push constraints if changed
- Note over Tests: Build CI Images<br>[latest]<br>Use latest constraints
- Tests ->> GitHub Registry: Push CI Image<br>[latest]
- Note over Tests: Build PROD Images<br>[latest]<br>Use latest constraints
- Tests ->> GitHub Registry: Push PROD Image<br>[latest]
- Note over Tests: Build ARM CI images
- Tests ->> GitHub Registry: Push ARM CI Image cache
- Tests -->> Airflow Repo: Status update
- deactivate Airflow Repo
- deactivate Tests
-```
-
-## Scheduled run
-
-```mermaid
-sequenceDiagram
- Note over Airflow Repo: scheduled
- Note over Tests: push<br>[Write Token]
- activate Airflow Repo
- Airflow Repo -->> Tests: Trigger 'schedule'
- activate Tests
- Note over Tests: Build info<br>All tests<br>Full matrix
- par
- GitHub Registry ->> Tests: Pull CI Images<br>[latest]
- Note over Tests: Build CI Images<br>[COMMIT_SHA]<br>Always upgrade deps
- and
- Note over Tests: OpenAPI client gen
- and
- Note over Tests: React WWW tests
- and
- Note over Tests: Test examples<br>PROD image building
- and
- Note over Tests: Build CI Images<br>Use original constraints
- end
- Tests ->> GitHub Registry: Push CI Images<br>[COMMIT_SHA]
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Verify CI Image<br>[COMMIT_SHA]
- par
- GitHub Registry ->> Tests: Pull PROD Images<br>[latest]
- Note over Tests: Build PROD Images<br>[COMMIT_SHA]
+ Note over Tests: Unit Tests<br>Python/DB matrix
and
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Run static checks
+ Note over Tests: Unit Tests<br>Python/Non-DB matrix
and
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Build docs / Spellcheck docs
+ Note over Tests: Integration Tests
and
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Test Pytest collection<br>[COMMIT_SHA]
- par
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Unit Tests<br>Python/DB matrix
- and
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Integration Tests
- and
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Quarantined Tests
- end
+ Note over Tests: Quarantined Tests
and
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Test provider <br>packages build
+ Note over Tests: Build/test provider packages<br>wheel, sdist, old airflow
and
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Test airflow <br>packages build
+ Note over Tests: Test airflow <br>release commands
and
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
Note over Tests: Helm tests
end
- Note over Tests: Summarize Warnings
- Tests ->> GitHub Registry: Push PROD Images<br>[COMMIT_SHA]
- GitHub Registry ->> Tests: Pull PROD Image<br>[COMMIT_SHA]
- Note over Tests: Verify PROD Image<br>[COMMIT_SHA]
+ Note over Tests: Skip waiting for PROD images
par
+ Note over Tests: Summarize Warnings
+ and
+ Artifacts ->> Tests: Download source,pypi,no-providers constraints
+ Note over Tests: Display constraints diff
+ Tests ->> Airflow Repo: Push constraints if changed to 'constraints-BRANCH'
+ and
GitHub Registry ->> Tests: Pull PROD Image<br>[COMMIT_SHA]
Note over Tests: Run Kubernetes <br>tests
and
GitHub Registry ->> Tests: Pull PROD Image<br>[COMMIT_SHA]
+ Note over Tests: Verify PROD Images<br>[COMMIT_SHA]
Note over Tests: Run docker-compose <br>tests
end
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Generate constraints
- Tests ->> Airflow Repo: Push constraints if changed
- Note over Tests: Build CI Images<br>[latest]<br>Use latest constraints
- Tests ->> GitHub Registry: Push CI Image cache + latest
- Note over Tests: Build PROD Images<br>[latest]<br>Use latest constraints
- Tests ->> GitHub Registry: Push PROD Image cache + latest
- Note over Tests: Build ARM CI images
- Tests ->> GitHub Registry: Push ARM CI Image cache
+ par
+ GitHub Registry ->> Tests: Use cache from registry
+ Note over Tests: Build CI latest images/cache<br>Use pushed constraints
+ Tests ->> GitHub Registry: Push CI latest images/cache
+ GitHub Registry ->> Tests: Use cache from registry
+ Note over Tests: Build PROD latest images/cache<br>Use pushed constraints
+ Tests ->> GitHub Registry: Push PROD latest images/cache
+ and
+ GitHub Registry ->> Tests: Use cache from registry
+ Note over Tests: Build ARM CI cache<br>Use pushed constraints
+ Tests ->> GitHub Registry: Push ARM CI cache
+ GitHub Registry ->> Tests: Use cache from registry
+ Note over Tests: Build ARM PROD cache<br>Use pushed constraints
+ Tests ->> GitHub Registry: Push ARM PROD cache
+ end
Tests -->> Airflow Repo: Status update
deactivate Airflow Repo
deactivate Tests
```
+
+## Scheduled run
+
+This is the flow that happens when a scheduled run is triggered. The "scheduled" workflow is aimed to
+run regularly (overnight) even if no new PRs are merged to "main". Scheduled run is generally the
+same as "Canary" run, with the difference that the image used to run the tests is built without using
+cache - it's always built from the scratch. This way we can check that no "system" dependencies in debian
+base image have changed and that the build is still reproducible. No separate diagram is needed for
+scheduled run as it is identical to that of "Canary" run.
diff --git a/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py b/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
index f33fdad2ef..08722caf30 100644
--- a/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
@@ -93,7 +93,12 @@ from airflow_breeze.utils.docker_command_utils import (
from airflow_breeze.utils.image import run_pull_image, run_pull_in_parallel, tag_image_as_latest
from airflow_breeze.utils.mark_image_as_refreshed import mark_image_as_refreshed
from airflow_breeze.utils.md5_build_check import md5sum_check_if_build_is_needed
-from airflow_breeze.utils.parallel import DockerBuildxProgressMatcher, check_async_run_results, run_with_pool
+from airflow_breeze.utils.parallel import (
+ DockerBuildxProgressMatcher,
+ ShowLastLineProgressMatcher,
+ check_async_run_results,
+ run_with_pool,
+)
from airflow_breeze.utils.path_utils import AIRFLOW_SOURCES_ROOT, BUILD_CACHE_DIR
from airflow_breeze.utils.python_versions import get_python_version_list
from airflow_breeze.utils.registry import login_to_github_docker_registry
@@ -476,6 +481,45 @@ def pull(
sys.exit(return_code)
+def run_verify_in_parallel(
+ image_params_list: list[BuildCiParams],
+ python_version_list: list[str],
+ extra_pytest_args: tuple[str, ...],
+ include_success_outputs: bool,
+ parallelism: int,
+ skip_cleanup: bool,
+ debug_resources: bool,
+) -> None:
+ with ci_group(f"Verifying CI images for {python_version_list}"):
+ all_params = [f"CI {image_params.python}" for image_params in image_params_list]
+ with run_with_pool(
+ parallelism=parallelism,
+ all_params=all_params,
+ debug_resources=debug_resources,
+ progress_matcher=ShowLastLineProgressMatcher(),
+ ) as (pool, outputs):
+ results = [
+ pool.apply_async(
+ verify_an_image,
+ kwds={
+ "image_name": image_params.airflow_image_name_with_tag,
+ "image_type": "CI",
+ "slim_image": False,
+ "extra_pytest_args": extra_pytest_args,
+ "output": outputs[index],
+ },
+ )
+ for index, image_params in enumerate(image_params_list)
+ ]
+ check_async_run_results(
+ results=results,
+ success="All images verified",
+ outputs=outputs,
+ include_success_outputs=include_success_outputs,
+ skip_cleanup=skip_cleanup,
+ )
+
+
@ci_image.command(
name="verify",
context_settings=dict(
@@ -484,22 +528,34 @@ def pull(
),
)
@option_python
+@option_python_versions
@option_github_repository
@option_image_tag_for_verifying
@option_image_name
@option_pull
@option_github_token
+@option_run_in_parallel
+@option_parallelism
+@option_skip_cleanup
+@option_include_success_outputs
+@option_debug_resources
@option_verbose
@option_dry_run
@click.argument("extra_pytest_args", nargs=-1, type=click.UNPROCESSED)
def verify(
python: str,
+ python_versions: str,
image_name: str,
image_tag: str | None,
pull: bool,
github_token: str,
github_repository: str,
- extra_pytest_args: tuple,
+ extra_pytest_args: tuple[str, ...],
+ run_in_parallel: bool,
+ parallelism: int,
+ skip_cleanup: bool,
+ debug_resources: bool,
+ include_success_outputs: bool,
):
"""Verify CI image."""
perform_environment_checks()
@@ -507,27 +563,54 @@ def verify(
github_token=github_token,
output=None,
)
- if image_name is None:
- build_params = BuildCiParams(
+ if (pull or image_name) and run_in_parallel:
+ get_console().print(
+ "[error]You cannot use --pull,--image-name and --run-in-parallel at the same time. " "Exiting[/]"
+ )
+ sys.exit(1)
+ if run_in_parallel:
+ base_build_params = BuildCiParams(
python=python,
- image_tag=image_tag,
github_repository=github_repository,
- github_token=github_token,
+ image_tag=image_tag,
)
- image_name = build_params.airflow_image_name_with_tag
- if pull:
- check_remote_ghcr_io_commands()
- command_to_run = ["docker", "pull", image_name]
- run_command(command_to_run, check=True)
- get_console().print(f"[info]Verifying CI image: {image_name}[/]")
- return_code, info = verify_an_image(
- image_name=image_name,
- output=None,
- image_type="CI",
- slim_image=False,
- extra_pytest_args=extra_pytest_args,
- )
- sys.exit(return_code)
+ python_version_list = get_python_version_list(python_versions)
+ params_list: list[BuildCiParams] = []
+ for python in python_version_list:
+ build_params = deepcopy(base_build_params)
+ build_params.python = python
+ params_list.append(build_params)
+ run_verify_in_parallel(
+ image_params_list=params_list,
+ python_version_list=python_version_list,
+ extra_pytest_args=extra_pytest_args,
+ include_success_outputs=include_success_outputs,
+ parallelism=parallelism,
+ skip_cleanup=skip_cleanup,
+ debug_resources=debug_resources,
+ )
+ else:
+ if image_name is None:
+ build_params = BuildCiParams(
+ python=python,
+ image_tag=image_tag,
+ github_repository=github_repository,
+ github_token=github_token,
+ )
+ image_name = build_params.airflow_image_name_with_tag
+ if pull:
+ check_remote_ghcr_io_commands()
+ command_to_run = ["docker", "pull", image_name]
+ run_command(command_to_run, check=True)
+ get_console().print(f"[info]Verifying CI image: {image_name}[/]")
+ return_code, info = verify_an_image(
+ image_name=image_name,
+ output=None,
+ image_type="CI",
+ slim_image=False,
+ extra_pytest_args=extra_pytest_args,
+ )
+ sys.exit(return_code)
def should_we_run_the_build(build_ci_params: BuildCiParams) -> bool:
diff --git a/dev/breeze/src/airflow_breeze/commands/ci_image_commands_config.py b/dev/breeze/src/airflow_breeze/commands/ci_image_commands_config.py
index e128aa035b..021bec3a68 100644
--- a/dev/breeze/src/airflow_breeze/commands/ci_image_commands_config.py
+++ b/dev/breeze/src/airflow_breeze/commands/ci_image_commands_config.py
@@ -144,6 +144,17 @@ CI_IMAGE_TOOLS_PARAMETERS: dict[str, list[dict[str, str | list[str]]]] = {
"--pull",
],
},
+ {
+ "name": "Parallel running",
+ "options": [
+ "--run-in-parallel",
+ "--parallelism",
+ "--python-versions",
+ "--skip-cleanup",
+ "--debug-resources",
+ "--include-success-outputs",
+ ],
+ },
{
"name": "Github authentication",
"options": [
diff --git a/dev/breeze/src/airflow_breeze/commands/production_image_commands.py b/dev/breeze/src/airflow_breeze/commands/production_image_commands.py
index ca1780738d..27f195bc2e 100644
--- a/dev/breeze/src/airflow_breeze/commands/production_image_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/production_image_commands.py
@@ -86,7 +86,12 @@ from airflow_breeze.utils.docker_command_utils import (
warm_up_docker_builder,
)
from airflow_breeze.utils.image import run_pull_image, run_pull_in_parallel, tag_image_as_latest
-from airflow_breeze.utils.parallel import DockerBuildxProgressMatcher, check_async_run_results, run_with_pool
+from airflow_breeze.utils.parallel import (
+ DockerBuildxProgressMatcher,
+ ShowLastLineProgressMatcher,
+ check_async_run_results,
+ run_with_pool,
+)
from airflow_breeze.utils.path_utils import AIRFLOW_SOURCES_ROOT, DOCKER_CONTEXT_DIR
from airflow_breeze.utils.python_versions import get_python_version_list
from airflow_breeze.utils.registry import login_to_github_docker_registry
@@ -448,6 +453,45 @@ def pull_prod_image(
sys.exit(return_code)
+def run_verify_in_parallel(
+ image_params_list: list[BuildProdParams],
+ python_version_list: list[str],
+ extra_pytest_args: tuple[str, ...],
+ include_success_outputs: bool,
+ parallelism: int,
+ skip_cleanup: bool,
+ debug_resources: bool,
+) -> None:
+ with ci_group(f"Verifying PROD images for {python_version_list}"):
+ all_params = [f"PROD {image_params.python}" for image_params in image_params_list]
+ with run_with_pool(
+ parallelism=parallelism,
+ all_params=all_params,
+ debug_resources=debug_resources,
+ progress_matcher=ShowLastLineProgressMatcher(),
+ ) as (pool, outputs):
+ results = [
+ pool.apply_async(
+ verify_an_image,
+ kwds={
+ "image_name": image_params.airflow_image_name_with_tag,
+ "image_type": "PROD",
+ "slim_image": False,
+ "extra_pytest_args": extra_pytest_args,
+ "output": outputs[index],
+ },
+ )
+ for index, image_params in enumerate(image_params_list)
+ ]
+ check_async_run_results(
+ results=results,
+ success="All images verified",
+ outputs=outputs,
+ include_success_outputs=include_success_outputs,
+ skip_cleanup=skip_cleanup,
+ )
+
+
@prod_image.command(
name="verify",
context_settings=dict(
@@ -455,22 +499,29 @@ def pull_prod_image(
allow_extra_args=True,
),
)
-@option_python
-@option_image_tag_for_verifying
-@option_image_name
-@option_pull
@click.option(
"--slim-image",
help="The image to verify is slim and non-slim tests should be skipped.",
is_flag=True,
)
+@click.argument("extra_pytest_args", nargs=-1, type=click.UNPROCESSED)
+@option_python
+@option_python_versions
+@option_image_tag_for_verifying
+@option_image_name
+@option_pull
@option_github_repository
@option_github_token
+@option_run_in_parallel
+@option_parallelism
+@option_skip_cleanup
+@option_include_success_outputs
+@option_debug_resources
@option_verbose
@option_dry_run
-@click.argument("extra_pytest_args", nargs=-1, type=click.UNPROCESSED)
def verify(
python: str,
+ python_versions: str,
github_repository: str,
image_name: str,
image_tag: str | None,
@@ -478,6 +529,11 @@ def verify(
slim_image: bool,
github_token: str,
extra_pytest_args: tuple,
+ run_in_parallel: bool,
+ parallelism: int,
+ skip_cleanup: bool,
+ debug_resources: bool,
+ include_success_outputs: bool,
):
"""Verify Production image."""
perform_environment_checks()
@@ -485,27 +541,54 @@ def verify(
github_token=github_token,
output=None,
)
- if image_name is None:
- build_params = BuildProdParams(
+ if (pull or image_name) and run_in_parallel:
+ get_console().print(
+ "[error]You cannot use --pull,--image-name and --run-in-parallel at the same time. " "Exiting[/]"
+ )
+ sys.exit(1)
+ if run_in_parallel:
+ base_build_params = BuildProdParams(
python=python,
- image_tag=image_tag,
github_repository=github_repository,
- github_token=github_token,
+ image_tag=image_tag,
)
- image_name = build_params.airflow_image_name_with_tag
- if pull:
- check_remote_ghcr_io_commands()
- command_to_run = ["docker", "pull", image_name]
- run_command(command_to_run, check=True)
- get_console().print(f"[info]Verifying PROD image: {image_name}[/]")
- return_code, info = verify_an_image(
- image_name=image_name,
- output=None,
- image_type="PROD",
- extra_pytest_args=extra_pytest_args,
- slim_image=slim_image,
- )
- sys.exit(return_code)
+ python_version_list = get_python_version_list(python_versions)
+ params_list: list[BuildProdParams] = []
+ for python in python_version_list:
+ build_params = deepcopy(base_build_params)
+ build_params.python = python
+ params_list.append(build_params)
+ run_verify_in_parallel(
+ image_params_list=params_list,
+ python_version_list=python_version_list,
+ extra_pytest_args=extra_pytest_args,
+ include_success_outputs=include_success_outputs,
+ parallelism=parallelism,
+ skip_cleanup=skip_cleanup,
+ debug_resources=debug_resources,
+ )
+ else:
+ if image_name is None:
+ build_params = BuildProdParams(
+ python=python,
+ image_tag=image_tag,
+ github_repository=github_repository,
+ github_token=github_token,
+ )
+ image_name = build_params.airflow_image_name_with_tag
+ if pull:
+ check_remote_ghcr_io_commands()
+ command_to_run = ["docker", "pull", image_name]
+ run_command(command_to_run, check=True)
+ get_console().print(f"[info]Verifying PROD image: {image_name}[/]")
+ return_code, info = verify_an_image(
+ image_name=image_name,
+ output=None,
+ image_type="PROD",
+ extra_pytest_args=extra_pytest_args,
+ slim_image=slim_image,
+ )
+ sys.exit(return_code)
def clean_docker_context_files():
diff --git a/dev/breeze/src/airflow_breeze/commands/production_image_commands_config.py b/dev/breeze/src/airflow_breeze/commands/production_image_commands_config.py
index 52291268f8..9d20567537 100644
--- a/dev/breeze/src/airflow_breeze/commands/production_image_commands_config.py
+++ b/dev/breeze/src/airflow_breeze/commands/production_image_commands_config.py
@@ -157,6 +157,17 @@ PRODUCTION_IMAGE_TOOLS_PARAMETERS: dict[str, list[dict[str, str | list[str]]]] =
"--pull",
],
},
+ {
+ "name": "Parallel running",
+ "options": [
+ "--run-in-parallel",
+ "--parallelism",
+ "--python-versions",
+ "--skip-cleanup",
+ "--debug-resources",
+ "--include-success-outputs",
+ ],
+ },
{
"name": "Github authentication",
"options": [
diff --git a/dev/breeze/src/airflow_breeze/utils/run_tests.py b/dev/breeze/src/airflow_breeze/utils/run_tests.py
index d2ce641fe4..c3fd500f08 100644
--- a/dev/breeze/src/airflow_breeze/utils/run_tests.py
+++ b/dev/breeze/src/airflow_breeze/utils/run_tests.py
@@ -33,7 +33,7 @@ def verify_an_image(
image_type: str,
output: Output | None,
slim_image: bool,
- extra_pytest_args: tuple,
+ extra_pytest_args: tuple[str, ...],
) -> tuple[int, str]:
command_result = run_command(
["docker", "inspect", image_name],
diff --git a/docker_tests/test_ci_image.py b/docker_tests/test_ci_image.py
index 2c3fabd7d9..7ec65c425c 100644
--- a/docker_tests/test_ci_image.py
+++ b/docker_tests/test_ci_image.py
@@ -22,12 +22,28 @@ from docker_tests.command_utils import run_command
from docker_tests.docker_tests_utils import display_dependency_conflict_message, docker_image
-class TestPythonPackages:
- def test_pip_dependencies_conflict(self):
- try:
- run_command(
- ["docker", "run", "--rm", "--entrypoint", "/bin/bash", docker_image, "-c", "pip check"]
- )
- except subprocess.CalledProcessError as ex:
- display_dependency_conflict_message()
- raise ex
+def test_pip_dependencies_conflict():
+ try:
+ run_command(["docker", "run", "--rm", "--entrypoint", "/bin/bash", docker_image, "-c", "pip check"])
+ except subprocess.CalledProcessError as ex:
+ display_dependency_conflict_message()
+ raise ex
+
+
+def test_providers_present():
+ try:
+ run_command(
+ [
+ "docker",
+ "run",
+ "--rm",
+ "--entrypoint",
+ "/bin/bash",
+ docker_image,
+ "-c",
+ "airflow providers list",
+ ],
+ )
+ except subprocess.CalledProcessError as ex:
+ display_dependency_conflict_message()
+ raise ex
diff --git a/images/breeze/output_ci-image_verify.svg b/images/breeze/output_ci-image_verify.svg
index 42b327ef9c..cb247415e9 100644
--- a/images/breeze/output_ci-image_verify.svg
+++ b/images/breeze/output_ci-image_verify.svg
@@ -1,4 +1,4 @@
-<svg class="rich-terminal" viewBox="0 0 1482 562.4" xmlns="http://www.w3.org/2000/svg">
+<svg class="rich-terminal" viewBox="0 0 1482 830.8" xmlns="http://www.w3.org/2000/svg">
<!-- Generated with Rich https://www.textualize.io -->
<style>
@@ -43,7 +43,7 @@
<defs>
<clipPath id="breeze-ci-image-verify-clip-terminal">
- <rect x="0" y="0" width="1463.0" height="511.4" />
+ <rect x="0" y="0" width="1463.0" height="779.8" />
</clipPath>
<clipPath id="breeze-ci-image-verify-line-0">
<rect x="0" y="1.5" width="1464" height="24.65"/>
@@ -105,9 +105,42 @@
<clipPath id="breeze-ci-image-verify-line-19">
<rect x="0" y="465.1" width="1464" height="24.65"/>
</clipPath>
+<clipPath id="breeze-ci-image-verify-line-20">
+ <rect x="0" y="489.5" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-ci-image-verify-line-21">
+ <rect x="0" y="513.9" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-ci-image-verify-line-22">
+ <rect x="0" y="538.3" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-ci-image-verify-line-23">
+ <rect x="0" y="562.7" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-ci-image-verify-line-24">
+ <rect x="0" y="587.1" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-ci-image-verify-line-25">
+ <rect x="0" y="611.5" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-ci-image-verify-line-26">
+ <rect x="0" y="635.9" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-ci-image-verify-line-27">
+ <rect x="0" y="660.3" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-ci-image-verify-line-28">
+ <rect x="0" y="684.7" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-ci-image-verify-line-29">
+ <rect x="0" y="709.1" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-ci-image-verify-line-30">
+ <rect x="0" y="733.5" width="1464" height="24.65"/>
+ </clipPath>
</defs>
- <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="560.4" rx="8"/><text class="breeze-ci-image-verify-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: ci-image verify</text>
+ <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="828.8" rx="8"/><text class="breeze-ci-image-verify-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: ci-image verify</text>
<g transform="translate(26,22)">
<circle cx="0" cy="0" r="7" fill="#ff5f57"/>
<circle cx="22" cy="0" r="7" fill="#febc2e"/>
@@ -129,15 +162,26 @@
</text><text class="breeze-ci-image-verify-r5" x="0" y="239.6" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-9)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="239.6" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-9)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="239.6" textLength="73.2" clip-path="url(#breeze-ci-image-verify-line-9)">-image</text><text class="breeze-ci-image-verify-r4" x="109.8" y="239.6" textLength="48.8" clip-p [...]
</text><text class="breeze-ci-image-verify-r5" x="0" y="264" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-10)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="264" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-10)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="264" textLength="61" clip-path="url(#breeze-ci-image-verify-line-10)">-pull</text><text class="breeze-ci-image-verify-r1" x="244" y="264" textLength="646.6" clip-path="url( [...]
</text><text class="breeze-ci-image-verify-r5" x="0" y="288.4" textLength="1464" clip-path="url(#breeze-ci-image-verify-line-11)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-verify-r1" x="1464" y="288.4" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-11)">
-</text><text class="breeze-ci-image-verify-r5" x="0" y="312.8" textLength="24.4" clip-path="url(#breeze-ci-image-verify-line-12)">╭─</text><text class="breeze-ci-image-verify-r5" x="24.4" y="312.8" textLength="280.6" clip-path="url(#breeze-ci-image-verify-line-12)"> Github authentication </text><text class="breeze-ci-image-verify-r5" x="305" y="312.8" textLength="1134.6" clip-path="url(#breeze-ci-image-verify-line-12)">────────────────────────────────────────────────────── [...]
-</text><text class="breeze-ci-image-verify-r5" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-13)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="337.2" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-13)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="337.2" textLength="85.4" clip-path="url(#breeze-ci-image-verify-line-13)">-github</text><text class="breeze-ci-image-verify-r4" x="122" y="337.2" textLength="134.2" cli [...]
-</text><text class="breeze-ci-image-verify-r5" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-14)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="361.6" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-14)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="361.6" textLength="85.4" clip-path="url(#breeze-ci-image-verify-line-14)">-github</text><text class="breeze-ci-image-verify-r4" x="122" y="361.6" textLength="73.2" clip [...]
-</text><text class="breeze-ci-image-verify-r5" x="0" y="386" textLength="1464" clip-path="url(#breeze-ci-image-verify-line-15)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-verify-r1" x="1464" y="386" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-15)">
-</text><text class="breeze-ci-image-verify-r5" x="0" y="410.4" textLength="24.4" clip-path="url(#breeze-ci-image-verify-line-16)">╭─</text><text class="breeze-ci-image-verify-r5" x="24.4" y="410.4" textLength="195.2" clip-path="url(#breeze-ci-image-verify-line-16)"> Common options </text><text class="breeze-ci-image-verify-r5" x="219.6" y="410.4" textLength="1220" clip-path="url(#breeze-ci-image-verify-line-16)">───────────────────────────────────────────────────────────── [...]
-</text><text class="breeze-ci-image-verify-r5" x="0" y="434.8" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-17)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="434.8" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-17)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="434.8" textLength="97.6" clip-path="url(#breeze-ci-image-verify-line-17)">-verbose</text><text class="breeze-ci-image-verify-r6" x="158.6" y="434.8" textLength="24.4" c [...]
-</text><text class="breeze-ci-image-verify-r5" x="0" y="459.2" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-18)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="459.2" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-18)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="459.2" textLength="48.8" clip-path="url(#breeze-ci-image-verify-line-18)">-dry</text><text class="breeze-ci-image-verify-r4" x="85.4" y="459.2" textLength="48.8" clip-p [...]
-</text><text class="breeze-ci-image-verify-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-19)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="483.6" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-19)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="483.6" textLength="61" clip-path="url(#breeze-ci-image-verify-line-19)">-help</text><text class="breeze-ci-image-verify-r6" x="158.6" y="483.6" textLength="24.4" clip-p [...]
-</text><text class="breeze-ci-image-verify-r5" x="0" y="508" textLength="1464" clip-path="url(#breeze-ci-image-verify-line-20)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-verify-r1" x="1464" y="508" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-20)">
+</text><text class="breeze-ci-image-verify-r5" x="0" y="312.8" textLength="24.4" clip-path="url(#breeze-ci-image-verify-line-12)">╭─</text><text class="breeze-ci-image-verify-r5" x="24.4" y="312.8" textLength="219.6" clip-path="url(#breeze-ci-image-verify-line-12)"> Parallel running </text><text class="breeze-ci-image-verify-r5" x="244" y="312.8" textLength="1195.6" clip-path="url(#breeze-ci-image-verify-line-12)">─────────────────────────────────────────────────────────── [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-13)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="337.2" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-13)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="337.2" textLength="48.8" clip-path="url(#breeze-ci-image-verify-line-13)">-run</text><text class="breeze-ci-image-verify-r4" x="85.4" y="337.2" textLength="146.4" clip- [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-14)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="361.6" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-14)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="361.6" textLength="146.4" clip-path="url(#breeze-ci-image-verify-line-14)">-parallelism</text><text class="breeze-ci-image-verify-r1" x="378.2" y="361.6" textLength="91 [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="386" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-15)">│</text><text class="breeze-ci-image-verify-r7" x="378.2" y="386" textLength="915" clip-path="url(#breeze-ci-image-verify-line-15)">(INTEGER RANGE)                                   &# [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="410.4" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-16)">│</text><text class="breeze-ci-image-verify-r5" x="378.2" y="410.4" textLength="915" clip-path="url(#breeze-ci-image-verify-line-16)">[default: 4; 1<=x<=8]                                 [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="434.8" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-17)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="434.8" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-17)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="434.8" textLength="85.4" clip-path="url(#breeze-ci-image-verify-line-17)">-python</text><text class="breeze-ci-image-verify-r4" x="122" y="434.8" textLength="109.8" cli [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="459.2" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-18)">│</text><text class="breeze-ci-image-verify-r5" x="378.2" y="459.2" textLength="951.6" clip-path="url(#breeze-ci-image-verify-line-18)">[default: 3.8 3.9 3.10 3.11]                              [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-19)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="483.6" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-19)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="483.6" textLength="61" clip-path="url(#breeze-ci-image-verify-line-19)">-skip</text><text class="breeze-ci-image-verify-r4" x="97.6" y="483.6" textLength="97.6" clip-pa [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="508" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-20)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="508" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-20)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="508" textLength="73.2" clip-path="url(#breeze-ci-image-verify-line-20)">-debug</text><text class="breeze-ci-image-verify-r4" x="109.8" y="508" textLength="122" clip-path="u [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-21)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="532.4" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-21)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="532.4" textLength="97.6" clip-path="url(#breeze-ci-image-verify-line-21)">-include</text><text class="breeze-ci-image-verify-r4" x="134.2" y="532.4" textLength="195.2" [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="556.8" textLength="1464" clip-path="url(#breeze-ci-image-verify-line-22)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-verify-r1" x="1464" y="556.8" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-22)">
+</text><text class="breeze-ci-image-verify-r5" x="0" y="581.2" textLength="24.4" clip-path="url(#breeze-ci-image-verify-line-23)">╭─</text><text class="breeze-ci-image-verify-r5" x="24.4" y="581.2" textLength="280.6" clip-path="url(#breeze-ci-image-verify-line-23)"> Github authentication </text><text class="breeze-ci-image-verify-r5" x="305" y="581.2" textLength="1134.6" clip-path="url(#breeze-ci-image-verify-line-23)">────────────────────────────────────────────────────── [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="605.6" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-24)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="605.6" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-24)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="605.6" textLength="85.4" clip-path="url(#breeze-ci-image-verify-line-24)">-github</text><text class="breeze-ci-image-verify-r4" x="122" y="605.6" textLength="134.2" cli [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="630" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-25)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="630" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-25)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="630" textLength="85.4" clip-path="url(#breeze-ci-image-verify-line-25)">-github</text><text class="breeze-ci-image-verify-r4" x="122" y="630" textLength="73.2" clip-path="u [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="654.4" textLength="1464" clip-path="url(#breeze-ci-image-verify-line-26)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-verify-r1" x="1464" y="654.4" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-26)">
+</text><text class="breeze-ci-image-verify-r5" x="0" y="678.8" textLength="24.4" clip-path="url(#breeze-ci-image-verify-line-27)">╭─</text><text class="breeze-ci-image-verify-r5" x="24.4" y="678.8" textLength="195.2" clip-path="url(#breeze-ci-image-verify-line-27)"> Common options </text><text class="breeze-ci-image-verify-r5" x="219.6" y="678.8" textLength="1220" clip-path="url(#breeze-ci-image-verify-line-27)">───────────────────────────────────────────────────────────── [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="703.2" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-28)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="703.2" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-28)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="703.2" textLength="97.6" clip-path="url(#breeze-ci-image-verify-line-28)">-verbose</text><text class="breeze-ci-image-verify-r6" x="158.6" y="703.2" textLength="24.4" c [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="727.6" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-29)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="727.6" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-29)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="727.6" textLength="48.8" clip-path="url(#breeze-ci-image-verify-line-29)">-dry</text><text class="breeze-ci-image-verify-r4" x="85.4" y="727.6" textLength="48.8" clip-p [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="752" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-30)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="752" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-30)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="752" textLength="61" clip-path="url(#breeze-ci-image-verify-line-30)">-help</text><text class="breeze-ci-image-verify-r6" x="158.6" y="752" textLength="24.4" clip-path="url [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="776.4" textLength="1464" clip-path="url(#breeze-ci-image-verify-line-31)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-verify-r1" x="1464" y="776.4" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-31)">
</text>
</g>
</g>
diff --git a/images/breeze/output_ci-image_verify.txt b/images/breeze/output_ci-image_verify.txt
index 6b16a49525..f454443f68 100644
--- a/images/breeze/output_ci-image_verify.txt
+++ b/images/breeze/output_ci-image_verify.txt
@@ -1 +1 @@
-c90dc7e20fce2351eb89d8d1ebbd35e7
+707a149f99bd49d37be5b8d0db844d69
diff --git a/images/breeze/output_prod-image_verify.svg b/images/breeze/output_prod-image_verify.svg
index 508fcbd34c..5633e0226c 100644
--- a/images/breeze/output_prod-image_verify.svg
+++ b/images/breeze/output_prod-image_verify.svg
@@ -1,4 +1,4 @@
-<svg class="rich-terminal" viewBox="0 0 1482 586.8" xmlns="http://www.w3.org/2000/svg">
+<svg class="rich-terminal" viewBox="0 0 1482 855.1999999999999" xmlns="http://www.w3.org/2000/svg">
<!-- Generated with Rich https://www.textualize.io -->
<style>
@@ -43,7 +43,7 @@
<defs>
<clipPath id="breeze-prod-image-verify-clip-terminal">
- <rect x="0" y="0" width="1463.0" height="535.8" />
+ <rect x="0" y="0" width="1463.0" height="804.1999999999999" />
</clipPath>
<clipPath id="breeze-prod-image-verify-line-0">
<rect x="0" y="1.5" width="1464" height="24.65"/>
@@ -108,9 +108,42 @@
<clipPath id="breeze-prod-image-verify-line-20">
<rect x="0" y="489.5" width="1464" height="24.65"/>
</clipPath>
+<clipPath id="breeze-prod-image-verify-line-21">
+ <rect x="0" y="513.9" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-prod-image-verify-line-22">
+ <rect x="0" y="538.3" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-prod-image-verify-line-23">
+ <rect x="0" y="562.7" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-prod-image-verify-line-24">
+ <rect x="0" y="587.1" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-prod-image-verify-line-25">
+ <rect x="0" y="611.5" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-prod-image-verify-line-26">
+ <rect x="0" y="635.9" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-prod-image-verify-line-27">
+ <rect x="0" y="660.3" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-prod-image-verify-line-28">
+ <rect x="0" y="684.7" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-prod-image-verify-line-29">
+ <rect x="0" y="709.1" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-prod-image-verify-line-30">
+ <rect x="0" y="733.5" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-prod-image-verify-line-31">
+ <rect x="0" y="757.9" width="1464" height="24.65"/>
+ </clipPath>
</defs>
- <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="584.8" rx="8"/><text class="breeze-prod-image-verify-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: prod-image verify</text>
+ <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="853.2" rx="8"/><text class="breeze-prod-image-verify-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: prod-image verify</text>
<g transform="translate(26,22)">
<circle cx="0" cy="0" r="7" fill="#ff5f57"/>
<circle cx="22" cy="0" r="7" fill="#febc2e"/>
@@ -133,15 +166,26 @@
</text><text class="breeze-prod-image-verify-r5" x="0" y="264" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-10)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="264" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-10)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="264" textLength="73.2" clip-path="url(#breeze-prod-image-verify-line-10)">-image</text><text class="breeze-prod-image-verify-r4" x="109.8" y="264" textLength="48. [...]
</text><text class="breeze-prod-image-verify-r5" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-11)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="288.4" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-11)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="288.4" textLength="61" clip-path="url(#breeze-prod-image-verify-line-11)">-pull</text><text class="breeze-prod-image-verify-r1" x="244" y="288.4" textLength=" [...]
</text><text class="breeze-prod-image-verify-r5" x="0" y="312.8" textLength="1464" clip-path="url(#breeze-prod-image-verify-line-12)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-verify-r1" x="1464" y="312.8" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-12)">
-</text><text class="breeze-prod-image-verify-r5" x="0" y="337.2" textLength="24.4" clip-path="url(#breeze-prod-image-verify-line-13)">╭─</text><text class="breeze-prod-image-verify-r5" x="24.4" y="337.2" textLength="280.6" clip-path="url(#breeze-prod-image-verify-line-13)"> Github authentication </text><text class="breeze-prod-image-verify-r5" x="305" y="337.2" textLength="1134.6" clip-path="url(#breeze-prod-image-verify-line-13)">────────────────────────────────────────── [...]
-</text><text class="breeze-prod-image-verify-r5" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-14)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="361.6" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-14)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="361.6" textLength="85.4" clip-path="url(#breeze-prod-image-verify-line-14)">-github</text><text class="breeze-prod-image-verify-r4" x="122" y="361.6" textLeng [...]
-</text><text class="breeze-prod-image-verify-r5" x="0" y="386" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-15)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="386" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-15)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="386" textLength="85.4" clip-path="url(#breeze-prod-image-verify-line-15)">-github</text><text class="breeze-prod-image-verify-r4" x="122" y="386" textLength="73.2 [...]
-</text><text class="breeze-prod-image-verify-r5" x="0" y="410.4" textLength="1464" clip-path="url(#breeze-prod-image-verify-line-16)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-verify-r1" x="1464" y="410.4" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-16)">
-</text><text class="breeze-prod-image-verify-r5" x="0" y="434.8" textLength="24.4" clip-path="url(#breeze-prod-image-verify-line-17)">╭─</text><text class="breeze-prod-image-verify-r5" x="24.4" y="434.8" textLength="195.2" clip-path="url(#breeze-prod-image-verify-line-17)"> Common options </text><text class="breeze-prod-image-verify-r5" x="219.6" y="434.8" textLength="1220" clip-path="url(#breeze-prod-image-verify-line-17)">───────────────────────────────────────────────── [...]
-</text><text class="breeze-prod-image-verify-r5" x="0" y="459.2" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-18)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="459.2" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-18)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="459.2" textLength="97.6" clip-path="url(#breeze-prod-image-verify-line-18)">-verbose</text><text class="breeze-prod-image-verify-r6" x="158.6" y="459.2" textL [...]
-</text><text class="breeze-prod-image-verify-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-19)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="483.6" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-19)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="483.6" textLength="48.8" clip-path="url(#breeze-prod-image-verify-line-19)">-dry</text><text class="breeze-prod-image-verify-r4" x="85.4" y="483.6" textLength [...]
-</text><text class="breeze-prod-image-verify-r5" x="0" y="508" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-20)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="508" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-20)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="508" textLength="61" clip-path="url(#breeze-prod-image-verify-line-20)">-help</text><text class="breeze-prod-image-verify-r6" x="158.6" y="508" textLength="24.4" [...]
-</text><text class="breeze-prod-image-verify-r5" x="0" y="532.4" textLength="1464" clip-path="url(#breeze-prod-image-verify-line-21)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-verify-r1" x="1464" y="532.4" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-21)">
+</text><text class="breeze-prod-image-verify-r5" x="0" y="337.2" textLength="24.4" clip-path="url(#breeze-prod-image-verify-line-13)">╭─</text><text class="breeze-prod-image-verify-r5" x="24.4" y="337.2" textLength="219.6" clip-path="url(#breeze-prod-image-verify-line-13)"> Parallel running </text><text class="breeze-prod-image-verify-r5" x="244" y="337.2" textLength="1195.6" clip-path="url(#breeze-prod-image-verify-line-13)">─────────────────────────────────────────────── [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-14)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="361.6" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-14)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="361.6" textLength="48.8" clip-path="url(#breeze-prod-image-verify-line-14)">-run</text><text class="breeze-prod-image-verify-r4" x="85.4" y="361.6" textLength [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="386" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-15)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="386" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-15)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="386" textLength="146.4" clip-path="url(#breeze-prod-image-verify-line-15)">-parallelism</text><text class="breeze-prod-image-verify-r1" x="378.2" y="386" textLeng [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="410.4" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-16)">│</text><text class="breeze-prod-image-verify-r7" x="378.2" y="410.4" textLength="915" clip-path="url(#breeze-prod-image-verify-line-16)">(INTEGER RANGE)                                 &# [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="434.8" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-17)">│</text><text class="breeze-prod-image-verify-r5" x="378.2" y="434.8" textLength="915" clip-path="url(#breeze-prod-image-verify-line-17)">[default: 4; 1<=x<=8]                               [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="459.2" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-18)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="459.2" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-18)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="459.2" textLength="85.4" clip-path="url(#breeze-prod-image-verify-line-18)">-python</text><text class="breeze-prod-image-verify-r4" x="122" y="459.2" textLeng [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-19)">│</text><text class="breeze-prod-image-verify-r5" x="378.2" y="483.6" textLength="951.6" clip-path="url(#breeze-prod-image-verify-line-19)">[default: 3.8 3.9 3.10 3.11]                            &# [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="508" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-20)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="508" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-20)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="508" textLength="61" clip-path="url(#breeze-prod-image-verify-line-20)">-skip</text><text class="breeze-prod-image-verify-r4" x="97.6" y="508" textLength="97.6" c [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-21)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="532.4" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-21)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="532.4" textLength="73.2" clip-path="url(#breeze-prod-image-verify-line-21)">-debug</text><text class="breeze-prod-image-verify-r4" x="109.8" y="532.4" textLen [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="556.8" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-22)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="556.8" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-22)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="556.8" textLength="97.6" clip-path="url(#breeze-prod-image-verify-line-22)">-include</text><text class="breeze-prod-image-verify-r4" x="134.2" y="556.8" textL [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="581.2" textLength="1464" clip-path="url(#breeze-prod-image-verify-line-23)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-verify-r1" x="1464" y="581.2" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-23)">
+</text><text class="breeze-prod-image-verify-r5" x="0" y="605.6" textLength="24.4" clip-path="url(#breeze-prod-image-verify-line-24)">╭─</text><text class="breeze-prod-image-verify-r5" x="24.4" y="605.6" textLength="280.6" clip-path="url(#breeze-prod-image-verify-line-24)"> Github authentication </text><text class="breeze-prod-image-verify-r5" x="305" y="605.6" textLength="1134.6" clip-path="url(#breeze-prod-image-verify-line-24)">────────────────────────────────────────── [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="630" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-25)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="630" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-25)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="630" textLength="85.4" clip-path="url(#breeze-prod-image-verify-line-25)">-github</text><text class="breeze-prod-image-verify-r4" x="122" y="630" textLength="134. [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="654.4" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-26)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="654.4" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-26)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="654.4" textLength="85.4" clip-path="url(#breeze-prod-image-verify-line-26)">-github</text><text class="breeze-prod-image-verify-r4" x="122" y="654.4" textLeng [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="678.8" textLength="1464" clip-path="url(#breeze-prod-image-verify-line-27)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-verify-r1" x="1464" y="678.8" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-27)">
+</text><text class="breeze-prod-image-verify-r5" x="0" y="703.2" textLength="24.4" clip-path="url(#breeze-prod-image-verify-line-28)">╭─</text><text class="breeze-prod-image-verify-r5" x="24.4" y="703.2" textLength="195.2" clip-path="url(#breeze-prod-image-verify-line-28)"> Common options </text><text class="breeze-prod-image-verify-r5" x="219.6" y="703.2" textLength="1220" clip-path="url(#breeze-prod-image-verify-line-28)">───────────────────────────────────────────────── [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="727.6" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-29)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="727.6" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-29)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="727.6" textLength="97.6" clip-path="url(#breeze-prod-image-verify-line-29)">-verbose</text><text class="breeze-prod-image-verify-r6" x="158.6" y="727.6" textL [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="752" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-30)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="752" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-30)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="752" textLength="48.8" clip-path="url(#breeze-prod-image-verify-line-30)">-dry</text><text class="breeze-prod-image-verify-r4" x="85.4" y="752" textLength="48.8" [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="776.4" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-31)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="776.4" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-31)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="776.4" textLength="61" clip-path="url(#breeze-prod-image-verify-line-31)">-help</text><text class="breeze-prod-image-verify-r6" x="158.6" y="776.4" textLength [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="800.8" textLength="1464" clip-path="url(#breeze-prod-image-verify-line-32)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-verify-r1" x="1464" y="800.8" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-32)">
</text>
</g>
</g>
diff --git a/images/breeze/output_prod-image_verify.txt b/images/breeze/output_prod-image_verify.txt
index 2b87281a7f..8301a9d978 100644
--- a/images/breeze/output_prod-image_verify.txt
+++ b/images/breeze/output_prod-image_verify.txt
@@ -1 +1 @@
-bd2b78738a7c388dbad6076c41a9f906
+3d7fdd3877862ce9bfad8018a5282745