You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2023/11/25 23:37:51 UTC
(airflow) 01/01: Optimize CI/PROD image waiting and verification in CI workflow
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch optimize-image-wait-verify
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 0d0a5a768835b99fd7887dc53637183d9b256de0
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sat Nov 25 14:05:23 2023 +0100
Optimize CI/PROD image waiting and verification in CI workflow
Currently both "wait-for-ci-images" and "preview-constraints"
jobs are waiting for images to be built - which means that they
both take a running worker slot (public runner) just to do
the waiting while the image is being built. Also "verify-image"
job is run as part of "wait-for-image" which adds additional
delay between being downloaded and dependent jobs starting.
This PR optimizes it quite a bit:
* preview-constraints job now depends on "wait-for-ci-images".
This means that only one slot will be busy while waiting
for images.
* both CI and PROD `verify-image` commands in breeze got
--run-in-parallel set of flags that allow the verification
to happen for all images in parallel.
* there are separate jobs started that verifies the images
(both for CI and PR images) - this way the job that waits
for images can finish a but earlier and allow the dependent
jobs to start in parallell to verifying the image.
* In case of the "in-workflow-build" the "wait-for-ci-images"
does not have to be run at all, because there wait-for-ci-images
depends on in-workflow build-ci-images job - so if that job
completes, we know image is built already and we do not have to
wait for it separately - so far we had to run it in order to
add `--verify` flag to verify the images. With separate job we
can run i in parallel to all the other waiting jobs.
* Also names and dependencies between jobs are updated, including
CI documentation describing diagrams of how CI workflows work.
The diagrams are cleaned-up/verified and updated. The separate
diagram for scheduled build has been removed as it was essentially
the same as "canary build". A paragraph description for every
type of workflow was added to add more context to the diagrams.
---
.github/workflows/ci.yml | 378 +++++++++++---------
CI.rst | 60 ++--
CI_DIAGRAMS.md | 383 +++++++++++----------
.../airflow_breeze/commands/ci_image_commands.py | 123 +++++--
.../commands/ci_image_commands_config.py | 11 +
.../commands/production_image_commands.py | 131 +++++--
.../commands/production_image_commands_config.py | 11 +
dev/breeze/src/airflow_breeze/utils/run_tests.py | 2 +-
docker_tests/test_ci_image.py | 34 +-
images/breeze/output_ci-image_verify.svg | 68 +++-
images/breeze/output_ci-image_verify.txt | 2 +-
images/breeze/output_prod-image_verify.svg | 68 +++-
images/breeze/output_prod-image_verify.txt | 2 +-
13 files changed, 817 insertions(+), 456 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 11256ab1de..01bcdc6bab 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -44,7 +44,7 @@ env:
CONSTRAINTS_GITHUB_REPOSITORY: >-
${{ secrets.CONSTRAINTS_GITHUB_REPOSITORY != '' &&
secrets.CONSTRAINTS_GITHUB_REPOSITORY || 'apache/airflow' }}
- # In builds from forks, this token is read-only. For scheduler/direct push it is WRITE one
+ # In builds from forks, this token is read-only. For scheduled/direct push it is WRITE one
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
IMAGE_TAG: "${{ github.event.pull_request.head.sha || github.sha }}"
USE_SUDO: "true"
@@ -320,148 +320,12 @@ jobs:
PYTHON_VERSIONS: ${{needs.build-info.outputs.all-python-versions-list-as-string}}
DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
BUILD_TIMEOUT_MINUTES: 70
-
- generate-constraints:
- permissions:
- contents: read
- timeout-minutes: 70
- name: >
- Preview constraints
- ${{needs.build-info.outputs.all-python-versions-list-as-string}}
- runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
- needs: [build-info, build-ci-images]
- env:
- RUNS_ON: "${{ needs.build-info.outputs.runs-on }}"
- PYTHON_VERSIONS: ${{needs.build-info.outputs.all-python-versions-list-as-string}}
- DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
- if: needs.build-info.outputs.ci-image-build == 'true'
- steps:
- - name: Cleanup repo
- run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
- - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
- uses: actions/checkout@v4
- with:
- persist-credentials: false
- - name: "Install Breeze"
- uses: ./.github/actions/breeze
- - name: Pull CI images ${{ env.PYTHON_VERSIONS }}:${{ env.IMAGE_TAG }}
- run: breeze ci-image pull --run-in-parallel --tag-as-latest --wait-for-image
- - name: "Source constraints"
- shell: bash
- run: >
- breeze release-management generate-constraints --run-in-parallel
- --airflow-constraints-mode constraints-source-providers
- - name: "No providers constraints"
- shell: bash
- timeout-minutes: 25
- run: >
- breeze release-management generate-constraints --run-in-parallel
- --airflow-constraints-mode constraints-no-providers
- - name: "PyPI constraints"
- shell: bash
- timeout-minutes: 25
- run: >
- breeze release-management generate-constraints --run-in-parallel
- --airflow-constraints-mode constraints
- - name: "Dependency upgrade summary"
- shell: bash
- run: |
- for PYTHON_VERSION in ${{ env.PYTHON_VERSIONS }}; do
- echo "Summarizing Python $PYTHON_VERSION"
- cat "files/constraints-${PYTHON_VERSION}"/*.md >> $GITHUB_STEP_SUMMARY || true
- done
- - name: "Upload constraint artifacts"
- uses: actions/upload-artifact@v3
- with:
- name: constraints
- path: ./files/constraints-*/constraints-*.txt
- retention-days: 7
-
- build-prod-images:
- timeout-minutes: 80
- name: >
- ${{needs.build-info.outputs.build-job-description}} PROD images
- ${{needs.build-info.outputs.all-python-versions-list-as-string}}
- runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
- needs: [build-info, build-ci-images]
- env:
- DEFAULT_BRANCH: ${{ needs.build-info.outputs.default-branch }}
- DEFAULT_CONSTRAINTS_BRANCH: ${{ needs.build-info.outputs.default-constraints-branch }}
- RUNS_ON: "${{needs.build-info.outputs.runs-on}}"
- BACKEND: sqlite
- DOCKER_CACHE: ${{ needs.build-info.outputs.cache-directive }}
- VERSION_SUFFIX_FOR_PYPI: "dev0"
- DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
- # Force more parallelism for build even on public images
- PARALLELISM: 6
- steps:
- - name: Cleanup repo
- run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
- if: needs.build-info.outputs.in-workflow-build == 'true'
- - uses: actions/checkout@v4
- with:
- ref: ${{ needs.build-info.outputs.targetCommitSha }}
- persist-credentials: false
- if: needs.build-info.outputs.in-workflow-build == 'true'
- - name: "Install Breeze"
- uses: ./.github/actions/breeze
- if: needs.build-info.outputs.in-workflow-build == 'true'
- - name: >
- Build PROD Images
- ${{needs.build-info.outputs.all-python-versions-list-as-string}}:${{env.IMAGE_TAG}}
- uses: ./.github/actions/build-prod-images
- if: needs.build-info.outputs.in-workflow-build == 'true'
- with:
- build-provider-packages: ${{ needs.build-info.outputs.default-branch == 'main' }}
- env:
- UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgrade-to-newer-dependencies }}
- DOCKER_CACHE: ${{ needs.build-info.outputs.cache-directive }}
- PYTHON_VERSIONS: ${{needs.build-info.outputs.all-python-versions-list-as-string}}
- DEBUG_RESOURCES: ${{ needs.build-info.outputs.debug-resources }}
-
- build-prod-images-bullseye:
- timeout-minutes: 80
- name: >
- ${{needs.build-info.outputs.build-job-description}} Bullseye PROD images
- ${{needs.build-info.outputs.all-python-versions-list-as-string}}
- runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
- needs: [build-info, build-ci-images]
- if: needs.build-info.outputs.canary-run == 'true'
- env:
- DEFAULT_BRANCH: ${{ needs.build-info.outputs.default-branch }}
- DEFAULT_CONSTRAINTS_BRANCH: ${{ needs.build-info.outputs.default-constraints-branch }}
- RUNS_ON: "${{needs.build-info.outputs.runs-on}}"
- BACKEND: sqlite
- DOCKER_CACHE: ${{ needs.build-info.outputs.cache-directive }}
- VERSION_SUFFIX_FOR_PYPI: "dev0"
- DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
- # Force more parallelism for build even on public images
- PARALLELISM: 6
- steps:
- - name: Cleanup repo
- run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
- - uses: actions/checkout@v3
- with:
- ref: ${{ needs.build-info.outputs.targetCommitSha }}
- persist-credentials: false
- submodules: recursive
- - name: "Install Breeze"
- uses: ./.github/actions/breeze
- - name: >
- Build Bullseye PROD Images
- ${{needs.build-info.outputs.all-python-versions-list-as-string}}:${{env.IMAGE_TAG}}
- uses: ./.github/actions/build-prod-images
- with:
- build-provider-packages: ${{ needs.build-info.outputs.default-branch == 'main' }}
+ - name: Verify CI images ${{ env.PYTHON_VERSIONS }}:${{ env.IMAGE_TAG }}
+ run: breeze ci-image verify --run-in-parallel
env:
- UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgrade-to-newer-dependencies }}
- DOCKER_CACHE: ${{ needs.build-info.outputs.cache-directive }}
PYTHON_VERSIONS: ${{needs.build-info.outputs.all-python-versions-list-as-string}}
- DEBUG_RESOURCES: ${{ needs.build-info.outputs.debug-resources }}
- DEBIAN_VERSION: "bullseye"
- # Do not override the "bookworm" image - just push a new bullseye image
- # TODO: improve caching for that build
- IMAGE_TAG: "bullseye-${{ github.event.pull_request.head.sha || github.sha }}"
+ DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
+ if: needs.build-info.outputs.in-workflow-build == 'true'
run-breeze-tests:
timeout-minutes: 10
@@ -565,6 +429,7 @@ jobs:
with:
fetch-depth: 2
persist-credentials: false
+
wait-for-ci-images:
timeout-minutes: 120
name: "Wait for CI images"
@@ -579,18 +444,107 @@ jobs:
steps:
- name: Cleanup repo
run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
+ if: needs.build-info.outputs.in-workflow-build == 'false'
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v4
with:
persist-credentials: false
+ if: needs.build-info.outputs.in-workflow-build == 'false'
- name: "Install Breeze"
uses: ./.github/actions/breeze
+ if: needs.build-info.outputs.in-workflow-build == 'false'
- name: Wait for CI images ${{ env.PYTHON_VERSIONS }}:${{ env.IMAGE_TAG }}
id: wait-for-images
- run: breeze ci-image pull --run-in-parallel --verify --wait-for-image --tag-as-latest
+ run: breeze ci-image pull --run-in-parallel --wait-for-image --tag-as-latest
env:
PYTHON_VERSIONS: ${{ needs.build-info.outputs.python-versions-list-as-string }}
DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
+ if: needs.build-info.outputs.in-workflow-build == 'false'
+
+ verify-ci-images:
+ timeout-minutes: 80
+ name: Verify CI images ${{needs.build-info.outputs.all-python-versions-list-as-string}}
+ runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
+ needs: [build-info, wait-for-ci-images]
+ env:
+ DEFAULT_BRANCH: ${{ needs.build-info.outputs.default-branch }}
+ DEFAULT_CONSTRAINTS_BRANCH: ${{ needs.build-info.outputs.default-constraints-branch }}
+ RUNS_ON: "${{needs.build-info.outputs.runs-on}}"
+ VERSION_SUFFIX_FOR_PYPI: "dev0"
+ steps:
+ - name: Cleanup repo
+ run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
+ - uses: actions/checkout@v4
+ with:
+ ref: ${{ needs.build-info.outputs.targetCommitSha }}
+ persist-credentials: false
+ - name: Pull CI images ${{ env.PYTHON_VERSIONS }}:${{ env.IMAGE_TAG }}
+ run: breeze ci-image pull --run-in-parallel --tag-as-latest
+ env:
+ # Force more parallelism for pull even on public images
+ PARALLELISM: 6
+ - name: Verify CI images ${{ env.PYTHON_VERSIONS }}:${{ env.IMAGE_TAG }}
+ run: breeze ci-image verify --run-in-parallel
+ env:
+ PYTHON_VERSIONS: ${{needs.build-info.outputs.all-python-versions-list-as-string}}
+ DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
+
+ preview-constraints:
+ permissions:
+ contents: read
+ timeout-minutes: 70
+ name: >
+ Preview constraints
+ ${{needs.build-info.outputs.all-python-versions-list-as-string}}
+ runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
+ needs: [build-info, wait-for-ci-images]
+ env:
+ RUNS_ON: "${{ needs.build-info.outputs.runs-on }}"
+ PYTHON_VERSIONS: ${{needs.build-info.outputs.all-python-versions-list-as-string}}
+ DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
+ if: needs.build-info.outputs.ci-image-build == 'true'
+ steps:
+ - name: Cleanup repo
+ run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
+ uses: actions/checkout@v4
+ with:
+ persist-credentials: false
+ - name: "Install Breeze"
+ uses: ./.github/actions/breeze
+ - name: Pull CI images ${{ env.PYTHON_VERSIONS }}:${{ env.IMAGE_TAG }}
+ run: breeze ci-image pull --run-in-parallel --tag-as-latest
+ - name: "Source constraints"
+ shell: bash
+ run: >
+ breeze release-management generate-constraints --run-in-parallel
+ --airflow-constraints-mode constraints-source-providers
+ - name: "No providers constraints"
+ shell: bash
+ timeout-minutes: 25
+ run: >
+ breeze release-management generate-constraints --run-in-parallel
+ --airflow-constraints-mode constraints-no-providers
+ - name: "PyPI constraints"
+ shell: bash
+ timeout-minutes: 25
+ run: >
+ breeze release-management generate-constraints --run-in-parallel
+ --airflow-constraints-mode constraints
+ - name: "Dependency upgrade summary"
+ shell: bash
+ run: |
+ for PYTHON_VERSION in ${{ env.PYTHON_VERSIONS }}; do
+ echo "Summarizing Python $PYTHON_VERSION"
+ cat "files/constraints-${PYTHON_VERSION}"/*.md >> $GITHUB_STEP_SUMMARY || true
+ done
+ - name: "Upload constraint artifacts"
+ uses: actions/upload-artifact@v3
+ with:
+ name: constraints
+ path: ./files/constraints-*/constraints-*.txt
+ retention-days: 7
+
static-checks:
timeout-minutes: 45
@@ -999,9 +953,7 @@ jobs:
JOB_ID: "helm-tests"
USE_XDIST: "true"
if: >
- needs.build-info.outputs.needs-helm-tests == 'true' &&
- (github.repository == 'apache/airflow' || github.event_name != 'schedule') &&
- needs.build-info.outputs.default-branch == 'main'
+ needs.build-info.outputs.needs-helm-tests == 'true' && needs.build-info.outputs.default-branch == 'main'
steps:
- name: Cleanup repo
run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
@@ -1644,10 +1596,93 @@ jobs:
echo
echo Total number of unique warnings $(cat ./artifacts/test-warnings*/* | sort | uniq | wc -l)
+ build-prod-images:
+ timeout-minutes: 80
+ name: >
+ ${{needs.build-info.outputs.build-job-description}} PROD images
+ ${{needs.build-info.outputs.all-python-versions-list-as-string}}
+ runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
+ needs: [build-info, build-ci-images]
+ env:
+ DEFAULT_BRANCH: ${{ needs.build-info.outputs.default-branch }}
+ DEFAULT_CONSTRAINTS_BRANCH: ${{ needs.build-info.outputs.default-constraints-branch }}
+ RUNS_ON: "${{needs.build-info.outputs.runs-on}}"
+ BACKEND: sqlite
+ VERSION_SUFFIX_FOR_PYPI: "dev0"
+ DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
+ # Force more parallelism for build even on public images
+ PARALLELISM: 6
+ steps:
+ - name: Cleanup repo
+ run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
+ if: needs.build-info.outputs.in-workflow-build == 'true'
+ - uses: actions/checkout@v4
+ with:
+ ref: ${{ needs.build-info.outputs.targetCommitSha }}
+ persist-credentials: false
+ if: needs.build-info.outputs.in-workflow-build == 'true'
+ - name: "Install Breeze"
+ uses: ./.github/actions/breeze
+ if: needs.build-info.outputs.in-workflow-build == 'true'
+ - name: >
+ Build PROD Images
+ ${{needs.build-info.outputs.all-python-versions-list-as-string}}:${{env.IMAGE_TAG}}
+ uses: ./.github/actions/build-prod-images
+ if: needs.build-info.outputs.in-workflow-build == 'true'
+ with:
+ build-provider-packages: ${{ needs.build-info.outputs.default-branch == 'main' }}
+ env:
+ UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgrade-to-newer-dependencies }}
+ DOCKER_CACHE: ${{ needs.build-info.outputs.cache-directive }}
+ PYTHON_VERSIONS: ${{needs.build-info.outputs.all-python-versions-list-as-string}}
+ DEBUG_RESOURCES: ${{ needs.build-info.outputs.debug-resources }}
+
+ build-prod-images-bullseye:
+ timeout-minutes: 80
+ name: >
+ ${{needs.build-info.outputs.build-job-description}} Bullseye PROD images
+ ${{needs.build-info.outputs.all-python-versions-list-as-string}}
+ runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
+ needs: [build-info, build-ci-images]
+ if: needs.build-info.outputs.canary-run == 'true'
+ env:
+ DEFAULT_BRANCH: ${{ needs.build-info.outputs.default-branch }}
+ DEFAULT_CONSTRAINTS_BRANCH: ${{ needs.build-info.outputs.default-constraints-branch }}
+ RUNS_ON: "${{needs.build-info.outputs.runs-on}}"
+ BACKEND: sqlite
+ VERSION_SUFFIX_FOR_PYPI: "dev0"
+ DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
+ # Force more parallelism for build even on public images
+ PARALLELISM: 6
+ steps:
+ - name: Cleanup repo
+ run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
+ - uses: actions/checkout@v3
+ with:
+ ref: ${{ needs.build-info.outputs.targetCommitSha }}
+ persist-credentials: false
+ submodules: recursive
+ - name: "Install Breeze"
+ uses: ./.github/actions/breeze
+ - name: >
+ Build Bullseye PROD Images
+ ${{needs.build-info.outputs.all-python-versions-list-as-string}}:${{env.IMAGE_TAG}}
+ uses: ./.github/actions/build-prod-images
+ with:
+ build-provider-packages: ${{ needs.build-info.outputs.default-branch == 'main' }}
+ env:
+ UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgrade-to-newer-dependencies }}
+ DOCKER_CACHE: ${{ needs.build-info.outputs.cache-directive }}
+ PYTHON_VERSIONS: ${{needs.build-info.outputs.all-python-versions-list-as-string}}
+ DEBUG_RESOURCES: ${{ needs.build-info.outputs.debug-resources }}
+ DEBIAN_VERSION: "bullseye"
+ # Do not override the "bookworm" image - just push a new bullseye image
+ # TODO: improve caching for that build
+ IMAGE_TAG: "bullseye-${{ github.event.pull_request.head.sha || github.sha }}"
wait-for-prod-images:
timeout-minutes: 80
- name: "Wait for & verify PROD images"
+ name: "Wait for PROD images"
runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
needs: [build-info, wait-for-ci-images, build-prod-images]
if: needs.build-info.outputs.prod-image-build == 'true'
@@ -1660,12 +1695,15 @@ jobs:
steps:
- name: Cleanup repo
run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
+ if: needs.build-info.outputs.in-workflow-build == 'false'
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v4
with:
persist-credentials: false
+ if: needs.build-info.outputs.in-workflow-build == 'false'
- name: "Install Breeze"
uses: ./.github/actions/breeze
+ if: needs.build-info.outputs.in-workflow-build == 'false'
- name: Wait for PROD images ${{ env.PYTHON_VERSIONS }}:${{ env.IMAGE_TAG }}
# We wait for the images to be available either from "build-images.yml' run as pull_request_target
# or from build-prod-images above.
@@ -1675,15 +1713,36 @@ jobs:
env:
PYTHON_VERSIONS: ${{ needs.build-info.outputs.python-versions-list-as-string }}
DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
+ if: needs.build-info.outputs.in-workflow-build == 'false'
+
+ verify-prod-images:
+ timeout-minutes: 80
+ name: Verify PROD images ${{needs.build-info.outputs.all-python-versions-list-as-string}}
+ runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
+ needs: [build-info, wait-for-prod-images]
+ env:
+ DEFAULT_BRANCH: ${{ needs.build-info.outputs.default-branch }}
+ DEFAULT_CONSTRAINTS_BRANCH: ${{ needs.build-info.outputs.default-constraints-branch }}
+ RUNS_ON: "${{needs.build-info.outputs.runs-on}}"
+ VERSION_SUFFIX_FOR_PYPI: "dev0"
+ steps:
+ - name: Cleanup repo
+ run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
+ - uses: actions/checkout@v4
+ with:
+ ref: ${{ needs.build-info.outputs.targetCommitSha }}
+ persist-credentials: false
+ - name: Pull PROD images ${{ env.PYTHON_VERSIONS }}:${{ env.IMAGE_TAG }}
+ run: breeze prod-image pull --run-in-parallel --tag-as-latest
+ env:
+ # Force more parallelism for pull even on public images
+ PARALLELISM: 6
- name: Verify PROD images ${{ env.PYTHON_VERSIONS }}:${{ env.IMAGE_TAG }}
- # We pull images again (which is a NOOP) but this time we also verify the images
- # Having it as a separate step allows us to see if the problem was with waiting or verification
- run: breeze prod-image pull --verify --run-in-parallel
+ run: breeze prod-image verify --run-in-parallel
env:
- PYTHON_VERSIONS: ${{ needs.build-info.outputs.python-versions-list-as-string }}
+ PYTHON_VERSIONS: ${{needs.build-info.outputs.all-python-versions-list-as-string}}
DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
-
test-docker-compose-quick-start:
timeout-minutes: 60
name: "Test docker-compose quick start"
@@ -1700,12 +1759,12 @@ jobs:
with:
fetch-depth: 2
persist-credentials: false
- - name: >
- Prepare breeze & PROD image:
- ${{needs.build-info.outputs.default-python-version}}:${{env.IMAGE_TAG}}
- uses: ./.github/actions/prepare_breeze_and_image
- with:
- pull-image-type: 'PROD'
+ - name: Pull PROD images ${{ env.PYTHON_VERSIONS }}:${{ env.IMAGE_TAG }}
+ run: breeze prod-image pull --run-in-parallel --tag-as-latest
+ env:
+ PYTHON_VERSIONS: ${{ needs.build-info.outputs.python-versions-list-as-string }}
+ # Force more parallelism for pull even on public images
+ PARALLELISM: 6
- name: "Test docker-compose quick start"
run: breeze testing docker-compose-tests
@@ -1741,6 +1800,8 @@ jobs:
run: breeze prod-image pull --run-in-parallel --tag-as-latest
env:
PYTHON_VERSIONS: ${{ needs.build-info.outputs.python-versions-list-as-string }}
+ # Force more parallelism for pull even on public images
+ PARALLELISM: 6
- name: "Cache bin folder with tools for kubernetes testing"
uses: actions/cache@v3
with:
@@ -1796,8 +1857,7 @@ jobs:
- tests-no-db
- tests-integration-postgres
- tests-integration-mysql
- # Skip when generate constraints fails
- - generate-constraints
+ - preview-constraints
env:
RUNS_ON: "${{needs.build-info.outputs.runs-on}}"
DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
diff --git a/CI.rst b/CI.rst
index 867686d8e7..5c2e00e80a 100644
--- a/CI.rst
+++ b/CI.rst
@@ -239,49 +239,39 @@ Regular PR builds run in a "stable" environment:
* no ARM image builds are build in the regular PRs
* lower probability of flaky tests for non-committer PRs (public runners and less parallelism)
+Maintainers can also run the "Pull Request run" from the "apache/airflow" repository by pushing
+to a branch in the "apache/airflow" repository. This is useful when you want to test a PR that
+changes the CI/CD infrastructure itself (for example changes to the CI/CD scripts or changes to
+the CI/CD workflows). In this case the PR is run in the context of the "apache/airflow" repository
+and has WRITE access to the GitHub Container Registry.
+
Canary run
----------
-Those runs are results of direct pushes done by the committers - basically merging of a Pull Request
-by the committers. Those runs execute in the context of the Apache Airflow Code Repository and have also
-write permission for GitHub resources (container registry, code repository).
-
-The main purpose for the run is to check if the code after merge still holds all the assertions - like
-whether it still builds, all tests are green. This is a "Canary" build that helps us to detect early
-problems with dependencies, image building, full matrix of tests in case they passed through selective checks.
-
-This is needed because some of the conflicting changes from multiple PRs might cause build and test failures
-after merge even if they do not fail in isolation. Also those runs are already reviewed and confirmed by the
-committers so they can be used to do some housekeeping:
+This is the flow that happens when a pull request is merged to the "main" branch or pushed to any of
+the "v2-*-test" branches. The "Canary" run attempts to upgrade dependencies to the latest versions
+and quickly pushes a preview of cache the CI/PROD images to the GitHub Registry - so that pull requests
+can quickly use the new cache - this is useful when Dockerfile or installation scripts change because such
+cache will already have the latest Dockerfile and scripts pushed even if some tests will fail.
+When successful, the run updates the constraints files in the "constraints-main" branch with the latest
+constraints and pushes both cache and latest CI/PROD images to the GitHub Registry.
-- pushing most recent image build in the PR to the GitHub Container Registry (for caching) including recent
- Dockerfile changes and setup.py/setup.cfg changes (Early Cache)
-- test that image in ``breeze`` command builds quickly
-- run full matrix of tests to detect any tests that will be mistakenly missed in ``selective checks``
-- upgrading to latest constraints and pushing those constraints if all tests succeed
-- refresh latest Python base images in case new patch-level is released
+When "Canary" build fails, it's often a sign that some of our dependencies released a new version that
+is not compatible with current tests or Airflow code, Also it might mean that a breaking change has been
+merged to "main". Both cases should be addressed quickly by the maintainers. The "broken main" by our code
+should be fixed quickly, while the "broken dependencies" can take a bit of time to fix as until the tests
+succeeds, constraints will not be updated, which means that regular PRs will continue using the old version
+of dependencies that already passed one of the previous "Canary" runs.
-The housekeeping is important - Python base images are refreshed with varying frequency (once every few months
-usually but sometimes several times per week) with the latest security and bug fixes.
Scheduled runs
--------------
-Those runs are results of (nightly) triggered job - only for ``main`` branch. The
-main purpose of the job is to check if there was no impact of external dependency changes on the Apache
-Airflow code (for example transitive dependencies released that fail the build). It also checks if the
-Docker images can be built from the scratch (again - to see if some dependencies have not changed - for
-example downloaded package releases etc.
-
-All runs consist of the same jobs, but the jobs behave slightly differently or they are skipped in different
-run categories. Here is a summary of the run categories with regards of the jobs they are running.
-Those jobs often have matrix run strategy which runs several different variations of the jobs
-(with different Backend type / Python version, type of the tests to run for example). The following chapter
-describes the workflows that execute for each run.
-
-Those runs and their corresponding ``Build Images`` runs are only executed in main ``apache/airflow``
-repository, they are not executed in forks - we want to be nice to the contributors and not use their
-free build minutes on GitHub Actions.
+This is the flow that happens when a scheduled run is triggered. The "scheduled" workflow is aimed to
+run regularly (overnight). Scheduled run is generally the same as "Canary" run, with the difference
+that the image is build always from the scratch and not from the cache. This way we can check that no
+"system" dependencies in debian base image have changed and that the build is still reproducible.
+No separate diagram is needed for scheduled run as it is identical to that of "Canary" run.
Workflows
=========
@@ -364,7 +354,7 @@ This workflow is a regular workflow that performs all checks of Airflow code.
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
| Build CI images | Builds images in-workflow (not in the ``build images``) | - | Yes | Yes (1) | Yes (4) |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
-| Generate constraints | Generates constraints that were updated in this build | Yes (2) | Yes (2) | Yes (2) | Yes (2) |
+| Preview constraints | Preview constraints that were updated in this build | Yes (2) | Yes (2) | Yes (2) | Yes (2) |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
| Build PROD images | Builds images in-workflow (not in the ``build images``) | - | Yes | Yes (1) | Yes (4) |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
diff --git a/CI_DIAGRAMS.md b/CI_DIAGRAMS.md
index 8bc1baabaf..9b349140da 100644
--- a/CI_DIAGRAMS.md
+++ b/CI_DIAGRAMS.md
@@ -23,11 +23,21 @@ You can see here the sequence diagrams of the flow happening during the CI Jobs.
## Pull request flow from fork
+This is the flow that happens when a pull request is created from a fork - which is the most frequent
+pull request flow that happens in Airflow. The "pull_request" workflow does not have write access
+to the GitHub Registry, so it cannot push the CI/PROD images there. Instead, we push the images
+from the "pull_request_target" workflow, which has write access to the GitHub Registry. Note that
+this workflow always uses scripts and workflows from the "target" branch of the "apache/airflow"
+repository, so the user submitting such pull request cannot override our build scripts and inject malicious
+code into the workflow that has potentially write access to the GitHub Registry (and can override cache).
+
+Security is the main reason why we have two workflows for pull requests and such complex workflows.
+
```mermaid
sequenceDiagram
Note over Airflow Repo: pull request
- Note over Tests: pull_request<br>[Read Token]
- Note over Build Images: pull_request_target<br>[Write Token]
+ Note over Tests: pull_request<br>[Write Token]
+ Note over Build Images: pull_request_target<br>[Unused Token]
activate Airflow Repo
Airflow Repo -->> Tests: Trigger 'pull_request'
activate Tests
@@ -39,6 +49,8 @@ sequenceDiagram
Note over Tests: Skip Build<br>(Runs in 'Build Images')<br>PROD Images
par
Note over Build Images: Build CI Images<br>[COMMIT_SHA]<br>Use latest constraints<br>or upgrade if setup changed
+ Build Images ->> GitHub Registry: Push CI Images<br>[COMMIT_SHA]
+ Build Images ->> Artifacts: Upload source constraints
and
Note over Tests: OpenAPI client gen
and
@@ -52,14 +64,18 @@ sequenceDiagram
Note over Tests: Run basic <br>static checks
end
end
- Build Images ->> GitHub Registry: Push CI Images<br>[COMMIT_SHA]
loop Wait for CI images
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
end
- Note over Tests: Verify CI Images<br>[COMMIT_SHA]
par
- GitHub Registry ->> Build Images: Pull PROD Images<br>[latest]
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Verify CI Images<br>[COMMIT_SHA]
+ Note over Tests: Generate/Preview constraints<br>source,pypi,no-providers
+ Tests ->> Artifacts: Upload source,pypi,no-providers constraints
+ and
+ Artifacts ->> Tests: Download source constraints
Note over Build Images: Build PROD Images<br>[COMMIT_SHA]
+ Build Images ->> GitHub Registry: Push PROD Images<br>[COMMIT_SHA]
and
opt
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
@@ -68,53 +84,78 @@ sequenceDiagram
and
opt
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Build docs / Spellcheck docs
+ Note over Tests: Build docs
+ end
+ and
+ opt
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Spellcheck docs
end
and
opt
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
Note over Tests: Test Pytest collection<br>[COMMIT_SHA]
- par
- opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Unit Tests<br>Python/DB matrix
- end
- and
- opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Integration Tests
- end
- and
- opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Quarantined Tests
- end
- end
end
and
opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Test provider <br>packages build
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Unit Tests<br>Python/DB matrix
+ end
+ and
+ opt
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Unit Tests<br>Python/Non-DB matrix
+ end
+ and
+ opt
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Integration Tests
+ end
+ and
+ opt
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Quarantined Tests
+ end
+ and
+ opt
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Test provider <br>packages build<br>wheel, sdist, old airflow
end
and
opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Test airflow <br>packages build
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Test airflow <br>release commands
end
and
opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Helm tests
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Helm tests
end
end
- Note over Tests: Summarize Warnings
- Build Images ->> GitHub Registry: Push PROD Images<br>[COMMIT_SHA]
- deactivate Build Images
- loop Wait for PROD images
- GitHub Registry ->> Tests: Pull PROD Images<br>[COMMIT_SHA]
+ par
+ Note over Tests: Summarize Warnings
+ and
+ opt
+ Artifacts ->> Tests: Download source,pypi,no-providers constraints
+ Note over Tests: Display constraints diff
+ end
+ and
+ opt
+ loop Wait for PROD images
+ GitHub Registry ->> Tests: Pull PROD Images<br>[COMMIT_SHA]
+ end
+ end
+ and
+ opt
+ Note over Tests: Build ARM CI images
+ end
end
- Note over Tests: Verify PROD Image<br>[COMMIT_SHA]
par
+ opt
+ GitHub Registry ->> Tests: Pull PROD Images<br>[COMMIT_SHA]
+ Note over Tests: Verify PROD Images<br>[COMMIT_SHA]
+ end
+ and
opt
GitHub Registry ->> Tests: Pull PROD Images<br>[COMMIT_SHA]
Note over Tests: Run Kubernetes <br>tests
@@ -125,11 +166,6 @@ sequenceDiagram
Note over Tests: Run docker-compose <br>tests
end
end
- opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Generate constraints
- end
- Note over Tests: Build ARM CI images
Tests -->> Airflow Repo: Status update
deactivate Airflow Repo
deactivate Tests
@@ -137,6 +173,16 @@ sequenceDiagram
## Pull request flow from "apache/airflow" repo
+The difference between this flow and the previous one is that the CI/PROD images are built in the
+CI workflow and pushed to the GitHub Registry from there. This cannot be done in case of fork
+pull request, because Pull Request from forks cannot have "write" access to GitHub Registry. All the steps
+except "Build Info" from the "Build Images" workflows are skipped in this case.
+
+THis workflow can be used by maintainers in case they have a Pull Request that changes the scripts and
+CI workflows used to build images, because in this case the "Build Images" workflow will use them
+from the Pull Request. This is safe, because the Pull Request is from the "apache/airflow" repository
+and only maintainers can push to that repository and create Pull Requests from it.
+
```mermaid
sequenceDiagram
Note over Airflow Repo: pull request
@@ -154,10 +200,12 @@ sequenceDiagram
Note over Tests: Build info<br>Decide on tests<br>Decide on Matrix (selective)
par
Note over Tests: Build CI Images<br>[COMMIT_SHA]<br>Use latest constraints<br>or upgrade if setup changed
+ Tests ->> GitHub Registry: Push CI Images<br>[COMMIT_SHA]
+ Tests ->> Artifacts: Upload source constraints
and
Note over Tests: OpenAPI client gen
and
- Note over Tests: Test UI
+ Note over Tests: React WWW tests
and
Note over Tests: Test examples<br>PROD image building
and
@@ -167,11 +215,16 @@ sequenceDiagram
Note over Tests: Run basic <br>static checks
end
end
- Tests ->> GitHub Registry: Push CI Images<br>[COMMIT_SHA]
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Verify CI Image<br>[COMMIT_SHA]
+ Note over Tests: Skip waiting for CI images
par
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Verify CI Images<br>[COMMIT_SHA]
+ Note over Tests: Generate/Preview constraints<br>source,pypi,no-providers
+ Tests ->> Artifacts: Upload source,pypi,no-providers constraints
+ and
+ Artifacts ->> Tests: Download source constraints
Note over Tests: Build PROD Images<br>[COMMIT_SHA]
+ Tests ->> GitHub Registry: Push PROD Images<br>[COMMIT_SHA]
and
opt
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
@@ -180,50 +233,70 @@ sequenceDiagram
and
opt
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Build docs/ Spellcheck docs
+ Note over Tests: Build docs
+ end
+ and
+ opt
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Spellcheck docs
end
and
opt
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
Note over Tests: Test Pytest collection<br>[COMMIT_SHA]
- par
- opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Unit Tests<br>Python/DB matrix/No DB
- end
- and
- opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Integration Tests
- end
- and
- opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Quarantined Tests
- end
- end
end
and
opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Test provider <br>packages build
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Unit Tests<br>Python/DB matrix
+ end
+ and
+ opt
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Unit Tests<br>Python/Non-DB matrix
+ end
+ and
+ opt
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Integration Tests
+ end
+ and
+ opt
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Quarantined Tests
+ end
+ and
+ opt
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Test provider <br>packages build<br>wheel, sdist, old airflow
end
and
opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Test airflow <br>packages build
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Test airflow <br>release commands
end
and
opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Helm tests
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Helm tests
end
end
- Note over Tests: Summarize Warnings
- Tests ->> GitHub Registry: Push PROD Images<br>[COMMIT_SHA]
- GitHub Registry ->> Tests: Pull PROD Image<br>[COMIT_SHA]
- Note over Tests: Verify PROD Image<br>[COMMIT_SHA]
+ Note over Tests: Skip waiting for PROD images
par
+ Note over Tests: Summarize Warnings
+ and
+ opt
+ Artifacts ->> Tests: Download source,pypi,no-providers constraints
+ Note over Tests: Display constraints diff
+ end
+ and
+ Note over Tests: Build ARM CI images
+ and
+ opt
+ GitHub Registry ->> Tests: Pull PROD Images<br>[COMMIT_SHA]
+ Note over Tests: Verify PROD Images<br>[COMMIT_SHA]
+ end
+ and
opt
GitHub Registry ->> Tests: Pull PROD Images<br>[COMMIT_SHA]
Note over Tests: Run Kubernetes <br>tests
@@ -234,11 +307,6 @@ sequenceDiagram
Note over Tests: Run docker-compose <br>tests
end
end
- opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Generate constraints
- end
- Note over Tests: Build ARM CI images
Tests -->> Airflow Repo: Status update
deactivate Airflow Repo
deactivate Tests
@@ -246,6 +314,14 @@ sequenceDiagram
## Merge "Canary" run
+This is the flow that happens when a pull request is merged to the "main" branch or pushed to any of
+the "v2-*-test" branches. The "Canary" run attempts to upgrade dependencies to the latest versions
+and quickly pushes a preview of cache the CI/PROD images to the GitHub Registry - so that pull requests
+can quickly use the new cache - this is useful when Dockerfile or installation scripts change because such
+cache will already have the latest Dockerfile and scripts pushed even if some tests will fail.
+When successful, the run updates the constraints files in the "constraints-main" branch with the latest
+constraints and pushes both cache and latest CI/PROD images to the GitHub Registry.
+
```mermaid
sequenceDiagram
Note over Airflow Repo: push/merge
@@ -256,6 +332,8 @@ sequenceDiagram
Note over Tests: Build info<br>All tests<br>Full matrix
par
Note over Tests: Build CI Images<br>[COMMIT_SHA]<br>Always upgrade deps
+ Tests ->> GitHub Registry: Push CI Images<br>[COMMIT_SHA]
+ Tests ->> Artifacts: Upload source constraints
and
Note over Tests: Check that image builds quickly
and
@@ -269,147 +347,88 @@ sequenceDiagram
and
Note over Tests: Test git clone on Windows
end
- Tests ->> GitHub Registry: Push CI Images<br>[COMMIT_SHA]
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Verify CI Image<br>[COMMIT_SHA]
+ Note over Tests: Skip waiting for CI images
par
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Verify CI Images<br>[COMMIT_SHA]
+ Note over Tests: Generate/Preview constraints<br>source,pypi,no-providers
+ Tests ->> Artifacts: Upload source,pypi,no-providers constraints
+ and
+ Artifacts ->> Tests: Download source constraints
Note over Tests: Build PROD Images<br>[COMMIT_SHA]
+ Tests ->> GitHub Registry: Push PROD Images<br>[COMMIT_SHA]
and
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
Note over Tests: Run static checks
and
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Build docs / Spellcheck docs
+ Note over Tests: Build docs
and
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Test Pytest collection<br>[COMMIT_SHA]
- par
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Unit Tests<br>Python/DB matrix
- and
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Integration Tests
- and
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Quarantined Tests
- end
- and
- opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Test provider <br>packages build
- end
+ Note over Tests: Spellcheck docs
and
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Test airflow <br>packages build
- and
- opt
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Helm tests
- end
- end
- Note over Tests: Summarize Warnings
- Tests ->> GitHub Registry: Push PROD Images<br>[COMMIT_SHA]
- GitHub Registry ->> Tests: Pull PROD Image<br>[COMMIT_SHA]
- Note over Tests: Verify PROD Image<br>[COMMIT_SHA]
- par
- GitHub Registry ->> Tests: Pull PROD Image<br>[COMMIT_SHA]
- Note over Tests: Run Kubernetes <br>tests
- and
- GitHub Registry ->> Tests: Pull PROD Image<br>[COMMIT_SHA]
- Note over Tests: Run docker-compose <br>tests
- end
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Generate constraints
- Tests ->> Airflow Repo: Push constraints if changed
- Note over Tests: Build CI Images<br>[latest]<br>Use latest constraints
- Tests ->> GitHub Registry: Push CI Image<br>[latest]
- Note over Tests: Build PROD Images<br>[latest]<br>Use latest constraints
- Tests ->> GitHub Registry: Push PROD Image<br>[latest]
- Note over Tests: Build ARM CI images
- Tests ->> GitHub Registry: Push ARM CI Image cache
- Tests -->> Airflow Repo: Status update
- deactivate Airflow Repo
- deactivate Tests
-```
-
-## Scheduled run
-
-```mermaid
-sequenceDiagram
- Note over Airflow Repo: scheduled
- Note over Tests: push<br>[Write Token]
- activate Airflow Repo
- Airflow Repo -->> Tests: Trigger 'schedule'
- activate Tests
- Note over Tests: Build info<br>All tests<br>Full matrix
- par
- GitHub Registry ->> Tests: Pull CI Images<br>[latest]
- Note over Tests: Build CI Images<br>[COMMIT_SHA]<br>Always upgrade deps
- and
- Note over Tests: OpenAPI client gen
- and
- Note over Tests: React WWW tests
- and
- Note over Tests: Test examples<br>PROD image building
+ Note over Tests: Test Pytest collection<br>[COMMIT_SHA]
and
- Note over Tests: Build CI Images<br>Use original constraints
- end
- Tests ->> GitHub Registry: Push CI Images<br>[COMMIT_SHA]
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Verify CI Image<br>[COMMIT_SHA]
- par
- GitHub Registry ->> Tests: Pull PROD Images<br>[latest]
- Note over Tests: Build PROD Images<br>[COMMIT_SHA]
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Unit Tests<br>Python/DB matrix
and
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Run static checks
+ Note over Tests: Unit Tests<br>Python/Non-DB matrix
and
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Build docs / Spellcheck docs
+ Note over Tests: Integration Tests
and
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Test Pytest collection<br>[COMMIT_SHA]
- par
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Unit Tests<br>Python/DB matrix
- and
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Integration Tests
- and
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Quarantined Tests
- end
+ Note over Tests: Quarantined Tests
and
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Test provider <br>packages build
+ Note over Tests: Test provider <br>packages build<br>wheel, sdist, old airflow
and
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Test airflow <br>packages build
+ Note over Tests: Test airflow <br>release commands
and
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
Note over Tests: Helm tests
end
- Note over Tests: Summarize Warnings
- Tests ->> GitHub Registry: Push PROD Images<br>[COMMIT_SHA]
- GitHub Registry ->> Tests: Pull PROD Image<br>[COMMIT_SHA]
- Note over Tests: Verify PROD Image<br>[COMMIT_SHA]
+ Note over Tests: Skip waiting for PROD images
par
+ Note over Tests: Summarize Warnings
+ and
+ Artifacts ->> Tests: Download source,pypi,no-providers constraints
+ Note over Tests: Display constraints diff
+ Tests ->> Airflow Repo: Push constraints if changed to 'constraints-BRANCH'
+ and
+ GitHub Registry ->> Tests: Pull PROD Images<br>[COMMIT_SHA]
+ Note over Tests: Verify PROD Images<br>[COMMIT_SHA]
+ and
GitHub Registry ->> Tests: Pull PROD Image<br>[COMMIT_SHA]
Note over Tests: Run Kubernetes <br>tests
and
GitHub Registry ->> Tests: Pull PROD Image<br>[COMMIT_SHA]
Note over Tests: Run docker-compose <br>tests
end
- GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
- Note over Tests: Generate constraints
- Tests ->> Airflow Repo: Push constraints if changed
- Note over Tests: Build CI Images<br>[latest]<br>Use latest constraints
- Tests ->> GitHub Registry: Push CI Image cache + latest
- Note over Tests: Build PROD Images<br>[latest]<br>Use latest constraints
- Tests ->> GitHub Registry: Push PROD Image cache + latest
- Note over Tests: Build ARM CI images
- Tests ->> GitHub Registry: Push ARM CI Image cache
+ par
+ Note over Tests: Build CI Images/cache<br>[latest]<br>Use pushed constraints
+ Tests ->> GitHub Registry: Push CI Images/cache[latest]
+ Note over Tests: Build PROD Images/cache<br>[latest]<br>Use pushed constraints
+ Tests ->> GitHub Registry: Push PROD Images/cache[latest]
+ and
+ Note over Tests: Build ARM CI images[latest]<br>Use pushed constraints
+ Tests ->> GitHub Registry: Push ARM CI Images[latest]
+ Note over Tests: Build ARM PROD images[latest]<br>Use pushed constraints
+ Tests ->> GitHub Registry: Push ARM PROD Images[latest]
+ end
Tests -->> Airflow Repo: Status update
deactivate Airflow Repo
deactivate Tests
```
+
+## Scheduled run
+
+This is the flow that happens when a scheduled run is triggered. The "scheduled" workflow is aimed to
+run regularly (overnight) even if no new PRs are merged to "main". Scheduled run is generally the
+same as "Canary" run, with the difference that the image used to run the tests is built without using
+cache - it's always built from the scratch. This way we can check that no "system" dependencies in debian
+base image have changed and that the build is still reproducible. No separate diagram is needed for
+scheduled run as it is identical to that of "Canary" run.
diff --git a/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py b/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
index f33fdad2ef..08722caf30 100644
--- a/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
@@ -93,7 +93,12 @@ from airflow_breeze.utils.docker_command_utils import (
from airflow_breeze.utils.image import run_pull_image, run_pull_in_parallel, tag_image_as_latest
from airflow_breeze.utils.mark_image_as_refreshed import mark_image_as_refreshed
from airflow_breeze.utils.md5_build_check import md5sum_check_if_build_is_needed
-from airflow_breeze.utils.parallel import DockerBuildxProgressMatcher, check_async_run_results, run_with_pool
+from airflow_breeze.utils.parallel import (
+ DockerBuildxProgressMatcher,
+ ShowLastLineProgressMatcher,
+ check_async_run_results,
+ run_with_pool,
+)
from airflow_breeze.utils.path_utils import AIRFLOW_SOURCES_ROOT, BUILD_CACHE_DIR
from airflow_breeze.utils.python_versions import get_python_version_list
from airflow_breeze.utils.registry import login_to_github_docker_registry
@@ -476,6 +481,45 @@ def pull(
sys.exit(return_code)
+def run_verify_in_parallel(
+ image_params_list: list[BuildCiParams],
+ python_version_list: list[str],
+ extra_pytest_args: tuple[str, ...],
+ include_success_outputs: bool,
+ parallelism: int,
+ skip_cleanup: bool,
+ debug_resources: bool,
+) -> None:
+ with ci_group(f"Verifying CI images for {python_version_list}"):
+ all_params = [f"CI {image_params.python}" for image_params in image_params_list]
+ with run_with_pool(
+ parallelism=parallelism,
+ all_params=all_params,
+ debug_resources=debug_resources,
+ progress_matcher=ShowLastLineProgressMatcher(),
+ ) as (pool, outputs):
+ results = [
+ pool.apply_async(
+ verify_an_image,
+ kwds={
+ "image_name": image_params.airflow_image_name_with_tag,
+ "image_type": "CI",
+ "slim_image": False,
+ "extra_pytest_args": extra_pytest_args,
+ "output": outputs[index],
+ },
+ )
+ for index, image_params in enumerate(image_params_list)
+ ]
+ check_async_run_results(
+ results=results,
+ success="All images verified",
+ outputs=outputs,
+ include_success_outputs=include_success_outputs,
+ skip_cleanup=skip_cleanup,
+ )
+
+
@ci_image.command(
name="verify",
context_settings=dict(
@@ -484,22 +528,34 @@ def pull(
),
)
@option_python
+@option_python_versions
@option_github_repository
@option_image_tag_for_verifying
@option_image_name
@option_pull
@option_github_token
+@option_run_in_parallel
+@option_parallelism
+@option_skip_cleanup
+@option_include_success_outputs
+@option_debug_resources
@option_verbose
@option_dry_run
@click.argument("extra_pytest_args", nargs=-1, type=click.UNPROCESSED)
def verify(
python: str,
+ python_versions: str,
image_name: str,
image_tag: str | None,
pull: bool,
github_token: str,
github_repository: str,
- extra_pytest_args: tuple,
+ extra_pytest_args: tuple[str, ...],
+ run_in_parallel: bool,
+ parallelism: int,
+ skip_cleanup: bool,
+ debug_resources: bool,
+ include_success_outputs: bool,
):
"""Verify CI image."""
perform_environment_checks()
@@ -507,27 +563,54 @@ def verify(
github_token=github_token,
output=None,
)
- if image_name is None:
- build_params = BuildCiParams(
+ if (pull or image_name) and run_in_parallel:
+ get_console().print(
+ "[error]You cannot use --pull,--image-name and --run-in-parallel at the same time. " "Exiting[/]"
+ )
+ sys.exit(1)
+ if run_in_parallel:
+ base_build_params = BuildCiParams(
python=python,
- image_tag=image_tag,
github_repository=github_repository,
- github_token=github_token,
+ image_tag=image_tag,
)
- image_name = build_params.airflow_image_name_with_tag
- if pull:
- check_remote_ghcr_io_commands()
- command_to_run = ["docker", "pull", image_name]
- run_command(command_to_run, check=True)
- get_console().print(f"[info]Verifying CI image: {image_name}[/]")
- return_code, info = verify_an_image(
- image_name=image_name,
- output=None,
- image_type="CI",
- slim_image=False,
- extra_pytest_args=extra_pytest_args,
- )
- sys.exit(return_code)
+ python_version_list = get_python_version_list(python_versions)
+ params_list: list[BuildCiParams] = []
+ for python in python_version_list:
+ build_params = deepcopy(base_build_params)
+ build_params.python = python
+ params_list.append(build_params)
+ run_verify_in_parallel(
+ image_params_list=params_list,
+ python_version_list=python_version_list,
+ extra_pytest_args=extra_pytest_args,
+ include_success_outputs=include_success_outputs,
+ parallelism=parallelism,
+ skip_cleanup=skip_cleanup,
+ debug_resources=debug_resources,
+ )
+ else:
+ if image_name is None:
+ build_params = BuildCiParams(
+ python=python,
+ image_tag=image_tag,
+ github_repository=github_repository,
+ github_token=github_token,
+ )
+ image_name = build_params.airflow_image_name_with_tag
+ if pull:
+ check_remote_ghcr_io_commands()
+ command_to_run = ["docker", "pull", image_name]
+ run_command(command_to_run, check=True)
+ get_console().print(f"[info]Verifying CI image: {image_name}[/]")
+ return_code, info = verify_an_image(
+ image_name=image_name,
+ output=None,
+ image_type="CI",
+ slim_image=False,
+ extra_pytest_args=extra_pytest_args,
+ )
+ sys.exit(return_code)
def should_we_run_the_build(build_ci_params: BuildCiParams) -> bool:
diff --git a/dev/breeze/src/airflow_breeze/commands/ci_image_commands_config.py b/dev/breeze/src/airflow_breeze/commands/ci_image_commands_config.py
index e128aa035b..021bec3a68 100644
--- a/dev/breeze/src/airflow_breeze/commands/ci_image_commands_config.py
+++ b/dev/breeze/src/airflow_breeze/commands/ci_image_commands_config.py
@@ -144,6 +144,17 @@ CI_IMAGE_TOOLS_PARAMETERS: dict[str, list[dict[str, str | list[str]]]] = {
"--pull",
],
},
+ {
+ "name": "Parallel running",
+ "options": [
+ "--run-in-parallel",
+ "--parallelism",
+ "--python-versions",
+ "--skip-cleanup",
+ "--debug-resources",
+ "--include-success-outputs",
+ ],
+ },
{
"name": "Github authentication",
"options": [
diff --git a/dev/breeze/src/airflow_breeze/commands/production_image_commands.py b/dev/breeze/src/airflow_breeze/commands/production_image_commands.py
index ca1780738d..27f195bc2e 100644
--- a/dev/breeze/src/airflow_breeze/commands/production_image_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/production_image_commands.py
@@ -86,7 +86,12 @@ from airflow_breeze.utils.docker_command_utils import (
warm_up_docker_builder,
)
from airflow_breeze.utils.image import run_pull_image, run_pull_in_parallel, tag_image_as_latest
-from airflow_breeze.utils.parallel import DockerBuildxProgressMatcher, check_async_run_results, run_with_pool
+from airflow_breeze.utils.parallel import (
+ DockerBuildxProgressMatcher,
+ ShowLastLineProgressMatcher,
+ check_async_run_results,
+ run_with_pool,
+)
from airflow_breeze.utils.path_utils import AIRFLOW_SOURCES_ROOT, DOCKER_CONTEXT_DIR
from airflow_breeze.utils.python_versions import get_python_version_list
from airflow_breeze.utils.registry import login_to_github_docker_registry
@@ -448,6 +453,45 @@ def pull_prod_image(
sys.exit(return_code)
+def run_verify_in_parallel(
+ image_params_list: list[BuildProdParams],
+ python_version_list: list[str],
+ extra_pytest_args: tuple[str, ...],
+ include_success_outputs: bool,
+ parallelism: int,
+ skip_cleanup: bool,
+ debug_resources: bool,
+) -> None:
+ with ci_group(f"Verifying PROD images for {python_version_list}"):
+ all_params = [f"PROD {image_params.python}" for image_params in image_params_list]
+ with run_with_pool(
+ parallelism=parallelism,
+ all_params=all_params,
+ debug_resources=debug_resources,
+ progress_matcher=ShowLastLineProgressMatcher(),
+ ) as (pool, outputs):
+ results = [
+ pool.apply_async(
+ verify_an_image,
+ kwds={
+ "image_name": image_params.airflow_image_name_with_tag,
+ "image_type": "PROD",
+ "slim_image": False,
+ "extra_pytest_args": extra_pytest_args,
+ "output": outputs[index],
+ },
+ )
+ for index, image_params in enumerate(image_params_list)
+ ]
+ check_async_run_results(
+ results=results,
+ success="All images verified",
+ outputs=outputs,
+ include_success_outputs=include_success_outputs,
+ skip_cleanup=skip_cleanup,
+ )
+
+
@prod_image.command(
name="verify",
context_settings=dict(
@@ -455,22 +499,29 @@ def pull_prod_image(
allow_extra_args=True,
),
)
-@option_python
-@option_image_tag_for_verifying
-@option_image_name
-@option_pull
@click.option(
"--slim-image",
help="The image to verify is slim and non-slim tests should be skipped.",
is_flag=True,
)
+@click.argument("extra_pytest_args", nargs=-1, type=click.UNPROCESSED)
+@option_python
+@option_python_versions
+@option_image_tag_for_verifying
+@option_image_name
+@option_pull
@option_github_repository
@option_github_token
+@option_run_in_parallel
+@option_parallelism
+@option_skip_cleanup
+@option_include_success_outputs
+@option_debug_resources
@option_verbose
@option_dry_run
-@click.argument("extra_pytest_args", nargs=-1, type=click.UNPROCESSED)
def verify(
python: str,
+ python_versions: str,
github_repository: str,
image_name: str,
image_tag: str | None,
@@ -478,6 +529,11 @@ def verify(
slim_image: bool,
github_token: str,
extra_pytest_args: tuple,
+ run_in_parallel: bool,
+ parallelism: int,
+ skip_cleanup: bool,
+ debug_resources: bool,
+ include_success_outputs: bool,
):
"""Verify Production image."""
perform_environment_checks()
@@ -485,27 +541,54 @@ def verify(
github_token=github_token,
output=None,
)
- if image_name is None:
- build_params = BuildProdParams(
+ if (pull or image_name) and run_in_parallel:
+ get_console().print(
+ "[error]You cannot use --pull,--image-name and --run-in-parallel at the same time. " "Exiting[/]"
+ )
+ sys.exit(1)
+ if run_in_parallel:
+ base_build_params = BuildProdParams(
python=python,
- image_tag=image_tag,
github_repository=github_repository,
- github_token=github_token,
+ image_tag=image_tag,
)
- image_name = build_params.airflow_image_name_with_tag
- if pull:
- check_remote_ghcr_io_commands()
- command_to_run = ["docker", "pull", image_name]
- run_command(command_to_run, check=True)
- get_console().print(f"[info]Verifying PROD image: {image_name}[/]")
- return_code, info = verify_an_image(
- image_name=image_name,
- output=None,
- image_type="PROD",
- extra_pytest_args=extra_pytest_args,
- slim_image=slim_image,
- )
- sys.exit(return_code)
+ python_version_list = get_python_version_list(python_versions)
+ params_list: list[BuildProdParams] = []
+ for python in python_version_list:
+ build_params = deepcopy(base_build_params)
+ build_params.python = python
+ params_list.append(build_params)
+ run_verify_in_parallel(
+ image_params_list=params_list,
+ python_version_list=python_version_list,
+ extra_pytest_args=extra_pytest_args,
+ include_success_outputs=include_success_outputs,
+ parallelism=parallelism,
+ skip_cleanup=skip_cleanup,
+ debug_resources=debug_resources,
+ )
+ else:
+ if image_name is None:
+ build_params = BuildProdParams(
+ python=python,
+ image_tag=image_tag,
+ github_repository=github_repository,
+ github_token=github_token,
+ )
+ image_name = build_params.airflow_image_name_with_tag
+ if pull:
+ check_remote_ghcr_io_commands()
+ command_to_run = ["docker", "pull", image_name]
+ run_command(command_to_run, check=True)
+ get_console().print(f"[info]Verifying PROD image: {image_name}[/]")
+ return_code, info = verify_an_image(
+ image_name=image_name,
+ output=None,
+ image_type="PROD",
+ extra_pytest_args=extra_pytest_args,
+ slim_image=slim_image,
+ )
+ sys.exit(return_code)
def clean_docker_context_files():
diff --git a/dev/breeze/src/airflow_breeze/commands/production_image_commands_config.py b/dev/breeze/src/airflow_breeze/commands/production_image_commands_config.py
index 52291268f8..9d20567537 100644
--- a/dev/breeze/src/airflow_breeze/commands/production_image_commands_config.py
+++ b/dev/breeze/src/airflow_breeze/commands/production_image_commands_config.py
@@ -157,6 +157,17 @@ PRODUCTION_IMAGE_TOOLS_PARAMETERS: dict[str, list[dict[str, str | list[str]]]] =
"--pull",
],
},
+ {
+ "name": "Parallel running",
+ "options": [
+ "--run-in-parallel",
+ "--parallelism",
+ "--python-versions",
+ "--skip-cleanup",
+ "--debug-resources",
+ "--include-success-outputs",
+ ],
+ },
{
"name": "Github authentication",
"options": [
diff --git a/dev/breeze/src/airflow_breeze/utils/run_tests.py b/dev/breeze/src/airflow_breeze/utils/run_tests.py
index d2ce641fe4..c3fd500f08 100644
--- a/dev/breeze/src/airflow_breeze/utils/run_tests.py
+++ b/dev/breeze/src/airflow_breeze/utils/run_tests.py
@@ -33,7 +33,7 @@ def verify_an_image(
image_type: str,
output: Output | None,
slim_image: bool,
- extra_pytest_args: tuple,
+ extra_pytest_args: tuple[str, ...],
) -> tuple[int, str]:
command_result = run_command(
["docker", "inspect", image_name],
diff --git a/docker_tests/test_ci_image.py b/docker_tests/test_ci_image.py
index 2c3fabd7d9..7ec65c425c 100644
--- a/docker_tests/test_ci_image.py
+++ b/docker_tests/test_ci_image.py
@@ -22,12 +22,28 @@ from docker_tests.command_utils import run_command
from docker_tests.docker_tests_utils import display_dependency_conflict_message, docker_image
-class TestPythonPackages:
- def test_pip_dependencies_conflict(self):
- try:
- run_command(
- ["docker", "run", "--rm", "--entrypoint", "/bin/bash", docker_image, "-c", "pip check"]
- )
- except subprocess.CalledProcessError as ex:
- display_dependency_conflict_message()
- raise ex
+def test_pip_dependencies_conflict():
+ try:
+ run_command(["docker", "run", "--rm", "--entrypoint", "/bin/bash", docker_image, "-c", "pip check"])
+ except subprocess.CalledProcessError as ex:
+ display_dependency_conflict_message()
+ raise ex
+
+
+def test_providers_present():
+ try:
+ run_command(
+ [
+ "docker",
+ "run",
+ "--rm",
+ "--entrypoint",
+ "/bin/bash",
+ docker_image,
+ "-c",
+ "airflow providers list",
+ ],
+ )
+ except subprocess.CalledProcessError as ex:
+ display_dependency_conflict_message()
+ raise ex
diff --git a/images/breeze/output_ci-image_verify.svg b/images/breeze/output_ci-image_verify.svg
index 42b327ef9c..cb247415e9 100644
--- a/images/breeze/output_ci-image_verify.svg
+++ b/images/breeze/output_ci-image_verify.svg
@@ -1,4 +1,4 @@
-<svg class="rich-terminal" viewBox="0 0 1482 562.4" xmlns="http://www.w3.org/2000/svg">
+<svg class="rich-terminal" viewBox="0 0 1482 830.8" xmlns="http://www.w3.org/2000/svg">
<!-- Generated with Rich https://www.textualize.io -->
<style>
@@ -43,7 +43,7 @@
<defs>
<clipPath id="breeze-ci-image-verify-clip-terminal">
- <rect x="0" y="0" width="1463.0" height="511.4" />
+ <rect x="0" y="0" width="1463.0" height="779.8" />
</clipPath>
<clipPath id="breeze-ci-image-verify-line-0">
<rect x="0" y="1.5" width="1464" height="24.65"/>
@@ -105,9 +105,42 @@
<clipPath id="breeze-ci-image-verify-line-19">
<rect x="0" y="465.1" width="1464" height="24.65"/>
</clipPath>
+<clipPath id="breeze-ci-image-verify-line-20">
+ <rect x="0" y="489.5" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-ci-image-verify-line-21">
+ <rect x="0" y="513.9" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-ci-image-verify-line-22">
+ <rect x="0" y="538.3" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-ci-image-verify-line-23">
+ <rect x="0" y="562.7" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-ci-image-verify-line-24">
+ <rect x="0" y="587.1" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-ci-image-verify-line-25">
+ <rect x="0" y="611.5" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-ci-image-verify-line-26">
+ <rect x="0" y="635.9" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-ci-image-verify-line-27">
+ <rect x="0" y="660.3" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-ci-image-verify-line-28">
+ <rect x="0" y="684.7" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-ci-image-verify-line-29">
+ <rect x="0" y="709.1" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-ci-image-verify-line-30">
+ <rect x="0" y="733.5" width="1464" height="24.65"/>
+ </clipPath>
</defs>
- <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="560.4" rx="8"/><text class="breeze-ci-image-verify-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: ci-image verify</text>
+ <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="828.8" rx="8"/><text class="breeze-ci-image-verify-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: ci-image verify</text>
<g transform="translate(26,22)">
<circle cx="0" cy="0" r="7" fill="#ff5f57"/>
<circle cx="22" cy="0" r="7" fill="#febc2e"/>
@@ -129,15 +162,26 @@
</text><text class="breeze-ci-image-verify-r5" x="0" y="239.6" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-9)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="239.6" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-9)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="239.6" textLength="73.2" clip-path="url(#breeze-ci-image-verify-line-9)">-image</text><text class="breeze-ci-image-verify-r4" x="109.8" y="239.6" textLength="48.8" clip-p [...]
</text><text class="breeze-ci-image-verify-r5" x="0" y="264" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-10)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="264" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-10)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="264" textLength="61" clip-path="url(#breeze-ci-image-verify-line-10)">-pull</text><text class="breeze-ci-image-verify-r1" x="244" y="264" textLength="646.6" clip-path="url( [...]
</text><text class="breeze-ci-image-verify-r5" x="0" y="288.4" textLength="1464" clip-path="url(#breeze-ci-image-verify-line-11)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-verify-r1" x="1464" y="288.4" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-11)">
-</text><text class="breeze-ci-image-verify-r5" x="0" y="312.8" textLength="24.4" clip-path="url(#breeze-ci-image-verify-line-12)">╭─</text><text class="breeze-ci-image-verify-r5" x="24.4" y="312.8" textLength="280.6" clip-path="url(#breeze-ci-image-verify-line-12)"> Github authentication </text><text class="breeze-ci-image-verify-r5" x="305" y="312.8" textLength="1134.6" clip-path="url(#breeze-ci-image-verify-line-12)">────────────────────────────────────────────────────── [...]
-</text><text class="breeze-ci-image-verify-r5" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-13)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="337.2" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-13)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="337.2" textLength="85.4" clip-path="url(#breeze-ci-image-verify-line-13)">-github</text><text class="breeze-ci-image-verify-r4" x="122" y="337.2" textLength="134.2" cli [...]
-</text><text class="breeze-ci-image-verify-r5" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-14)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="361.6" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-14)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="361.6" textLength="85.4" clip-path="url(#breeze-ci-image-verify-line-14)">-github</text><text class="breeze-ci-image-verify-r4" x="122" y="361.6" textLength="73.2" clip [...]
-</text><text class="breeze-ci-image-verify-r5" x="0" y="386" textLength="1464" clip-path="url(#breeze-ci-image-verify-line-15)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-verify-r1" x="1464" y="386" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-15)">
-</text><text class="breeze-ci-image-verify-r5" x="0" y="410.4" textLength="24.4" clip-path="url(#breeze-ci-image-verify-line-16)">╭─</text><text class="breeze-ci-image-verify-r5" x="24.4" y="410.4" textLength="195.2" clip-path="url(#breeze-ci-image-verify-line-16)"> Common options </text><text class="breeze-ci-image-verify-r5" x="219.6" y="410.4" textLength="1220" clip-path="url(#breeze-ci-image-verify-line-16)">───────────────────────────────────────────────────────────── [...]
-</text><text class="breeze-ci-image-verify-r5" x="0" y="434.8" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-17)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="434.8" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-17)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="434.8" textLength="97.6" clip-path="url(#breeze-ci-image-verify-line-17)">-verbose</text><text class="breeze-ci-image-verify-r6" x="158.6" y="434.8" textLength="24.4" c [...]
-</text><text class="breeze-ci-image-verify-r5" x="0" y="459.2" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-18)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="459.2" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-18)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="459.2" textLength="48.8" clip-path="url(#breeze-ci-image-verify-line-18)">-dry</text><text class="breeze-ci-image-verify-r4" x="85.4" y="459.2" textLength="48.8" clip-p [...]
-</text><text class="breeze-ci-image-verify-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-19)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="483.6" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-19)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="483.6" textLength="61" clip-path="url(#breeze-ci-image-verify-line-19)">-help</text><text class="breeze-ci-image-verify-r6" x="158.6" y="483.6" textLength="24.4" clip-p [...]
-</text><text class="breeze-ci-image-verify-r5" x="0" y="508" textLength="1464" clip-path="url(#breeze-ci-image-verify-line-20)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-verify-r1" x="1464" y="508" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-20)">
+</text><text class="breeze-ci-image-verify-r5" x="0" y="312.8" textLength="24.4" clip-path="url(#breeze-ci-image-verify-line-12)">╭─</text><text class="breeze-ci-image-verify-r5" x="24.4" y="312.8" textLength="219.6" clip-path="url(#breeze-ci-image-verify-line-12)"> Parallel running </text><text class="breeze-ci-image-verify-r5" x="244" y="312.8" textLength="1195.6" clip-path="url(#breeze-ci-image-verify-line-12)">─────────────────────────────────────────────────────────── [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-13)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="337.2" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-13)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="337.2" textLength="48.8" clip-path="url(#breeze-ci-image-verify-line-13)">-run</text><text class="breeze-ci-image-verify-r4" x="85.4" y="337.2" textLength="146.4" clip- [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-14)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="361.6" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-14)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="361.6" textLength="146.4" clip-path="url(#breeze-ci-image-verify-line-14)">-parallelism</text><text class="breeze-ci-image-verify-r1" x="378.2" y="361.6" textLength="91 [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="386" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-15)">│</text><text class="breeze-ci-image-verify-r7" x="378.2" y="386" textLength="915" clip-path="url(#breeze-ci-image-verify-line-15)">(INTEGER RANGE)                                   &# [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="410.4" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-16)">│</text><text class="breeze-ci-image-verify-r5" x="378.2" y="410.4" textLength="915" clip-path="url(#breeze-ci-image-verify-line-16)">[default: 4; 1<=x<=8]                                 [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="434.8" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-17)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="434.8" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-17)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="434.8" textLength="85.4" clip-path="url(#breeze-ci-image-verify-line-17)">-python</text><text class="breeze-ci-image-verify-r4" x="122" y="434.8" textLength="109.8" cli [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="459.2" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-18)">│</text><text class="breeze-ci-image-verify-r5" x="378.2" y="459.2" textLength="951.6" clip-path="url(#breeze-ci-image-verify-line-18)">[default: 3.8 3.9 3.10 3.11]                              [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-19)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="483.6" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-19)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="483.6" textLength="61" clip-path="url(#breeze-ci-image-verify-line-19)">-skip</text><text class="breeze-ci-image-verify-r4" x="97.6" y="483.6" textLength="97.6" clip-pa [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="508" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-20)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="508" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-20)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="508" textLength="73.2" clip-path="url(#breeze-ci-image-verify-line-20)">-debug</text><text class="breeze-ci-image-verify-r4" x="109.8" y="508" textLength="122" clip-path="u [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-21)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="532.4" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-21)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="532.4" textLength="97.6" clip-path="url(#breeze-ci-image-verify-line-21)">-include</text><text class="breeze-ci-image-verify-r4" x="134.2" y="532.4" textLength="195.2" [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="556.8" textLength="1464" clip-path="url(#breeze-ci-image-verify-line-22)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-verify-r1" x="1464" y="556.8" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-22)">
+</text><text class="breeze-ci-image-verify-r5" x="0" y="581.2" textLength="24.4" clip-path="url(#breeze-ci-image-verify-line-23)">╭─</text><text class="breeze-ci-image-verify-r5" x="24.4" y="581.2" textLength="280.6" clip-path="url(#breeze-ci-image-verify-line-23)"> Github authentication </text><text class="breeze-ci-image-verify-r5" x="305" y="581.2" textLength="1134.6" clip-path="url(#breeze-ci-image-verify-line-23)">────────────────────────────────────────────────────── [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="605.6" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-24)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="605.6" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-24)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="605.6" textLength="85.4" clip-path="url(#breeze-ci-image-verify-line-24)">-github</text><text class="breeze-ci-image-verify-r4" x="122" y="605.6" textLength="134.2" cli [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="630" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-25)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="630" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-25)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="630" textLength="85.4" clip-path="url(#breeze-ci-image-verify-line-25)">-github</text><text class="breeze-ci-image-verify-r4" x="122" y="630" textLength="73.2" clip-path="u [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="654.4" textLength="1464" clip-path="url(#breeze-ci-image-verify-line-26)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-verify-r1" x="1464" y="654.4" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-26)">
+</text><text class="breeze-ci-image-verify-r5" x="0" y="678.8" textLength="24.4" clip-path="url(#breeze-ci-image-verify-line-27)">╭─</text><text class="breeze-ci-image-verify-r5" x="24.4" y="678.8" textLength="195.2" clip-path="url(#breeze-ci-image-verify-line-27)"> Common options </text><text class="breeze-ci-image-verify-r5" x="219.6" y="678.8" textLength="1220" clip-path="url(#breeze-ci-image-verify-line-27)">───────────────────────────────────────────────────────────── [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="703.2" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-28)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="703.2" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-28)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="703.2" textLength="97.6" clip-path="url(#breeze-ci-image-verify-line-28)">-verbose</text><text class="breeze-ci-image-verify-r6" x="158.6" y="703.2" textLength="24.4" c [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="727.6" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-29)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="727.6" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-29)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="727.6" textLength="48.8" clip-path="url(#breeze-ci-image-verify-line-29)">-dry</text><text class="breeze-ci-image-verify-r4" x="85.4" y="727.6" textLength="48.8" clip-p [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="752" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-30)">│</text><text class="breeze-ci-image-verify-r4" x="24.4" y="752" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-30)">-</text><text class="breeze-ci-image-verify-r4" x="36.6" y="752" textLength="61" clip-path="url(#breeze-ci-image-verify-line-30)">-help</text><text class="breeze-ci-image-verify-r6" x="158.6" y="752" textLength="24.4" clip-path="url [...]
+</text><text class="breeze-ci-image-verify-r5" x="0" y="776.4" textLength="1464" clip-path="url(#breeze-ci-image-verify-line-31)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-verify-r1" x="1464" y="776.4" textLength="12.2" clip-path="url(#breeze-ci-image-verify-line-31)">
</text>
</g>
</g>
diff --git a/images/breeze/output_ci-image_verify.txt b/images/breeze/output_ci-image_verify.txt
index 6b16a49525..f454443f68 100644
--- a/images/breeze/output_ci-image_verify.txt
+++ b/images/breeze/output_ci-image_verify.txt
@@ -1 +1 @@
-c90dc7e20fce2351eb89d8d1ebbd35e7
+707a149f99bd49d37be5b8d0db844d69
diff --git a/images/breeze/output_prod-image_verify.svg b/images/breeze/output_prod-image_verify.svg
index 508fcbd34c..5633e0226c 100644
--- a/images/breeze/output_prod-image_verify.svg
+++ b/images/breeze/output_prod-image_verify.svg
@@ -1,4 +1,4 @@
-<svg class="rich-terminal" viewBox="0 0 1482 586.8" xmlns="http://www.w3.org/2000/svg">
+<svg class="rich-terminal" viewBox="0 0 1482 855.1999999999999" xmlns="http://www.w3.org/2000/svg">
<!-- Generated with Rich https://www.textualize.io -->
<style>
@@ -43,7 +43,7 @@
<defs>
<clipPath id="breeze-prod-image-verify-clip-terminal">
- <rect x="0" y="0" width="1463.0" height="535.8" />
+ <rect x="0" y="0" width="1463.0" height="804.1999999999999" />
</clipPath>
<clipPath id="breeze-prod-image-verify-line-0">
<rect x="0" y="1.5" width="1464" height="24.65"/>
@@ -108,9 +108,42 @@
<clipPath id="breeze-prod-image-verify-line-20">
<rect x="0" y="489.5" width="1464" height="24.65"/>
</clipPath>
+<clipPath id="breeze-prod-image-verify-line-21">
+ <rect x="0" y="513.9" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-prod-image-verify-line-22">
+ <rect x="0" y="538.3" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-prod-image-verify-line-23">
+ <rect x="0" y="562.7" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-prod-image-verify-line-24">
+ <rect x="0" y="587.1" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-prod-image-verify-line-25">
+ <rect x="0" y="611.5" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-prod-image-verify-line-26">
+ <rect x="0" y="635.9" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-prod-image-verify-line-27">
+ <rect x="0" y="660.3" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-prod-image-verify-line-28">
+ <rect x="0" y="684.7" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-prod-image-verify-line-29">
+ <rect x="0" y="709.1" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-prod-image-verify-line-30">
+ <rect x="0" y="733.5" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-prod-image-verify-line-31">
+ <rect x="0" y="757.9" width="1464" height="24.65"/>
+ </clipPath>
</defs>
- <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="584.8" rx="8"/><text class="breeze-prod-image-verify-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: prod-image verify</text>
+ <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="853.2" rx="8"/><text class="breeze-prod-image-verify-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: prod-image verify</text>
<g transform="translate(26,22)">
<circle cx="0" cy="0" r="7" fill="#ff5f57"/>
<circle cx="22" cy="0" r="7" fill="#febc2e"/>
@@ -133,15 +166,26 @@
</text><text class="breeze-prod-image-verify-r5" x="0" y="264" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-10)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="264" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-10)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="264" textLength="73.2" clip-path="url(#breeze-prod-image-verify-line-10)">-image</text><text class="breeze-prod-image-verify-r4" x="109.8" y="264" textLength="48. [...]
</text><text class="breeze-prod-image-verify-r5" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-11)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="288.4" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-11)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="288.4" textLength="61" clip-path="url(#breeze-prod-image-verify-line-11)">-pull</text><text class="breeze-prod-image-verify-r1" x="244" y="288.4" textLength=" [...]
</text><text class="breeze-prod-image-verify-r5" x="0" y="312.8" textLength="1464" clip-path="url(#breeze-prod-image-verify-line-12)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-verify-r1" x="1464" y="312.8" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-12)">
-</text><text class="breeze-prod-image-verify-r5" x="0" y="337.2" textLength="24.4" clip-path="url(#breeze-prod-image-verify-line-13)">╭─</text><text class="breeze-prod-image-verify-r5" x="24.4" y="337.2" textLength="280.6" clip-path="url(#breeze-prod-image-verify-line-13)"> Github authentication </text><text class="breeze-prod-image-verify-r5" x="305" y="337.2" textLength="1134.6" clip-path="url(#breeze-prod-image-verify-line-13)">────────────────────────────────────────── [...]
-</text><text class="breeze-prod-image-verify-r5" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-14)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="361.6" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-14)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="361.6" textLength="85.4" clip-path="url(#breeze-prod-image-verify-line-14)">-github</text><text class="breeze-prod-image-verify-r4" x="122" y="361.6" textLeng [...]
-</text><text class="breeze-prod-image-verify-r5" x="0" y="386" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-15)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="386" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-15)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="386" textLength="85.4" clip-path="url(#breeze-prod-image-verify-line-15)">-github</text><text class="breeze-prod-image-verify-r4" x="122" y="386" textLength="73.2 [...]
-</text><text class="breeze-prod-image-verify-r5" x="0" y="410.4" textLength="1464" clip-path="url(#breeze-prod-image-verify-line-16)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-verify-r1" x="1464" y="410.4" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-16)">
-</text><text class="breeze-prod-image-verify-r5" x="0" y="434.8" textLength="24.4" clip-path="url(#breeze-prod-image-verify-line-17)">╭─</text><text class="breeze-prod-image-verify-r5" x="24.4" y="434.8" textLength="195.2" clip-path="url(#breeze-prod-image-verify-line-17)"> Common options </text><text class="breeze-prod-image-verify-r5" x="219.6" y="434.8" textLength="1220" clip-path="url(#breeze-prod-image-verify-line-17)">───────────────────────────────────────────────── [...]
-</text><text class="breeze-prod-image-verify-r5" x="0" y="459.2" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-18)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="459.2" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-18)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="459.2" textLength="97.6" clip-path="url(#breeze-prod-image-verify-line-18)">-verbose</text><text class="breeze-prod-image-verify-r6" x="158.6" y="459.2" textL [...]
-</text><text class="breeze-prod-image-verify-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-19)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="483.6" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-19)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="483.6" textLength="48.8" clip-path="url(#breeze-prod-image-verify-line-19)">-dry</text><text class="breeze-prod-image-verify-r4" x="85.4" y="483.6" textLength [...]
-</text><text class="breeze-prod-image-verify-r5" x="0" y="508" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-20)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="508" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-20)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="508" textLength="61" clip-path="url(#breeze-prod-image-verify-line-20)">-help</text><text class="breeze-prod-image-verify-r6" x="158.6" y="508" textLength="24.4" [...]
-</text><text class="breeze-prod-image-verify-r5" x="0" y="532.4" textLength="1464" clip-path="url(#breeze-prod-image-verify-line-21)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-verify-r1" x="1464" y="532.4" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-21)">
+</text><text class="breeze-prod-image-verify-r5" x="0" y="337.2" textLength="24.4" clip-path="url(#breeze-prod-image-verify-line-13)">╭─</text><text class="breeze-prod-image-verify-r5" x="24.4" y="337.2" textLength="219.6" clip-path="url(#breeze-prod-image-verify-line-13)"> Parallel running </text><text class="breeze-prod-image-verify-r5" x="244" y="337.2" textLength="1195.6" clip-path="url(#breeze-prod-image-verify-line-13)">─────────────────────────────────────────────── [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-14)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="361.6" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-14)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="361.6" textLength="48.8" clip-path="url(#breeze-prod-image-verify-line-14)">-run</text><text class="breeze-prod-image-verify-r4" x="85.4" y="361.6" textLength [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="386" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-15)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="386" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-15)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="386" textLength="146.4" clip-path="url(#breeze-prod-image-verify-line-15)">-parallelism</text><text class="breeze-prod-image-verify-r1" x="378.2" y="386" textLeng [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="410.4" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-16)">│</text><text class="breeze-prod-image-verify-r7" x="378.2" y="410.4" textLength="915" clip-path="url(#breeze-prod-image-verify-line-16)">(INTEGER RANGE)                                 &# [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="434.8" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-17)">│</text><text class="breeze-prod-image-verify-r5" x="378.2" y="434.8" textLength="915" clip-path="url(#breeze-prod-image-verify-line-17)">[default: 4; 1<=x<=8]                               [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="459.2" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-18)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="459.2" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-18)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="459.2" textLength="85.4" clip-path="url(#breeze-prod-image-verify-line-18)">-python</text><text class="breeze-prod-image-verify-r4" x="122" y="459.2" textLeng [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-19)">│</text><text class="breeze-prod-image-verify-r5" x="378.2" y="483.6" textLength="951.6" clip-path="url(#breeze-prod-image-verify-line-19)">[default: 3.8 3.9 3.10 3.11]                            &# [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="508" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-20)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="508" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-20)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="508" textLength="61" clip-path="url(#breeze-prod-image-verify-line-20)">-skip</text><text class="breeze-prod-image-verify-r4" x="97.6" y="508" textLength="97.6" c [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-21)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="532.4" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-21)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="532.4" textLength="73.2" clip-path="url(#breeze-prod-image-verify-line-21)">-debug</text><text class="breeze-prod-image-verify-r4" x="109.8" y="532.4" textLen [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="556.8" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-22)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="556.8" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-22)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="556.8" textLength="97.6" clip-path="url(#breeze-prod-image-verify-line-22)">-include</text><text class="breeze-prod-image-verify-r4" x="134.2" y="556.8" textL [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="581.2" textLength="1464" clip-path="url(#breeze-prod-image-verify-line-23)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-verify-r1" x="1464" y="581.2" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-23)">
+</text><text class="breeze-prod-image-verify-r5" x="0" y="605.6" textLength="24.4" clip-path="url(#breeze-prod-image-verify-line-24)">╭─</text><text class="breeze-prod-image-verify-r5" x="24.4" y="605.6" textLength="280.6" clip-path="url(#breeze-prod-image-verify-line-24)"> Github authentication </text><text class="breeze-prod-image-verify-r5" x="305" y="605.6" textLength="1134.6" clip-path="url(#breeze-prod-image-verify-line-24)">────────────────────────────────────────── [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="630" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-25)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="630" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-25)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="630" textLength="85.4" clip-path="url(#breeze-prod-image-verify-line-25)">-github</text><text class="breeze-prod-image-verify-r4" x="122" y="630" textLength="134. [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="654.4" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-26)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="654.4" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-26)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="654.4" textLength="85.4" clip-path="url(#breeze-prod-image-verify-line-26)">-github</text><text class="breeze-prod-image-verify-r4" x="122" y="654.4" textLeng [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="678.8" textLength="1464" clip-path="url(#breeze-prod-image-verify-line-27)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-verify-r1" x="1464" y="678.8" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-27)">
+</text><text class="breeze-prod-image-verify-r5" x="0" y="703.2" textLength="24.4" clip-path="url(#breeze-prod-image-verify-line-28)">╭─</text><text class="breeze-prod-image-verify-r5" x="24.4" y="703.2" textLength="195.2" clip-path="url(#breeze-prod-image-verify-line-28)"> Common options </text><text class="breeze-prod-image-verify-r5" x="219.6" y="703.2" textLength="1220" clip-path="url(#breeze-prod-image-verify-line-28)">───────────────────────────────────────────────── [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="727.6" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-29)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="727.6" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-29)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="727.6" textLength="97.6" clip-path="url(#breeze-prod-image-verify-line-29)">-verbose</text><text class="breeze-prod-image-verify-r6" x="158.6" y="727.6" textL [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="752" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-30)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="752" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-30)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="752" textLength="48.8" clip-path="url(#breeze-prod-image-verify-line-30)">-dry</text><text class="breeze-prod-image-verify-r4" x="85.4" y="752" textLength="48.8" [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="776.4" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-31)">│</text><text class="breeze-prod-image-verify-r4" x="24.4" y="776.4" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-31)">-</text><text class="breeze-prod-image-verify-r4" x="36.6" y="776.4" textLength="61" clip-path="url(#breeze-prod-image-verify-line-31)">-help</text><text class="breeze-prod-image-verify-r6" x="158.6" y="776.4" textLength [...]
+</text><text class="breeze-prod-image-verify-r5" x="0" y="800.8" textLength="1464" clip-path="url(#breeze-prod-image-verify-line-32)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-verify-r1" x="1464" y="800.8" textLength="12.2" clip-path="url(#breeze-prod-image-verify-line-32)">
</text>
</g>
</g>
diff --git a/images/breeze/output_prod-image_verify.txt b/images/breeze/output_prod-image_verify.txt
index 2b87281a7f..8301a9d978 100644
--- a/images/breeze/output_prod-image_verify.txt
+++ b/images/breeze/output_prod-image_verify.txt
@@ -1 +1 @@
-bd2b78738a7c388dbad6076c41a9f906
+3d7fdd3877862ce9bfad8018a5282745