You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2022/04/23 07:12:23 UTC
[airflow] branch main updated: Use new Breese for building, pulling and verifying the images. (#23104)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 8b6b0848a3 Use new Breese for building, pulling and verifying the images. (#23104)
8b6b0848a3 is described below
commit 8b6b0848a3cacf9999477d6af4d2a87463f03026
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Sat Apr 23 09:12:11 2022 +0200
Use new Breese for building, pulling and verifying the images. (#23104)
We have the new Python-based breeze and we want to replace all
the functionality used from the old Breeze with the new one.
The most important is image management. It is used in many
places and this PR replaces all the places and removes image
builidng, pushing, pulling to use the new Breeze everywhere.
That includes:
* Building and pushing CI image on CI
* Building and pushing PROD image on CI which includes:
* building Airflow packages
* building Provider packages
* waiting for images in parallel (both CI and and PROD)
* veifying the images (both CI and PROD)
All those commands have been moved to breeze.py and that required
some modification in the Breeze parameter handling - mainly related
to adding spaces when help was displayed because long list of
packages looked very bad in help output.
It's been easier to implement it in one big PR as using
image building and pulling was deeply embedded in many scripts.
With this change all the scripts in CI use directly breeze
and combine using command line parameters with evn variables
directly in the job that execute the breeze commands which
makes it much easier to understand what is going on and
repeat it locally using Breeze.
Fixes: #22825
Fixes: #23077
Fixes: #23076
Fixes: #22829
Fixes: #22828
Fixes: #22826
Fixes: #20961
Fixes: #23102
Fixes: #21098
---
.github/boring-cyborg.yml | 1 -
.github/workflows/build-images.yml | 95 +-
.github/workflows/ci.yml | 398 ++++---
.gitignore | 3 +-
.pre-commit-config.yaml | 64 +-
BREEZE.rst | 143 ++-
CI.rst | 7 +-
CONTRIBUTING.rst | 2 -
Dockerfile.ci | 28 +-
INSTALL | 2 -
README.md | 2 +-
STATIC_CODE_CHECKS.rst | 6 +-
TESTING.rst | 6 +-
.../providers/amazon/.latest-doc-only-change.txt | 1 +
.../providers/asana/.latest-doc-only-change.txt | 2 +-
.../providers/cloudant/.latest-doc-only-change.txt | 2 +-
.../cncf/kubernetes/.latest-doc-only-change.txt | 1 +
airflow/providers/databricks/CHANGELOG.rst | 3 +
.../providers/datadog/.latest-doc-only-change.txt | 2 +-
.../dbt/cloud/.latest-doc-only-change.txt | 1 +
.../providers/dingding/.latest-doc-only-change.txt | 2 +-
.../providers/docker/.latest-doc-only-change.txt | 2 +-
.../elasticsearch/.latest-doc-only-change.txt | 2 +-
.../providers/github/.latest-doc-only-change.txt | 1 +
.../providers/google/.latest-doc-only-change.txt | 1 +
airflow/providers/hashicorp/CHANGELOG.rst | 3 +
airflow/providers/jdbc/.latest-doc-only-change.txt | 2 +-
.../providers/jenkins/.latest-doc-only-change.txt | 2 +-
.../microsoft/azure/.latest-doc-only-change.txt | 2 +-
.../microsoft/winrm/.latest-doc-only-change.txt | 2 +-
.../providers/oracle/.latest-doc-only-change.txt | 2 +-
.../providers/presto/.latest-doc-only-change.txt | 2 +-
.../providers/qubole/.latest-doc-only-change.txt | 2 +-
.../providers/trino/.latest-doc-only-change.txt | 2 +-
breeze-complete | 66 +-
breeze-legacy | 957 +--------------
dev/PROVIDER_PACKAGE_DETAILS.md | 33 +-
dev/README_RELEASE_AIRFLOW.md | 10 +-
dev/README_RELEASE_PROVIDER_PACKAGES.md | 13 +-
dev/REFRESHING_CI_CACHE.md | 77 +-
dev/breeze/src/airflow_breeze/breeze.py | 1216 ++++++++++++++++----
.../build_image/ci/build_ci_image.py | 247 ++--
.../build_image/ci/build_ci_params.py | 42 +-
.../build_image/prod/build_prod_image.py | 96 +-
.../build_image/prod/build_prod_params.py | 22 +-
dev/breeze/src/airflow_breeze/global_constants.py | 54 +-
dev/breeze/src/airflow_breeze/shell/enter_shell.py | 156 +--
.../src/airflow_breeze/shell/shell_params.py | 11 +-
.../breeze/src/airflow_breeze/utils/ci_group.py | 37 +-
dev/breeze/src/airflow_breeze/utils/confirm.py | 5 +-
dev/breeze/src/airflow_breeze/utils/constraints.py | 79 ++
.../airflow_breeze/utils/docker_command_utils.py | 73 +-
.../src/airflow_breeze/utils/md5_build_check.py | 9 +-
dev/breeze/src/airflow_breeze/utils/parallel.py | 81 ++
dev/breeze/src/airflow_breeze/utils/path_utils.py | 2 +-
dev/breeze/src/airflow_breeze/utils/pulll_image.py | 170 +++
dev/breeze/src/airflow_breeze/utils/registry.py | 17 +-
dev/breeze/src/airflow_breeze/utils/run_tests.py | 80 ++
dev/breeze/src/airflow_breeze/utils/run_utils.py | 11 +-
.../breeze/tests/conftest.py | 16 +-
dev/breeze/tests/test_cache.py | 2 +-
dev/breeze/tests/test_find_airflow_directory.py | 2 +-
dev/breeze/tests/test_prod_image.py | 2 +-
dev/provider_packages/README.md | 95 +-
.../enter_breeze_provider_package_tests.sh | 38 -
dev/provider_packages/prepare_provider_packages.py | 62 +-
dev/refresh_images.sh | 16 +-
docker-context-files/{README.md => .README.md} | 0
.../commits.rst | 3 +-
.../apache-airflow-providers-hashicorp/commits.rst | 3 +-
.../installation/supported-versions.rst | 2 +-
docs/docker-stack/build-arg-ref.rst | 2 -
docs/docker-stack/build.rst | 2 +-
images/breeze/output-breeze-commands.svg | 136 ---
images/breeze/output-build-docs.svg | 76 +-
images/breeze/output-build-image.svg | 47 +-
images/breeze/output-build-prod-image.svg | 61 +-
images/breeze/output-cleanup.svg | 16 +-
images/breeze/output-commands.svg | 73 +-
images/breeze/output-config.svg | 7 +-
...omplete.svg => output-docker-compose-tests.svg} | 22 +-
...t-shell.svg => output-generate-constraints.svg} | 43 +-
...lete.svg => output-prepare-airflow-package.svg} | 25 +-
...g => output-prepare-provider-documentation.svg} | 32 +-
...te.svg => output-prepare-provider-packages.svg} | 37 +-
.../{output-shell.svg => output-pull-image.svg} | 43 +-
...output-shell.svg => output-pull-prod-image.svg} | 43 +-
images/breeze/output-self-upgrade.svg | 10 +-
images/breeze/output-setup-autocomplete.svg | 2 +-
images/breeze/output-shell.svg | 31 +-
images/breeze/output-start-airflow.svg | 31 +-
images/breeze/output-static-checks.svg | 49 +-
...up-autocomplete.svg => output-verify-image.svg} | 22 +-
...tocomplete.svg => output-verify-prod-image.svg} | 22 +-
images/breeze/output-version.svg | 2 +-
.../ci/build_airflow/ci_build_airflow_packages.sh | 23 -
.../ci/constraints/ci_generate_all_constraints.sh | 50 -
scripts/ci/constraints/ci_generate_constraints.sh | 31 -
scripts/ci/docker-compose/_docker.env | 5 +-
scripts/ci/docker-compose/base.yml | 5 +-
scripts/ci/docker-compose/devcontainer.env | 4 +-
scripts/ci/docs/ci_docs.sh | 25 -
scripts/ci/docs/ci_docs_prepare.sh | 22 -
scripts/ci/images/ci_build_ci_image_on_ci.sh | 30 -
scripts/ci/images/ci_build_prod_image_on_ci.sh | 37 -
scripts/ci/images/ci_pull_ci_image_on_ci.sh | 34 -
scripts/ci/images/ci_push_production_images.sh | 34 -
.../ci_run_docker_compose_quick_start_test.sh | 28 -
scripts/ci/images/ci_run_docker_tests.py | 105 --
.../images/ci_wait_for_and_verify_all_ci_images.sh | 49 -
.../ci_wait_for_and_verify_all_prod_images.sh | 49 -
.../ci/images/ci_wait_for_and_verify_ci_image.sh | 43 -
.../ci/images/ci_wait_for_and_verify_prod_image.sh | 40 -
scripts/ci/images/self_terminate.sh | 0
...tup_cluster_and_deploy_airflow_to_kubernetes.sh | 2 -
scripts/ci/libraries/_all_libs.sh | 6 -
scripts/ci/libraries/_build_images.sh | 862 --------------
scripts/ci/libraries/_initialization.sh | 81 +-
scripts/ci/libraries/_runs.sh | 85 --
scripts/ci/libraries/_script_init.sh | 4 +-
scripts/ci/libraries/_verify_image.sh | 37 -
.../ci/pre_commit/pre_commit_breeze_cmd_line.py | 2 +-
scripts/ci/pre_commit/pre_commit_check_license.sh | 2 +-
.../pre_commit_check_order_dockerfile_extras.py | 69 +-
scripts/ci/pre_commit/pre_commit_flake8.py | 60 +
scripts/ci/pre_commit/pre_commit_flake8.sh | 41 -
scripts/ci/pre_commit/pre_commit_helm_lint.sh | 2 +-
.../pre_commit_inline_scripts_in_docker.py | 2 +-
scripts/ci/pre_commit/pre_commit_insert_extras.py | 66 +-
.../ci/pre_commit/pre_commit_lint_dockerfile.sh | 2 +-
.../pre_commit/pre_commit_migration_reference.py | 163 +--
.../pre_commit/pre_commit_migration_reference.sh | 42 -
scripts/ci/pre_commit/pre_commit_mypy.py | 60 +
scripts/ci/pre_commit/pre_commit_mypy.sh | 44 -
...ersions.py => pre_commit_supported_versions.py} | 0
scripts/ci/pre_commit/pre_commit_ui_lint.py | 57 +
.../pre_commit_update_breeze_config_hash.py | 2 +-
scripts/ci/pre_commit/pre_commit_www_lint.py | 57 +
.../ci_install_and_test_provider_packages.sh | 7 +-
.../ci_prepare_provider_documentation.sh | 23 -
.../ci_prepare_provider_packages.sh | 23 -
scripts/ci/static_checks/run_static_checks.sh | 4 -
scripts/ci/static_checks/ui_lint.sh | 31 -
scripts/ci/static_checks/www_lint.sh | 31 -
scripts/ci/testing/ci_run_airflow_testing.sh | 4 -
scripts/ci/testing/ci_run_quarantined_tests.sh | 4 -
scripts/ci/testing/run_downgrade_test.sh | 4 -
scripts/ci/testing/run_offline_sql_test.sh | 4 -
scripts/docker/entrypoint_ci.sh | 14 +-
scripts/in_container/_in_container_utils.sh | 5 +-
.../run_install_and_test_provider_packages.sh | 18 +-
.../run_migration_reference.py} | 2 +-
.../in_container/run_prepare_airflow_packages.sh | 10 -
.../run_prepare_provider_documentation.sh | 26 +-
scripts/tools/confirm | 6 +-
tests/dag_processing/test_manager.py | 2 +-
.../providers/google/cloud/utils/base_gcp_mock.py | 4 +-
.../google/bigquery/example_bigquery_sensors.py | 8 +-
tests/utils/test_db_cleanup.py | 2 +-
159 files changed, 3337 insertions(+), 4692 deletions(-)
diff --git a/.github/boring-cyborg.yml b/.github/boring-cyborg.yml
index 38011f2874..607d4fb6cd 100644
--- a/.github/boring-cyborg.yml
+++ b/.github/boring-cyborg.yml
@@ -188,7 +188,6 @@ labelPRBasedOnFilePath:
area:production-image:
- Dockerfile
- docs/docker-stack/**/*
- - scripts/ci/libraries/_verify_image.sh
- docker_tests/**/*
# Various Flags to control behaviour of the "Labeler"
diff --git a/.github/workflows/build-images.yml b/.github/workflows/build-images.yml
index 79fc2ad154..04d295d369 100644
--- a/.github/workflows/build-images.yml
+++ b/.github/workflows/build-images.yml
@@ -24,7 +24,7 @@ permissions:
contents: read
env:
MOUNT_SELECTED_LOCAL_SOURCES: "false"
- FORCE_ANSWER_TO_QUESTIONS: "yes"
+ ANSWER: "yes"
CHECK_IMAGE_FOR_REBUILD: "true"
SKIP_CHECK_REMOTE_IMAGE: "true"
DEBIAN_VERSION: "bullseye"
@@ -41,13 +41,13 @@ env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
LOGIN_TO_GITHUB_REGISTRY: "true"
IMAGE_TAG_FOR_THE_BUILD: "${{ github.event.pull_request.head.sha || github.sha }}"
+ WITH_CI_GROUPS: "true"
concurrency:
group: build-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
-
build-info:
timeout-minutes: 10
name: "Build Info"
@@ -167,8 +167,8 @@ jobs:
github.event.pull_request.head.repo.full_name != 'apache/airflow'
env:
RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn)[0] }}
- BACKEND: sqlite
PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
+ BACKEND: sqlite
outputs: ${{toJSON(needs.build-info.outputs) }}
steps:
- uses: actions/checkout@v2
@@ -212,10 +212,10 @@ jobs:
# changed in the image built - we should only override those that are executed to build
# the image.
run: |
- rm -rf "scripts/ci"
- rm -rf "dev"
- mv "main-airflow/scripts/ci" "scripts"
- mv "main-airflow/dev" "."
+ rm -rfv "scripts/ci"
+ rm -rfv "dev"
+ mv -v "main-airflow/scripts/ci" "scripts"
+ mv -v "main-airflow/dev" "."
- uses: actions/setup-python@v2
with:
python-version: ${{ needs.build-info.outputs.defaultPythonVersion }}
@@ -224,19 +224,17 @@ jobs:
- run: ./scripts/ci/install_breeze.sh
- name: "Free space"
run: airflow-freespace
- - name: "Build/Push CI image ${{ matrix.python-version }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- run: >
- breeze build-image --push-image
- --upgrade-to-newer-dependencies "${{ needs.build-info.outputs.upgradeToNewerDependencies }}"
- --docker-cache "${{ needs.build-info.outputs.cacheDirective }}"
- --image-tag "${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- - name: >
- Push empty CI images to finish waiting jobs:
- ${{ matrix.python-version }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
+ - name: Build & Push CI image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+ run: breeze build-image --push-image
+ env:
+ UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgradeToNewerDependencies }}
+ DOCKER_CACHE: ${{ needs.build-info.outputs.cacheDirective }}
+ IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+ - name: Push empty CI image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
if: failure() || cancelled()
- run: >
- breeze build-image --push-image --empty-image
- --github-registry-push-image-tag "${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
+ run: breeze build-image --push-image --empty-image
+ env:
+ IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
- name: "Candidates for pip resolver backtrack triggers: ${{ matrix.python-version }}"
if: failure() || cancelled()
run: airflow-find-newer-dependencies --max-age 1 --python "${{ matrix.python-version }}"
@@ -257,10 +255,8 @@ jobs:
github.event.pull_request.head.repo.full_name != 'apache/airflow'
env:
RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn)[0] }}
- BACKEND: sqlite
PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
- DOCKER_CACHE: ${{ needs.build-info.outputs.cacheDirective }}
- VERSION_SUFFIX_FOR_PYPI: ".dev0"
+ BACKEND: sqlite
steps:
- uses: actions/checkout@v2
with:
@@ -303,10 +299,10 @@ jobs:
# changed in the image built - we should only override those that are executed to build
# the image.
run: |
- rm -rf "scripts/ci"
- rm -rf "dev"
- mv "main-airflow/scripts/ci" "scripts"
- mv "main-airflow/dev" "."
+ rm -rfv "scripts/ci"
+ rm -rfv "dev"
+ mv -v "main-airflow/scripts/ci" "scripts"
+ mv -v "main-airflow/dev" "."
- uses: actions/setup-python@v2
with:
python-version: ${{ needs.build-info.outputs.defaultPythonVersion }}
@@ -315,24 +311,39 @@ jobs:
- run: ./scripts/ci/install_breeze.sh
- name: "Free space"
run: airflow-freespace
- - name: "Pull CI image for PROD build: ${{ matrix.python-version }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- run: ./scripts/ci/images/ci_pull_ci_image_on_ci.sh
+ - name: >
+ Pull CI image for PROD build:
+ ${{ env.PYTHON_MAJOR_MINOR_VERSION }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+ run: breeze pull-image --tag-as-latest
env:
- GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- - name: "Build PROD image ${{ matrix.python-version }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- run: ./scripts/ci/images/ci_build_prod_image_on_ci.sh
+ # Always use default Python version of CI image for preparing packages
+ PYTHON_MAJOR_MINOR_VERSION: ${{ needs.build-info.outputs.defaultPythonVersion }}
+ IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+ - name: "Cleanup dist and context file"
+ run: rm -fv ./dist/* ./docker-context-files/*
+ - name: "Prepare providers packages"
+ run: >
+ breeze prepare-provider-packages
+ --package-list-file ./scripts/ci/installed_providers.txt
+ --package-format wheel
+ --version-suffix-for-pypi dev0
+ - name: "Prepare airflow package"
+ run: breeze prepare-airflow-package --package-format wheel --version-suffix-for-pypi dev0
+ - name: "Move dist packages to docker-context files"
+ run: mv -v ./dist/*.whl ./docker-context-files
+ - name: Build & Push PROD image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+ run: >
+ breeze build-prod-image
+ --push-image
+ --install-from-docker-context-files
+ --disable-airflow-repo-cache
+ --disable-pypi
env:
- GITHUB_REGISTRY_PULL_IMAGE_TAG: "latest"
- GITHUB_REGISTRY_PUSH_IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgradeToNewerDependencies }}
DOCKER_CACHE: ${{ needs.build-info.outputs.cacheDirective }}
- - name: "Push PROD image ${{ matrix.python-version }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- run: ./scripts/ci/images/ci_push_production_images.sh
- env:
- GITHUB_REGISTRY_PUSH_IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
- - name: >
- Push empty PROD images to finish waiting jobs:
- ${{ matrix.python-version }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
+ IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+ - name: Push empty PROD image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
if: failure() || cancelled()
- run: >
- breeze build-prod-image --push-image --empty-image --image-tag "${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
+ run: breeze build-prod-image --push-image --empty-image
+ env:
+ IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 129b1c7193..1961bca289 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -29,7 +29,7 @@ permissions:
contents: read
env:
MOUNT_SELECTED_LOCAL_SOURCES: "false"
- FORCE_ANSWER_TO_QUESTIONS: "yes"
+ ANSWER: "yes"
CHECK_IMAGE_FOR_REBUILD: "true"
SKIP_CHECK_REMOTE_IMAGE: "true"
DEBIAN_VERSION: "bullseye"
@@ -47,6 +47,7 @@ env:
LOGIN_TO_GITHUB_REGISTRY: "true"
ENABLE_TEST_COVERAGE: "${{ github.event_name == 'push' }}"
IMAGE_TAG_FOR_THE_BUILD: "${{ github.event.pull_request.head.sha || github.sha }}"
+ WITH_CI_GROUPS: "true"
concurrency:
group: ci-${{ github.event.pull_request.number || github.ref }}
@@ -276,7 +277,6 @@ jobs:
PR_LABELS: ${{ steps.get-latest-pr-labels.outputs.pullRequestLabels }}
GITHUB_CONTEXT: ${{ toJson(github) }}
-
build-ci-images:
permissions:
packages: write
@@ -290,9 +290,6 @@ jobs:
fail-fast: true
env:
RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn)[0] }}
- BACKEND: sqlite
- PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
- outputs: ${{toJSON(needs.build-info.outputs) }}
steps:
- uses: actions/checkout@v2
with:
@@ -327,12 +324,13 @@ jobs:
- name: "Free space"
run: airflow-freespace
if: needs.build-info.outputs.inWorkflowBuild == 'true'
- - name: "Build/Push CI image ${{ matrix.python-version }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- run: >
- breeze build-image --push-image
- --upgrade-to-newer-dependencies "${{ needs.build-info.outputs.upgradeToNewerDependencies }}"
- --docker-cache "${{ needs.build-info.outputs.cacheDirective }}"
- --image-tag "${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
+ - name: Build & Push CI image ${{ matrix.python-version }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+ run: breeze build-image --push-image
+ env:
+ PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
+ UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgradeToNewerDependencies }}
+ DOCKER_CACHE: ${{ needs.build-info.outputs.cacheDirective }}
+ IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
if: needs.build-info.outputs.inWorkflowBuild == 'true'
- name: "Candidates for pip resolver backtrack triggers: ${{ matrix.python-version }}"
if: failure() || cancelled()
@@ -354,7 +352,7 @@ jobs:
BACKEND: sqlite
PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
DOCKER_CACHE: ${{ needs.build-info.outputs.cacheDirective }}
- VERSION_SUFFIX_FOR_PYPI: ".dev0"
+ VERSION_SUFFIX_FOR_PYPI: "dev0"
steps:
- uses: actions/checkout@v2
with:
@@ -389,28 +387,46 @@ jobs:
- name: "Free space"
run: airflow-freespace
if: needs.build-info.outputs.inWorkflowBuild == 'true'
- - name: "Pull CI image for PROD build: ${{ matrix.python-version }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- run: ./scripts/ci/images/ci_pull_ci_image_on_ci.sh
+ - name: >
+ Pull CI image for PROD build:
+ ${{ needs.build-info.outputs.defaultPythonVersion }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
+ run: breeze pull-image --tag-as-latest
env:
- GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
+ # Always use default Python version of CI image for preparing packages
+ PYTHON_MAJOR_MINOR_VERSION: ${{ needs.build-info.outputs.defaultPythonVersion }}
+ IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
if: needs.build-info.outputs.inWorkflowBuild == 'true'
- - name: "Build PROD image ${{ matrix.python-version }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- run: ./scripts/ci/images/ci_build_prod_image_on_ci.sh
+ - name: "Cleanup dist and context file"
+ run: rm -fv ./dist/* ./docker-context-files/*
if: needs.build-info.outputs.inWorkflowBuild == 'true'
+ - name: "Prepare providers packages"
+ run: >
+ breeze prepare-provider-packages
+ --package-list-file ./scripts/ci/installed_providers.txt
+ --package-format wheel --version-suffix-for-pypi dev0
+ if: needs.build-info.outputs.inWorkflowBuild == 'true'
+ - name: "Prepare airflow package"
+ run: breeze prepare-airflow-package --package-format wheel --version-suffix-for-pypi dev0
+ if: needs.build-info.outputs.inWorkflowBuild == 'true'
+ - name: "Move dist packages to docker-context files"
+ run: mv -v ./dist/*.whl ./docker-context-files
+ if: needs.build-info.outputs.inWorkflowBuild == 'true'
+ - name: Build & Push PROD image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+ run: >
+ breeze build-prod-image
+ --push-image
+ --install-from-docker-context-files
+ --disable-airflow-repo-cache
+ --disable-pypi
env:
- GITHUB_REGISTRY_PULL_IMAGE_TAG: "latest"
- GITHUB_REGISTRY_PUSH_IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgradeToNewerDependencies }}
DOCKER_CACHE: ${{ needs.build-info.outputs.cacheDirective }}
- - name: "Push PROD image ${{ matrix.python-version }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- run: ./scripts/ci/images/ci_push_production_images.sh
+ IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
if: needs.build-info.outputs.inWorkflowBuild == 'true'
- env:
- GITHUB_REGISTRY_PUSH_IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
run-new-breeze-tests:
timeout-minutes: 10
- name: breeze tests
+ name: Breeze unit tests
runs-on: ${{ fromJson(needs.build-info.outputs.runsOn) }}
needs: [build-info]
steps:
@@ -510,35 +526,6 @@ jobs:
python -m pip install -r ./docker_tests/requirements.txt &&
python -m pytest docker_tests/test_examples_of_prod_image_building.py -n auto --color=yes
- test-docker-compose-quick-start:
- timeout-minutes: 60
- name: "Test docker-compose quick start"
- runs-on: ${{ fromJson(needs.build-info.outputs.runsOn) }}
- needs: [build-info, wait-for-prod-images]
- if: needs.build-info.outputs.image-build == 'true'
- steps:
- - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
- uses: actions/checkout@v2
- with:
- fetch-depth: 2
- persist-credentials: false
- - name: "Setup python"
- uses: actions/setup-python@v2
- with:
- python-version: ${{ needs.build-info.outputs.defaultPythonVersion }}
- cache: 'pip'
- cache-dependency-path: ./dev/breeze/setup*
- - run: ./scripts/ci/install_breeze.sh
- - name: "Free space"
- run: airflow-freespace
- - name: "Cache virtualenv environment"
- uses: actions/cache@v2
- with:
- path: '.build/.docker_venv'
- key: ${{ runner.os }}-docker-venv-${{ hashFiles('scripts/ci/images/ci_run_docker_tests.py') }}
- - name: "Test docker-compose quick start"
- run: ./scripts/ci/images/ci_run_docker_compose_quick_start_test.sh
-
wait-for-ci-images:
timeout-minutes: 120
name: "Wait for CI images"
@@ -562,26 +549,12 @@ jobs:
- run: ./scripts/ci/install_breeze.sh
- name: "Free space"
run: airflow-freespace
- - name: "Cache virtualenv environment"
- uses: actions/cache@v2
- with:
- path: '.build/.docker_venv'
- key: ${{ runner.os }}-docker-venv-${{ hashFiles('scripts/ci/images/ci_run_docker_tests.py') }}
- - name: >
- Wait for CI images
- ${{ needs.build-info.outputs.pythonVersions }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+ - name: Wait for CI images ${{ env.PYTHON_VERSIONS }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
id: wait-for-images
- # We wait for the images to be available either from the "build-images.yml" workflow or
- # from the build-ci-image above.
- # We are utilising single job to wait for all images because this job merely waits
- # for the images to be available and run tests with the images.
- run: ./scripts/ci/images/ci_wait_for_and_verify_all_ci_images.sh
+ run: breeze pull-image --run-in-parallel --verify-image --wait-for-image
env:
- CURRENT_PYTHON_MAJOR_MINOR_VERSIONS_AS_STRING: >
- ${{needs.build-info.outputs.pythonVersionsListAsString}}
- GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- VERIFY_IMAGE: "true"
- PR_LABELS: "${{ needs.build-info.outputs.pullRequestLabels }}"
+ PYTHON_VERSIONS: ${{ needs.build-info.outputs.pythonVersionsListAsString }}
+ IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
static-checks:
timeout-minutes: 30
@@ -590,8 +563,7 @@ jobs:
needs: [build-info, wait-for-ci-images]
env:
RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn) }}
- MOUNT_SELECTED_LOCAL_SOURCES: "true"
- PYTHON_MAJOR_MINOR_VERSION: ${{needs.build-info.outputs.defaultPythonVersion}}
+ PYTHON_MAJOR_MINOR_VERSION: ${{ needs.build-info.outputs.defaultPythonVersion }}
if: needs.build-info.outputs.basic-checks-only == 'false'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
@@ -607,10 +579,11 @@ jobs:
- run: ./scripts/ci/install_breeze.sh
- name: "Free space"
run: airflow-freespace
- - name: "Pull CI image ${{env.PYTHON_MAJOR_MINOR_VERSION}}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- run: ./scripts/ci/images/ci_pull_ci_image_on_ci.sh
+ - name: >
+ Pull CI image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+ run: breeze pull-image --tag-as-latest
env:
- GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
+ IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
- name: "Get Python version"
run: "echo \"::set-output name=host-python-version::$(python -c
'import platform; print(platform.python_version())')\""
@@ -654,7 +627,6 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn) }}
SKIP: "build,mypy,flake8,identity"
MOUNT_SELECTED_LOCAL_SOURCES: "true"
- PYTHON_MAJOR_MINOR_VERSION: ${{needs.build-info.outputs.defaultPythonVersion}}
if: needs.build-info.outputs.basic-checks-only == 'true'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
@@ -668,8 +640,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
cache: 'pip'
cache-dependency-path: ./dev/breeze/setup*
- run: ./scripts/ci/install_breeze.sh
- - name: >
- Fetch incoming commit ${{ github.sha }} with its parent
+ - name: Fetch incoming commit ${{ github.sha }} with its parent
uses: actions/checkout@v2
with:
ref: ${{ github.sha }}
@@ -706,6 +677,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
if: needs.build-info.outputs.docs-build == 'true'
env:
RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn) }}
+ PYTHON_MAJOR_MINOR_VERSION: ${{ needs.build-info.outputs.defaultPythonVersion }}
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
@@ -720,22 +692,20 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
- run: ./scripts/ci/install_breeze.sh
- name: "Free space"
run: airflow-freespace
- - name: "Pull CI image ${{env.PYTHON_MAJOR_MINOR_VERSION}}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- run: ./scripts/ci/images/ci_pull_ci_image_on_ci.sh
+ - name: Pull CI image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+ run: breeze pull-image --tag-as-latest
env:
- GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- - name: "Fetch inventory versions"
- run: ./scripts/ci/docs/ci_docs_prepare.sh
+ IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
- uses: actions/cache@v2
id: cache-doc-inventories
with:
path: ./docs/_inventory_cache/
- key: docs-inventory-v1-${{ hashFiles('constraints.txt') }}
+ key: docs-inventory-${{ hashFiles('setup.py','setup.cfg','pyproject.toml;') }}
restore-keys: |
- docs-inventory-${{ hashFiles('constraints.txt') }}
+ docs-inventory-${{ hashFiles('setup.py','setup.cfg','pyproject.toml;') }}
docs-inventory-
- name: "Build docs"
- run: ./scripts/ci/docs/ci_docs.sh
+ run: breeze build-docs
- name: Configure AWS credentials
uses: ./.github/actions/configure-aws-credentials
if: >
@@ -758,11 +728,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
needs: [build-info, wait-for-ci-images]
env:
RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn) }}
- AIRFLOW_EXTRAS: "all"
- PYTHON_MAJOR_MINOR_VERSION: ${{needs.build-info.outputs.defaultPythonVersion}}
- VERSION_SUFFIX_FOR_PYPI: ".dev0"
- NON_INTERACTIVE: "true"
- GENERATE_PROVIDERS_ISSUE: "true"
+ PYTHON_MAJOR_MINOR_VERSION: ${{ needs.build-info.outputs.defaultPythonVersion }}
if: needs.build-info.outputs.image-build == 'true' && needs.build-info.outputs.default-branch == 'main'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
@@ -779,20 +745,19 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
- run: ./scripts/ci/install_breeze.sh
- name: "Free space"
run: airflow-freespace
- - name: "Pull CI image ${{env.PYTHON_MAJOR_MINOR_VERSION}}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- run: ./scripts/ci/images/ci_pull_ci_image_on_ci.sh
+ - name: >
+ Pull CI image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+ run: breeze pull-image --tag-as-latest
env:
- GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
+ IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+ - name: "Cleanup dist files"
+ run: rm -fv ./dist/*
- name: "Prepare provider documentation"
- run: ./scripts/ci/provider_packages/ci_prepare_provider_documentation.sh
+ run: breeze prepare-provider-documentation --answer yes
- name: "Prepare provider packages: wheel"
- run: ./scripts/ci/provider_packages/ci_prepare_provider_packages.sh
- env:
- PACKAGE_FORMAT: "wheel"
+ run: breeze prepare-provider-packages --package-format wheel --version-suffix-for-pypi dev0
- name: "Prepare airflow package: wheel"
- run: ./scripts/ci/build_airflow/ci_build_airflow_packages.sh
- env:
- PACKAGE_FORMAT: "wheel"
+ run: breeze prepare-airflow-package --package-format wheel --version-suffix-for-pypi dev0
- name: "Install and test provider packages and airflow via wheel files"
run: ./scripts/ci/provider_packages/ci_install_and_test_provider_packages.sh
env:
@@ -804,7 +769,6 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
run: ./scripts/ci/provider_packages/ci_install_and_test_provider_packages.sh
env:
USE_AIRFLOW_VERSION: "2.1.0"
- SKIP_TWINE_CHECK: "true"
PACKAGE_FORMAT: "wheel"
prepare-test-provider-packages-sdist:
@@ -814,11 +778,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
needs: [build-info, wait-for-ci-images]
env:
RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn) }}
- AIRFLOW_EXTRAS: "all"
- PYTHON_MAJOR_MINOR_VERSION: ${{needs.build-info.outputs.defaultPythonVersion}}
- VERSION_SUFFIX_FOR_PYPI: ".dev0"
- NON_INTERACTIVE: "true"
- GENERATE_PROVIDERS_ISSUE: "true"
+ PYTHON_MAJOR_MINOR_VERSION: ${{ needs.build-info.outputs.defaultPythonVersion }}
if: needs.build-info.outputs.image-build == 'true' && needs.build-info.outputs.default-branch == 'main'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
@@ -835,18 +795,17 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
- run: ./scripts/ci/install_breeze.sh
- name: "Free space"
run: airflow-freespace
- - name: "Pull CI image ${{env.PYTHON_MAJOR_MINOR_VERSION}}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- run: ./scripts/ci/images/ci_pull_ci_image_on_ci.sh
+ - name: >
+ Pull CI image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+ run: breeze pull-image --tag-as-latest
env:
- GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
+ IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+ - name: "Cleanup dist files"
+ run: rm -fv ./dist/*
- name: "Prepare provider packages: sdist"
- run: ./scripts/ci/provider_packages/ci_prepare_provider_packages.sh
- env:
- PACKAGE_FORMAT: "sdist"
+ run: breeze prepare-provider-packages --package-format sdist --version-suffix-for-pypi dev0
- name: "Prepare airflow package: sdist"
- run: ./scripts/ci/build_airflow/ci_build_airflow_packages.sh
- env:
- PACKAGE_FORMAT: "sdist"
+ run: breeze prepare-airflow-package --package-format sdist --version-suffix-for-pypi dev0
- name: "Upload provider distribution artifacts"
uses: actions/upload-artifact@v2
with:
@@ -858,6 +817,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
env:
USE_AIRFLOW_VERSION: "sdist"
PACKAGE_FORMAT: "sdist"
+ AIRFLOW_EXTRAS: "all"
tests-helm:
timeout-minutes: 80
@@ -889,10 +849,11 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
- run: ./scripts/ci/install_breeze.sh
- name: "Free space"
run: airflow-freespace
- - name: "Pull CI image ${{env.PYTHON_MAJOR_MINOR_VERSION}}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- run: ./scripts/ci/images/ci_pull_ci_image_on_ci.sh
+ - name: >
+ Pull CI image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+ run: breeze pull-image --tag-as-latest
env:
- GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
+ IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
- name: "Tests: Helm"
run: ./scripts/ci/testing/ci_run_airflow_testing.sh
env:
@@ -915,8 +876,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
uses: actions/upload-artifact@v2
if: needs.build-info.outputs.runCoverage == 'true'
with:
- name: >
- coverage-helm
+ name: coverage-helm
path: "./files/coverage*.xml"
retention-days: 7
@@ -936,9 +896,9 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
env:
RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn) }}
BACKEND: postgres
- PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
POSTGRES_VERSION: ${{ matrix.postgres-version }}
TEST_TYPES: "${{needs.build-info.outputs.testTypes}}"
+ PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
if: needs.build-info.outputs.run-tests == 'true'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
@@ -954,10 +914,10 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
- run: ./scripts/ci/install_breeze.sh
- name: "Free space"
run: airflow-freespace
- - name: "Pull CI image ${{env.PYTHON_MAJOR_MINOR_VERSION}}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- run: ./scripts/ci/images/ci_pull_ci_image_on_ci.sh
+ - name: Pull CI image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+ run: breeze pull-image --tag-as-latest
env:
- GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
+ IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
- name: "Test downgrade"
run: ./scripts/ci/testing/run_downgrade_test.sh
- name: "Test Offline SQL generation"
@@ -984,8 +944,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
uses: actions/upload-artifact@v2
if: needs.build-info.outputs.runCoverage == 'true'
with:
- name: >
- coverage-postgres-${{matrix.python-version}}-${{matrix.postgres-version}}
+ name: coverage-postgres-${{matrix.python-version}}-${{matrix.postgres-version}}
path: "./files/coverage*.xml"
retention-days: 7
@@ -1004,9 +963,9 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
env:
RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn) }}
BACKEND: mysql
- PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
MYSQL_VERSION: ${{ matrix.mysql-version }}
TEST_TYPES: "${{needs.build-info.outputs.testTypes}}"
+ PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
if: needs.build-info.outputs.run-tests == 'true'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
@@ -1022,10 +981,10 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
- run: ./scripts/ci/install_breeze.sh
- name: "Free space"
run: airflow-freespace
- - name: "Pull CI image ${{env.PYTHON_MAJOR_MINOR_VERSION}}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- run: ./scripts/ci/images/ci_pull_ci_image_on_ci.sh
+ - name: Pull CI image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+ run: breeze pull-image --tag-as-latest
env:
- GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
+ IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
- name: "Test downgrade"
run: ./scripts/ci/testing/run_downgrade_test.sh
- name: "Test Offline SQL generation"
@@ -1071,9 +1030,9 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
env:
RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn) }}
BACKEND: mssql
- PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
MSSQL_VERSION: ${{ matrix.mssql-version }}
TEST_TYPES: "${{needs.build-info.outputs.testTypes}}"
+ PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
if: needs.build-info.outputs.run-tests == 'true'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
@@ -1089,10 +1048,10 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
- run: ./scripts/ci/install_breeze.sh
- name: "Free space"
run: airflow-freespace
- - name: "Pull CI image ${{env.PYTHON_MAJOR_MINOR_VERSION}}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- run: ./scripts/ci/images/ci_pull_ci_image_on_ci.sh
+ - name: Pull CI image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+ run: breeze pull-image --tag-as-latest
env:
- GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
+ IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
- name: "Test downgrade"
run: ./scripts/ci/testing/run_downgrade_test.sh
- name: "Tests: ${{needs.build-info.outputs.testTypes}}"
@@ -1135,8 +1094,8 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
env:
RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn) }}
BACKEND: sqlite
- PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
TEST_TYPES: "${{needs.build-info.outputs.testTypes}}"
+ PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
if: needs.build-info.outputs.run-tests == 'true'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
@@ -1152,10 +1111,10 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
- run: ./scripts/ci/install_breeze.sh
- name: "Free space"
run: airflow-freespace
- - name: "Pull CI image ${{env.PYTHON_MAJOR_MINOR_VERSION}}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- run: ./scripts/ci/images/ci_pull_ci_image_on_ci.sh
+ - name: Pull CI image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+ run: breeze pull-image --tag-as-latest
env:
- GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
+ IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
- name: "Test downgrade"
run: ./scripts/ci/testing/run_downgrade_test.sh
- name: "Tests: ${{needs.build-info.outputs.testTypes}}"
@@ -1192,12 +1151,10 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
needs: [build-info, wait-for-ci-images]
env:
RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn) }}
- PYTHON_MAJOR_MINOR_VERSION: ${{ needs.build-info.outputs.defaultPythonVersion }}
MYSQL_VERSION: ${{needs.build-info.outputs.defaultMySQLVersion}}
POSTGRES_VERSION: ${{needs.build-info.outputs.defaultPostgresVersion}}
TEST_TYPES: "Quarantined"
- NUM_RUNS: 10
- GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ PYTHON_MAJOR_MINOR_VERSION: ${{ needs.build-info.outputs.defaultPythonVersion }}
if: needs.build-info.outputs.run-tests == 'true'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
@@ -1225,10 +1182,10 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
- run: ./scripts/ci/install_breeze.sh
- name: "Free space"
run: airflow-freespace
- - name: "Pull CI image ${{env.PYTHON_MAJOR_MINOR_VERSION}}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- run: ./scripts/ci/images/ci_pull_ci_image_on_ci.sh
+ - name: Pull CI image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+ run: breeze pull-image --tag-as-latest
env:
- GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
+ IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
- name: "Tests: Quarantined"
run: ./scripts/ci/testing/ci_run_quarantined_tests.sh
env:
@@ -1289,7 +1246,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
with:
path: ./coverage-files
- name: "Removes unnecessary artifacts"
- run: ls ./coverage-files | grep -v coverage | xargs rm -rf
+ run: ls ./coverage-files | grep -v coverage | xargs rm -rfv
- name: "Upload all coverage reports to codecov"
uses: ./.github/actions/codecov-action
with:
@@ -1324,21 +1281,45 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
with:
path: '.build/.docker_venv'
key: ${{ runner.os }}-docker-venv-${{ hashFiles('scripts/ci/images/ci_run_docker_tests.py') }}
- - name: >
- Wait for PROD images
- ${{ needs.build-info.outputs.pythonVersions }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+ - name: Wait for PROD images ${{ env.PYTHON_VERSIONS }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
# We wait for the images to be available either from "build-images.yml' run as pull_request_target
# or from build-prod-image above.
# We are utilising single job to wait for all images because this job merely waits
# For the images to be available and test them.
- id: wait-for-images
- run: ./scripts/ci/images/ci_wait_for_and_verify_all_prod_images.sh
+ run: breeze pull-prod-image --verify-image --wait-for-image --run-in-parallel
env:
- CURRENT_PYTHON_MAJOR_MINOR_VERSIONS_AS_STRING: >
- ${{needs.build-info.outputs.pythonVersionsListAsString}}
- GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- VERIFY_IMAGE: "true"
- PR_LABELS: "${{ needs.build-info.outputs.pullRequestLabels }}"
+ PYTHON_VERSIONS: ${{ needs.build-info.outputs.pythonVersionsListAsString }}
+ IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+
+ test-docker-compose-quick-start:
+ timeout-minutes: 60
+ name: "Test docker-compose quick start"
+ runs-on: ${{ fromJson(needs.build-info.outputs.runsOn) }}
+ needs: [build-info, wait-for-prod-images]
+ if: needs.build-info.outputs.image-build == 'true'
+ env:
+ PYTHON_MAJOR_MINOR_VERSION: ${{ needs.build-info.outputs.defaultPythonVersion }}
+ steps:
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
+ uses: actions/checkout@v2
+ with:
+ fetch-depth: 2
+ persist-credentials: false
+ - name: "Setup python"
+ uses: actions/setup-python@v2
+ with:
+ python-version: ${{ needs.build-info.outputs.defaultPythonVersion }}
+ cache: 'pip'
+ cache-dependency-path: ./dev/breeze/setup*
+ - run: ./scripts/ci/install_breeze.sh
+ - name: "Free space"
+ run: airflow-freespace
+ - name: Pull PROD image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+ run: breeze pull-prod-image --tag-as-latest
+ env:
+ IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+ - name: "Test docker-compose quick start"
+ run: breeze docker-compose-tests
tests-kubernetes:
timeout-minutes: 70
@@ -1364,7 +1345,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
${{needs.build-info.outputs.kubernetesVersionsListAsString}}
if: >
( needs.build-info.outputs.run-kubernetes-tests == 'true' ||
- needs.build-info.outputs.needs-helm-tests == 'true' ) &&
+ needs.build-info.outputs.needs-helm-tests == 'true' ) &&
needs.build-info.outputs.default-branch == 'main'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
@@ -1380,11 +1361,11 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
- run: ./scripts/ci/install_breeze.sh
- name: "Free space"
run: airflow-freespace
- - name: "Get all PROD images: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- run: ./scripts/ci/images/ci_wait_for_and_verify_all_prod_images.sh
+ - name: Pull PROD images ${{ env.PYTHON_VERSIONS }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+ run: breeze pull-prod-image --run-in-parallel --tag-as-latest
env:
- GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- VERIFY_IMAGE: "false"
+ PYTHON_VERSIONS: ${{ needs.build-info.outputs.pythonVersionsListAsString }}
+ IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
- name: "Cache bin folder with tools for kubernetes testing"
uses: actions/cache@v2
with:
@@ -1401,8 +1382,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
uses: actions/upload-artifact@v2
if: failure() || cancelled()
with:
- name: >
- kind-logs-${{matrix.executor}}
+ name: kind-logs-${{matrix.executor}}
path: /tmp/kind_logs_*
retention-days: 7
@@ -1441,11 +1421,11 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
- run: ./scripts/ci/install_breeze.sh
- name: "Free space"
run: airflow-freespace
- - name: "Get all PROD images: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- run: ./scripts/ci/images/ci_wait_for_and_verify_all_prod_images.sh
+ - name: Pull PROD images ${{ env.PYTHON_VERSIONS }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+ run: breeze pull-prod-image --run-in-parallel --tag-as-latest
env:
- GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- VERIFY_IMAGE: "false"
+ PYTHON_VERSIONS: ${{ needs.build-info.outputs.pythonVersionsListAsString }}
+ IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
- name: "Cache virtualenv for kubernetes testing"
uses: actions/cache@v2
with:
@@ -1473,8 +1453,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
uses: actions/upload-artifact@v2
if: failure() || cancelled()
with:
- name: >
- kind-logs-KubernetesExecutor
+ name: kind-logs-KubernetesExecutor
path: /tmp/kind_logs_*
retention-days: 7
@@ -1495,8 +1474,6 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
- tests-postgres
env:
RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn) }}
- PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
- CURRENT_PYTHON_MAJOR_MINOR_VERSIONS_AS_STRING: ${{needs.build-info.outputs.pythonVersionsListAsString}}
if: needs.build-info.outputs.upgradeToNewerDependencies != 'false'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
@@ -1513,28 +1490,16 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
- run: ./scripts/ci/install_breeze.sh
- name: "Free space"
run: airflow-freespace
- - name: >
- Wait for CI images
- ${{ needs.build-info.outputs.pythonVersions }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
- run: ./scripts/ci/images/ci_wait_for_and_verify_all_ci_images.sh
- env:
- GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- VERIFY_IMAGE: "false"
- - name: "Generate constraints with PyPI providers"
- run: ./scripts/ci/constraints/ci_generate_all_constraints.sh
- env:
- GENERATE_CONSTRAINTS_MODE: "pypi-providers"
- PR_LABELS: "${{ needs.build-info.outputs.pullRequestLabels }}"
- - name: "Generate constraints with source providers"
- run: ./scripts/ci/constraints/ci_generate_all_constraints.sh
- env:
- GENERATE_CONSTRAINTS_MODE: "source-providers"
- PR_LABELS: "${{ needs.build-info.outputs.pullRequestLabels }}"
- - name: "Generate constraints without providers"
- run: ./scripts/ci/constraints/ci_generate_all_constraints.sh
+ - name: Pull CI images ${{ env.PYTHON_VERSIONS }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+ run: breeze pull-image --run-in-parallel --tag-as-latest
env:
- GENERATE_CONSTRAINTS_MODE: "no-providers"
- PR_LABELS: "${{ needs.build-info.outputs.pullRequestLabels }}"
+ PYTHON_VERSIONS: ${{ needs.build-info.outputs.pythonVersionsListAsString }}
+ IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+ - name: "Generate constraints"
+ run: |
+ breeze generate-constraints --generate-constraints-mode source-providers --answer yes
+ breeze generate-constraints --generate-constraints-mode pypi-providers --answer yes
+ breeze generate-constraints --generate-constraints-mode no-providers --answer yes
- name: "Set constraints branch name"
id: constraints-branch
run: ./scripts/ci/constraints/ci_branch_constraints.sh
@@ -1594,27 +1559,50 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
run: airflow-freespace
- name: "Start ARM instance"
run: ./scripts/ci/images/ci_start_arm_instance_and_connect_to_docker.sh
- - name: "Build CI image cache and push ${{env.PYTHON_MAJOR_MINOR_VERSION}}"
- run: ./scripts/ci/images/ci_build_ci_image_on_ci.sh
+ - name: "Build & Push CI image ${{ matrix.python-version }}:latest"
+ run: breeze build-image --push-image
env:
- GITHUB_REGISTRY_PULL_IMAGE_TAG: "latest"
- GITHUB_REGISTRY_PUSH_IMAGE_TAG: "latest"
+ PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
UPGRADE_TO_NEWER_DEPENDENCIES: "false"
+ DOCKER_CACHE: ${{ needs.build-info.outputs.cacheDirective }}
+ IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
PLATFORM: "linux/amd64,linux/arm64"
PREPARE_BUILDX_CACHE: "true"
- - name: "Pull CI image for PROD build: ${{ matrix.python-version }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- run: ./scripts/ci/images/ci_pull_ci_image_on_ci.sh
+ VERSION_SUFFIX_FOR_PYPI: dev0
+ - name: >
+ Pull CI image for PROD build
+ ${{ env.PYTHON_MAJOR_MINOR_VERSION }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
+ run: breeze pull-image --tag-as-latest
env:
- GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- - name: "Build PROD image cache and push ${{env.PYTHON_MAJOR_MINOR_VERSION}}"
- run: ./scripts/ci/images/ci_build_prod_image_on_ci.sh
+ # Always use default Python version of CI image for preparing packages
+ PYTHON_MAJOR_MINOR_VERSION: ${{ needs.build-info.outputs.defaultPythonVersion }}
+ IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+ - name: "Cleanup dist and context file"
+ run: rm -fv ./dist/* ./docker-context-files/*
+ - name: "Prepare providers packages"
+ run: >
+ breeze prepare-provider-packages
+ --package-list-file ./scripts/ci/installed_providers.txt
+ --package-format wheel
+ - name: "Prepare airflow package"
+ run: breeze prepare-airflow-package --package-format wheel
+ - name: "Move dist packages to docker-context files"
+ run: mv -v ./dist/*.whl ./docker-context-files
+ - name: Build & Push PROD image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
+ run: >
+ breeze build-prod-image
+ --push-image
+ --install-from-docker-context-files
+ --disable-airflow-repo-cache
+ --disable-pypi
env:
- GITHUB_REGISTRY_PULL_IMAGE_TAG: "latest"
- GITHUB_REGISTRY_PUSH_IMAGE_TAG: "latest"
+ PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
UPGRADE_TO_NEWER_DEPENDENCIES: "false"
+ DOCKER_CACHE: ${{ needs.build-info.outputs.cacheDirective }}
+ IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
PLATFORM: "linux/amd64,linux/arm64"
PREPARE_BUILDX_CACHE: "true"
- VERSION_SUFFIX_FOR_PYPI: ".dev0"
+ VERSION_SUFFIX_FOR_PYPI: "dev0"
- name: "Stop ARM instance"
run: ./scripts/ci/images/ci_stop_arm_instance.sh
if: always()
diff --git a/.gitignore b/.gitignore
index a47b9ba269..9a00d53fa3 100644
--- a/.gitignore
+++ b/.gitignore
@@ -194,7 +194,8 @@ log.txt*
/airflow/providers/__init__.py
# Docker context files
-/docker-context-files
+/docker-context-files/*
+!/docker-context-files/.README.md
# Local .terraform directories
**/.terraform/*
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index a3507f46bb..18aec9c6f2 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -360,9 +360,9 @@ repos:
additional_dependencies: ['rich']
- id: update-supported-versions
name: Updates supported versions in documentation
- entry: ./scripts/ci/pre_commit/supported_versions.py
+ entry: ./scripts/ci/pre_commit/pre_commit_supported_versions.py
language: python
- files: ^scripts/ci/pre_commit/supported_versions\.py$|^README\.md$|^docs/apache-airflow/supported-versions\.rst$
+ files: ^scripts/ci/pre_commit/pre_commit_supported_versions\.py$|^README\.md$|^docs/apache-airflow/supported-versions\.rst$
pass_filenames: false
additional_dependencies: ['tabulate']
- id: check-revision-heads-map
@@ -721,20 +721,6 @@ repos:
pass_filenames: false
files: ^Dockerfile$|^Dockerfile.ci$|^scripts/docker/.*$
require_serial: true
- - id: ui-lint
- name: ESLint against airflow/ui
- language: node
- 'types_or': [javascript, tsx, ts]
- files: ^airflow/ui/
- entry: ./scripts/ci/static_checks/ui_lint.sh
- pass_filenames: false
- - id: www-lint
- name: ESLint against current UI JavaScript files
- language: node
- 'types_or': [javascript]
- files: ^airflow/www/static/js/
- entry: ./scripts/ci/static_checks/www_lint.sh
- pass_filenames: false
- id: changelog-duplicates
name: Check changelogs for duplicate entries
language: python
@@ -751,41 +737,63 @@ repos:
# The below pre-commits are those requiring CI image to be built
- id: mypy
name: Run mypy for dev
- language: system
- entry: ./scripts/ci/pre_commit/pre_commit_mypy.sh
+ language: python
+ entry: ./scripts/ci/pre_commit/pre_commit_mypy.py
files: ^dev/.*\.py$
require_serial: true
+ additional_dependencies: ['rich']
- id: mypy
name: Run mypy for core
- language: system
- entry: ./scripts/ci/pre_commit/pre_commit_mypy.sh --namespace-packages
+ language: python
+ entry: ./scripts/ci/pre_commit/pre_commit_mypy.py --namespace-packages
files: \.py$
exclude: ^provider_packages|^docs|^airflow/_vendor/|^airflow/providers|^airflow/migrations|^dev
require_serial: true
+ additional_dependencies: ['rich']
- id: mypy
name: Run mypy for providers
- language: system
- entry: ./scripts/ci/pre_commit/pre_commit_mypy.sh --namespace-packages
+ language: python
+ entry: ./scripts/ci/pre_commit/pre_commit_mypy.py --namespace-packages
files: ^airflow/providers/.*\.py$
require_serial: true
+ additional_dependencies: ['rich']
- id: mypy
name: Run mypy for /docs/ folder
- language: system
- entry: ./scripts/ci/pre_commit/pre_commit_mypy.sh
+ language: python
+ entry: ./scripts/ci/pre_commit/pre_commit_mypy.py
files: ^docs/.*\.py$
exclude: ^docs/rtd-deprecation
require_serial: true
+ additional_dependencies: ['rich']
- id: flake8
name: Run flake8
- language: system
- entry: ./scripts/ci/pre_commit/pre_commit_flake8.sh
+ language: python
+ entry: ./scripts/ci/pre_commit/pre_commit_flake8.py
files: \.py$
pass_filenames: true
exclude: ^airflow/_vendor/
+ additional_dependencies: ['rich']
+ - id: ui-lint
+ name: ESLint against airflow/ui
+ language: python
+ 'types_or': [javascript, tsx, ts]
+ files: ^airflow/ui/
+ entry: ./scripts/ci/pre_commit/pre_commit_ui_lint.py
+ pass_filenames: false
+ additional_dependencies: ['rich']
+ - id: www-lint
+ name: ESLint against current UI JavaScript files
+ language: python
+ 'types_or': [javascript]
+ files: ^airflow/www/static/js/
+ entry: ./scripts/ci/pre_commit/pre_commit_www_lint.py
+ pass_filenames: false
+ additional_dependencies: ['rich']
- id: migration-reference
name: Update migration ref doc
- language: system
- entry: ./scripts/ci/pre_commit/pre_commit_migration_reference.sh
+ language: python
+ entry: ./scripts/ci/pre_commit/pre_commit_migration_reference.py
pass_filenames: false
files: ^airflow/migrations/versions/.*\.py$|^docs/apache-airflow/migrations-ref\.rst$
+ additional_dependencies: ['rich']
## ONLY ADD PRE-COMMITS HERE THAT REQUIRE CI IMAGE
diff --git a/BREEZE.rst b/BREEZE.rst
index ff585346f7..0e3f8bf58e 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -455,11 +455,21 @@ Regular development tasks:
* Start containerised, development-friendly airflow installation with ``breeze start-airflow`` command
* Build documentation with ``breeze build-docs`` command
* Initialize local virtualenv with ``./scripts/tools/initialize_virtualenv.py`` command
-* Build CI docker image with ``breeze build-image`` command
* Cleanup breeze with ``breeze cleanup`` command
* Run static checks with autocomplete support ``breeze static-check`` command
* Run test specified with ``./breeze-legacy tests`` command
+CI Image tasks:
+
+* Build CI docker image with ``breeze build-image`` command
+* Pull CI images in parallel ``breeze pull-image`` command
+* Verify CI image ``breeze verify-image`` command
+
+PROD Image tasks:
+* Build PROD image with ``breeze build-prod-image`` command
+* Pull PROD image in parallel ``breeze pull-prod-image`` command
+* Verify CI image ``breeze verify-prod-image`` command
+
Additional management tasks:
* Join running interactive shell with ``./breeze-legacy exec`` command
@@ -467,6 +477,10 @@ Additional management tasks:
* Execute arbitrary command in the test environment with ``breeze shell`` command
* Execute arbitrary docker-compose command with ``./breeze-legacy docker-compose`` command
+Docker compose tests:
+
+* Run docker-compose tests with ``breeze docker-compose-tests`` command.
+
Kubernetes tests related:
* Manage KinD Kubernetes cluster and deploy Airflow to KinD cluster ``./breeze-legacy kind-cluster`` commands
@@ -485,10 +499,10 @@ do not need):
* Prepare cache for CI: ``breeze build-image --prepare-build-cache`` and
``breeze build-prod image --prepare-build-cache``(needs buildx plugin and write access to registry ghcr.io)
-* Generate constraints with ``./breeze-legacy generate-constraints`` (needed when conflicting changes are merged)
-* Prepare airflow packages: ``./breeze-legacy prepare-airflow-packages`` (when releasing Airflow)
-* Prepare provider documentation ``./breeze-legacy prepare-provider-documentation`` and prepare provider packages
- ``./breeze-legacy prepare-provider-packages`` (when releasing provider packages)
+* Generate constraints with ``breeze generate-constraints`` (needed when conflicting changes are merged)
+* Prepare airflow packages: ``breeze prepare-airflow-package`` (when releasing Airflow)
+* Prepare provider documentation ``breeze prepare-provider-documentation`` and prepare provider packages
+ ``breeze prepare-provider-packages`` (when releasing provider packages)
Details of Breeze usage
=======================
@@ -626,7 +640,7 @@ command but it is very similar to current ``breeze`` command):
</a>
</div>
-Building CI images
+Managing CI images
------------------
With Breeze you can build images that are used by Airflow CI and production ones.
@@ -667,6 +681,24 @@ Those are all available flags of ``build-image`` command:
:width: 100%
:alt: Breeze build-image
+You can also pull the CI images locally in parallel with optional verification.
+
+Those are all available flags of ``pull-image`` command:
+
+.. image:: ./images/breeze/output-pull-image.svg
+ :width: 100%
+ :alt: Breeze pull-image
+
+Finally, you can verify CI image by running tests - either with the pulled/built images or
+with an arbitrary image.
+
+Those are all available flags of ``verify-image`` command:
+
+.. image:: ./images/breeze/output-verify-image.svg
+ :width: 100%
+ :alt: Breeze verify-image
+
+
Preparing packages
------------------
@@ -678,12 +710,31 @@ You can read more about testing provider packages in
There are several commands that you can run in Breeze to manage and build packages:
-* preparing Provider Readme files
+* preparing Provider documentation files
* preparing Airflow packages
* preparing Provider packages
-Preparing provider readme files is part of the release procedure by the release managers
-and it is described in detail in `dev <dev/README.md>`_ .
+Preparing provider documentation files is part of the release procedure by the release managers
+and it is described in detail in `dev <dev/README_RELEASE_PROVIDER_PACKAGES.md>`_ .
+
+The below example perform documentation preparation for provider packages.
+
+.. code-block:: bash
+
+ breeze prepare-provider-documentation
+
+By default, the documentation preparation runs package verification to check if all packages are
+importable, but you can add ``--skip-package-verification`` to skip it.
+
+.. code-block:: bash
+
+ breeze prepare-provider-documentation --skip-package-verification
+
+You can also add ``--answer yes`` to perform non-interactive build.
+
+.. image:: ./images/breeze/output-prepare-provider-documentation.svg
+ :width: 100%
+ :alt: Breeze prepare-provider-documentation
The packages are prepared in ``dist`` folder. Note, that this command cleans up the ``dist`` folder
before running, so you should run it before generating airflow package below as it will be removed.
@@ -692,7 +743,7 @@ The below example builds provider packages in the wheel format.
.. code-block:: bash
- ./breeze-legacy prepare-provider-packages
+ breeze prepare-provider-packages
If you run this command without packages, you will prepare all packages, you can however specify
providers that you would like to build. By default ``both`` types of packages are prepared (
@@ -700,20 +751,23 @@ providers that you would like to build. By default ``both`` types of packages ar
.. code-block:: bash
- ./breeze-legacy prepare-provider-packages google amazon
+ breeze prepare-provider-packages google amazon
You can see all providers available by running this command:
.. code-block:: bash
- ./breeze-legacy prepare-provider-packages -- --help
+ breeze prepare-provider-packages --help
+.. image:: ./images/breeze/output-prepare-provider-packages.svg
+ :width: 100%
+ :alt: Breeze prepare-provider-packages
-You can also prepare airflow packages using breeze:
+You can prepare airflow packages using breeze:
.. code-block:: bash
- ./breeze-legacy prepare-airflow-packages
+ breeze prepare-airflow-package
This prepares airflow .whl package in the dist folder.
@@ -722,10 +776,13 @@ default is to build ``both`` type of packages ``sdist`` and ``wheel``.
.. code-block:: bash
- ./breeze-legacy prepare-airflow-packages --package-format=wheel
+ breeze prepare-airflow-package --package-format=wheel
+.. image:: ./images/breeze/output-prepare-airflow-package.svg
+ :width: 100%
+ :alt: Breeze prepare-airflow-package
-Building Production images
+Managing Production images
--------------------------
The **Production image** is also maintained in GitHub Container Registry for Caching
@@ -794,6 +851,25 @@ command but it is very similar to current ``breeze`` command):
</a>
</div>
+You can also pull PROD images in parallel with optional verification.
+
+Those are all available flags of ``pull-prod-image`` command:
+
+.. image:: ./images/breeze/output-pull-prod-image.svg
+ :width: 100%
+ :alt: Breeze pull-prod-image
+
+Finally, you can verify PROD image by running tests - either with the pulled/built images or
+with an arbitrary image.
+
+Those are all available flags of ``verify-prod-image`` command:
+
+.. image:: ./images/breeze/output-verify-prod-image.svg
+ :width: 100%
+ :alt: Breeze verify-prod-image
+
+
+
Running static checks
---------------------
@@ -913,13 +989,19 @@ files are stored in separated orphan branches: ``constraints-main``, ``constrain
Those are constraint files as described in detail in the
`<CONTRIBUTING.rst#pinned-constraint-files>`_ contributing documentation.
-You can use ``./breeze-legacy generate-constraints`` command to manually generate constraints for
-a single python version and single constraint mode like this:
-.. code-block:: bash
+You can use ``breeze generate-constraints`` command to manually generate constraints for
+all or selected python version and single constraint mode like this:
- ./breeze-legacy generate-constraints --generate-constraints-mode pypi-providers
+.. warning::
+ In order to generate constraints, you need to build all images with ``--upgrade-to-newer-dependencies``
+ flag - for all python versions.
+
+
+.. code-block:: bash
+
+ breeze generate-constraints --generate-constraints-mode pypi-providers
Constraints are generated separately for each python version and there are separate constraints modes:
@@ -937,6 +1019,12 @@ Constraints are generated separately for each python version and there are separ
providers. If you want to manage airflow separately and then add providers individually, you can
use those. Use ``no-providers`` mode for that.
+Those are all available flags of ``generate-constraints`` command:
+
+.. image:: ./images/breeze/output-generate-constraints.svg
+ :width: 100%
+ :alt: Breeze generate-constraints
+
In case someone modifies setup.py, the scheduled CI Tests automatically upgrades and
pushes changes to the constraint files, however you can also perform test run of this locally using
the procedure described in `Refreshing CI Cache <dev/REFRESHING_CI_CACHE.md#manually-generating-constraint-files>`_
@@ -994,6 +1082,17 @@ but it is not available in the ``breeze`` command):
</a>
</div>
+Running docker-compose tests
+----------------------------
+
+You can use Breeze to run docker-compose tests. Those tests are run using Production image
+and they are running test with the Quick-start docker compose we have.
+
+.. image:: ./images/breeze/output-docker-compose-tests.svg
+ :width: 100%
+ :alt: Breeze generate-constraints
+
+
Running Kubernetes tests
------------------------
@@ -1119,12 +1218,12 @@ when rebuilding or removing an image and in few other cases - actions that take
or could be potentially destructive. You can force answer to the questions by providing an
``--answer`` flag in the commands that support it.
-For automation scripts, you can export the ``FORCE_ANSWER_TO_QUESTIONS`` variable (and set it to
+For automation scripts, you can export the ``ANSWER`` variable (and set it to
``y``, ``n``, ``q``, ``yes``, ``no``, ``quit`` - in all case combinations).
.. code-block::
- export FORCE_ANSWER_TO_QUESTIONS="yes"
+ export ANSWER="yes"
Fixing File/Directory Ownership
-------------------------------
diff --git a/CI.rst b/CI.rst
index 10b1b666fc..74623c60e1 100644
--- a/CI.rst
+++ b/CI.rst
@@ -158,7 +158,7 @@ You can use those variables when you try to reproduce the build locally.
| | | | | builds it forces rebuild, regardless if it |
| | | | | is determined to be needed. |
+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
-| ``FORCE_ANSWER_TO_QUESTIONS`` | | yes | yes | This variable determines if answer to questions |
+| ``ANSWER`` | | yes | yes | This variable determines if answer to questions |
| | | | | during the build process should be |
| | | | | automatically given. For local development, |
| | | | | the user is occasionally asked to provide |
@@ -188,11 +188,6 @@ You can use those variables when you try to reproduce the build locally.
+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
| ``HOST_HOME`` | | | | Home directory on the host. |
+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
-| Version suffix variables |
-+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
-| ``VERSION_SUFFIX_FOR_PYPI`` | | | | Version suffix used during provider |
-| | | | | package preparation for PyPI builds. |
-+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
| Git variables |
+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
| ``COMMIT_SHA`` | | GITHUB_SHA | GITHUB_SHA | SHA of the commit of the build is run |
diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
index 7494cb0142..7f00045fd5 100644
--- a/CONTRIBUTING.rst
+++ b/CONTRIBUTING.rst
@@ -645,7 +645,6 @@ all dependencies needed in the CI environment.
This is the full list of those extras:
.. START EXTRAS HERE
-
airbyte, alibaba, all, all_dbs, amazon, apache.atlas, apache.beam, apache.cassandra, apache.drill,
apache.druid, apache.hdfs, apache.hive, apache.kylin, apache.livy, apache.pig, apache.pinot,
apache.spark, apache.sqoop, apache.webhdfs, arangodb, asana, async, atlas, aws, azure, cassandra,
@@ -658,7 +657,6 @@ neo4j, odbc, openfaas, opsgenie, oracle, pagerduty, pandas, papermill, password,
postgres, presto, qds, qubole, rabbitmq, redis, s3, salesforce, samba, segment, sendgrid, sentry,
sftp, singularity, slack, snowflake, spark, sqlite, ssh, statsd, tableau, telegram, trino, vertica,
virtualenv, webhdfs, winrm, yandex, zendesk
-
.. END EXTRAS HERE
Provider packages
diff --git a/Dockerfile.ci b/Dockerfile.ci
index f195e52c26..02aac0b55b 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -620,13 +620,6 @@ fi
LD_PRELOAD="/usr/lib/$(uname -m)-linux-gnu/libstdc++.so.6"
export LD_PRELOAD
-if [[ $(uname -m) == "arm64" || $(uname -m) == "aarch64" ]]; then
- if [[ ${BACKEND} == "mysql" || ${BACKEND} == "mssql" ]]; then
- echo "${COLOR_RED}ARM platform is not supported for ${BACKEND} backend. Exiting.${COLOR_RESET}"
- exit 1
- fi
-fi
-
chmod 1777 /tmp
AIRFLOW_SOURCES=$(cd "${IN_CONTAINER_DIR}/../.." || exit 1; pwd)
@@ -639,6 +632,13 @@ export AIRFLOW_HOME=${AIRFLOW_HOME:=${HOME}}
if [[ ${SKIP_ENVIRONMENT_INITIALIZATION=} != "true" ]]; then
+ if [[ $(uname -m) == "arm64" || $(uname -m) == "aarch64" ]]; then
+ if [[ ${BACKEND:=} == "mysql" || ${BACKEND} == "mssql" ]]; then
+ echo "${COLOR_RED}ARM platform is not supported for ${BACKEND} backend. Exiting.${COLOR_RESET}"
+ exit 1
+ fi
+ fi
+
echo
echo "${COLOR_BLUE}Running Initialization. Your basic configuration is:${COLOR_RESET}"
echo
@@ -979,13 +979,6 @@ fi
LD_PRELOAD="/usr/lib/$(uname -m)-linux-gnu/libstdc++.so.6"
export LD_PRELOAD
-if [[ $(uname -m) == "arm64" || $(uname -m) == "aarch64" ]]; then
- if [[ ${BACKEND} == "mysql" || ${BACKEND} == "mssql" ]]; then
- echo "${COLOR_RED}ARM platform is not supported for ${BACKEND} backend. Exiting.${COLOR_RESET}"
- exit 1
- fi
-fi
-
chmod 1777 /tmp
AIRFLOW_SOURCES=$(cd "${IN_CONTAINER_DIR}/../.." || exit 1; pwd)
@@ -998,6 +991,13 @@ export AIRFLOW_HOME=${AIRFLOW_HOME:=${HOME}}
if [[ ${SKIP_ENVIRONMENT_INITIALIZATION=} != "true" ]]; then
+ if [[ $(uname -m) == "arm64" || $(uname -m) == "aarch64" ]]; then
+ if [[ ${BACKEND:=} == "mysql" || ${BACKEND} == "mssql" ]]; then
+ echo "${COLOR_RED}ARM platform is not supported for ${BACKEND} backend. Exiting.${COLOR_RESET}"
+ exit 1
+ fi
+ fi
+
echo
echo "${COLOR_BLUE}Running Initialization. Your basic configuration is:${COLOR_RESET}"
echo
diff --git a/INSTALL b/INSTALL
index 8773e86d4c..c5b3b2a76d 100644
--- a/INSTALL
+++ b/INSTALL
@@ -94,7 +94,6 @@ pip install .[async,google,amazon] \
The list of available extras:
# START EXTRAS HERE
-
airbyte, alibaba, all, all_dbs, amazon, apache.atlas, apache.beam, apache.cassandra, apache.drill,
apache.druid, apache.hdfs, apache.hive, apache.kylin, apache.livy, apache.pig, apache.pinot,
apache.spark, apache.sqoop, apache.webhdfs, arangodb, asana, async, atlas, aws, azure, cassandra,
@@ -107,7 +106,6 @@ neo4j, odbc, openfaas, opsgenie, oracle, pagerduty, pandas, papermill, password,
postgres, presto, qds, qubole, rabbitmq, redis, s3, salesforce, samba, segment, sendgrid, sentry,
sftp, singularity, slack, snowflake, spark, sqlite, ssh, statsd, tableau, telegram, trino, vertica,
virtualenv, webhdfs, winrm, yandex, zendesk
-
# END EXTRAS HERE
# For installing Airflow in development environments - see CONTRIBUTING.rst
diff --git a/README.md b/README.md
index b8f84fa583..b44b227131 100644
--- a/README.md
+++ b/README.md
@@ -268,7 +268,7 @@ packages:
Apache Airflow version life cycle:
-<!-- This table is automatically updated by pre-commit scripts/ci/pre-commit/supported_versions.py -->
+<!-- This table is automatically updated by pre-commit scripts/ci/pre_commit/pre_commit_supported_versions.py -->
<!-- Beginning of auto-generated table -->
| Version | Current Patch/Minor | State | First Release | Limited Support | EOL/Terminated |
diff --git a/STATIC_CODE_CHECKS.rst b/STATIC_CODE_CHECKS.rst
index 88f406b1f5..00c4f55516 100644
--- a/STATIC_CODE_CHECKS.rst
+++ b/STATIC_CODE_CHECKS.rst
@@ -222,7 +222,7 @@ require Breeze Docker images to be installed locally.
------------------------------------ ---------------------------------------------------------------- ------------
``markdownlint`` Lints Markdown files
------------------------------------ ---------------------------------------------------------------- ------------
-``migration-reference`` Update migration reference doc and revision metadata
+``migration-reference`` Update migration reference doc and revision metadata *
------------------------------------ ---------------------------------------------------------------- ------------
``mixed-line-ending`` Detects if mixed line ending is used (\r vs. \r\n)
------------------------------------ ---------------------------------------------------------------- ------------
@@ -272,7 +272,7 @@ require Breeze Docker images to be installed locally.
------------------------------------ ---------------------------------------------------------------- ------------
``trailing-whitespace`` Removes trailing whitespace at end of line
------------------------------------ ---------------------------------------------------------------- ------------
-``ui-lint`` Static checks of airflow/ui/ folder
+``ui-lint`` Static checks of airflow/ui/ folder *
------------------------------------ ---------------------------------------------------------------- ------------
``update-breeze-file`` Update output of breeze commands in BREEZE.rst
------------------------------------ ---------------------------------------------------------------- ------------
@@ -294,7 +294,7 @@ require Breeze Docker images to be installed locally.
------------------------------------ ---------------------------------------------------------------- ------------
``verify-db-migrations-documented`` Verify DB Migrations have been documented
------------------------------------ ---------------------------------------------------------------- ------------
-``www-lint`` Static checks of js in airflow/www/static/js/ folder
+``www-lint`` Static checks of js in airflow/www/static/js/ folder *
------------------------------------ ---------------------------------------------------------------- ------------
``yesqa`` Removes unnecessary noqa statements
------------------------------------ ---------------------------------------------------------------- ------------
diff --git a/TESTING.rst b/TESTING.rst
index 5012d72e66..215875534a 100644
--- a/TESTING.rst
+++ b/TESTING.rst
@@ -559,7 +559,7 @@ need to run the following steps:
.. code-block:: bash
- ./breeze-legacy prepare-provider-packages [PACKAGE ...]
+ breeze prepare-provider-packages [PACKAGE ...]
If you run this command without packages, you will prepare all packages. However, You can specify
providers that you would like to build if you just want to build few provider packages.
@@ -570,7 +570,7 @@ before running, so you should run it before generating ``apache-airflow`` packag
.. code-block:: bash
- ./breeze-legacy prepare-airflow-packages
+ breeze prepare-airflow-package
This prepares airflow .whl package in the dist folder.
@@ -1086,7 +1086,7 @@ example, the below command will build google, postgres and mysql wheel packages:
.. code-block:: bash
- ./breeze-legacy prepare-provider-packages -- google postgres mysql
+ breeze prepare-provider-packages google postgres mysql
Those packages will be prepared in ./dist folder. This folder is mapped to /dist folder
when you enter Breeze, so it is easy to automate installing those packages for testing.
diff --git a/airflow/providers/amazon/.latest-doc-only-change.txt b/airflow/providers/amazon/.latest-doc-only-change.txt
new file mode 100644
index 0000000000..8668d3142d
--- /dev/null
+++ b/airflow/providers/amazon/.latest-doc-only-change.txt
@@ -0,0 +1 @@
+9e1ac6e425aa52a55601bb2b5587fd97d361bfcc
diff --git a/airflow/providers/asana/.latest-doc-only-change.txt b/airflow/providers/asana/.latest-doc-only-change.txt
index 2812409864..9ab50d6e01 100644
--- a/airflow/providers/asana/.latest-doc-only-change.txt
+++ b/airflow/providers/asana/.latest-doc-only-change.txt
@@ -1 +1 @@
-6c3a67d4fccafe4ab6cd9ec8c7bacf2677f17038
+6933022e94acf139b2dea9a589bb8b25c62a5d20
diff --git a/airflow/providers/cloudant/.latest-doc-only-change.txt b/airflow/providers/cloudant/.latest-doc-only-change.txt
index 2812409864..9ab50d6e01 100644
--- a/airflow/providers/cloudant/.latest-doc-only-change.txt
+++ b/airflow/providers/cloudant/.latest-doc-only-change.txt
@@ -1 +1 @@
-6c3a67d4fccafe4ab6cd9ec8c7bacf2677f17038
+6933022e94acf139b2dea9a589bb8b25c62a5d20
diff --git a/airflow/providers/cncf/kubernetes/.latest-doc-only-change.txt b/airflow/providers/cncf/kubernetes/.latest-doc-only-change.txt
new file mode 100644
index 0000000000..8be8f80b1c
--- /dev/null
+++ b/airflow/providers/cncf/kubernetes/.latest-doc-only-change.txt
@@ -0,0 +1 @@
+c7399c7190750ba705b8255b7a92de2554e6eef3
diff --git a/airflow/providers/databricks/CHANGELOG.rst b/airflow/providers/databricks/CHANGELOG.rst
index 2b5bcd1e2b..48c3cdab04 100644
--- a/airflow/providers/databricks/CHANGELOG.rst
+++ b/airflow/providers/databricks/CHANGELOG.rst
@@ -45,6 +45,9 @@ Misc
* ``Fix new MyPy errors in main (#22884)``
* ``Prepare mid-April provider documentation. (#22819)``
+.. Review and move the new changes to one of the sections above:
+ * ``Prepare for RC2 release of March Databricks provider (#22979)``
+
2.5.0
.....
diff --git a/airflow/providers/datadog/.latest-doc-only-change.txt b/airflow/providers/datadog/.latest-doc-only-change.txt
index 2812409864..9ab50d6e01 100644
--- a/airflow/providers/datadog/.latest-doc-only-change.txt
+++ b/airflow/providers/datadog/.latest-doc-only-change.txt
@@ -1 +1 @@
-6c3a67d4fccafe4ab6cd9ec8c7bacf2677f17038
+6933022e94acf139b2dea9a589bb8b25c62a5d20
diff --git a/airflow/providers/dbt/cloud/.latest-doc-only-change.txt b/airflow/providers/dbt/cloud/.latest-doc-only-change.txt
new file mode 100644
index 0000000000..f5f87d5b66
--- /dev/null
+++ b/airflow/providers/dbt/cloud/.latest-doc-only-change.txt
@@ -0,0 +1 @@
+49e336ae0302b386a2f47269a6d13988382d975f
diff --git a/airflow/providers/dingding/.latest-doc-only-change.txt b/airflow/providers/dingding/.latest-doc-only-change.txt
index 2812409864..56688bb57c 100644
--- a/airflow/providers/dingding/.latest-doc-only-change.txt
+++ b/airflow/providers/dingding/.latest-doc-only-change.txt
@@ -1 +1 @@
-6c3a67d4fccafe4ab6cd9ec8c7bacf2677f17038
+7fe97fb86ffbf51db2d54e9a1b46520f5591579a
diff --git a/airflow/providers/docker/.latest-doc-only-change.txt b/airflow/providers/docker/.latest-doc-only-change.txt
index b4f26fa175..9ab50d6e01 100644
--- a/airflow/providers/docker/.latest-doc-only-change.txt
+++ b/airflow/providers/docker/.latest-doc-only-change.txt
@@ -1 +1 @@
-aa2cb5545f09d694b9143b323efcd4f6b6c66e60
+6933022e94acf139b2dea9a589bb8b25c62a5d20
diff --git a/airflow/providers/elasticsearch/.latest-doc-only-change.txt b/airflow/providers/elasticsearch/.latest-doc-only-change.txt
index e7c3c940c9..9ab50d6e01 100644
--- a/airflow/providers/elasticsearch/.latest-doc-only-change.txt
+++ b/airflow/providers/elasticsearch/.latest-doc-only-change.txt
@@ -1 +1 @@
-602abe8394fafe7de54df7e73af56de848cdf617
+6933022e94acf139b2dea9a589bb8b25c62a5d20
diff --git a/airflow/providers/github/.latest-doc-only-change.txt b/airflow/providers/github/.latest-doc-only-change.txt
new file mode 100644
index 0000000000..9ab50d6e01
--- /dev/null
+++ b/airflow/providers/github/.latest-doc-only-change.txt
@@ -0,0 +1 @@
+6933022e94acf139b2dea9a589bb8b25c62a5d20
diff --git a/airflow/providers/google/.latest-doc-only-change.txt b/airflow/providers/google/.latest-doc-only-change.txt
new file mode 100644
index 0000000000..9e228285e5
--- /dev/null
+++ b/airflow/providers/google/.latest-doc-only-change.txt
@@ -0,0 +1 @@
+07a13bb7088d4a8ba50efbf14981150656b020a7
diff --git a/airflow/providers/hashicorp/CHANGELOG.rst b/airflow/providers/hashicorp/CHANGELOG.rst
index 93b76e3e3e..533590f107 100644
--- a/airflow/providers/hashicorp/CHANGELOG.rst
+++ b/airflow/providers/hashicorp/CHANGELOG.rst
@@ -27,6 +27,9 @@ Features
* ``Update secrets backends to use get_conn_value instead of get_conn_uri (#22348)``
+.. Review and move the new changes to one of the sections above:
+ * ``Prepare mid-April provider documentation. (#22819)``
+
2.1.4
.....
diff --git a/airflow/providers/jdbc/.latest-doc-only-change.txt b/airflow/providers/jdbc/.latest-doc-only-change.txt
index cda183acd3..f5f87d5b66 100644
--- a/airflow/providers/jdbc/.latest-doc-only-change.txt
+++ b/airflow/providers/jdbc/.latest-doc-only-change.txt
@@ -1 +1 @@
-cb73053211367e2c2dd76d5279cdc7dc7b190124
+49e336ae0302b386a2f47269a6d13988382d975f
diff --git a/airflow/providers/jenkins/.latest-doc-only-change.txt b/airflow/providers/jenkins/.latest-doc-only-change.txt
index 2812409864..0ebf71e073 100644
--- a/airflow/providers/jenkins/.latest-doc-only-change.txt
+++ b/airflow/providers/jenkins/.latest-doc-only-change.txt
@@ -1 +1 @@
-6c3a67d4fccafe4ab6cd9ec8c7bacf2677f17038
+e5dd6fdcfd2f53ed90e29070711c121de447b404
diff --git a/airflow/providers/microsoft/azure/.latest-doc-only-change.txt b/airflow/providers/microsoft/azure/.latest-doc-only-change.txt
index cda183acd3..f5f87d5b66 100644
--- a/airflow/providers/microsoft/azure/.latest-doc-only-change.txt
+++ b/airflow/providers/microsoft/azure/.latest-doc-only-change.txt
@@ -1 +1 @@
-cb73053211367e2c2dd76d5279cdc7dc7b190124
+49e336ae0302b386a2f47269a6d13988382d975f
diff --git a/airflow/providers/microsoft/winrm/.latest-doc-only-change.txt b/airflow/providers/microsoft/winrm/.latest-doc-only-change.txt
index 7d499ac48c..f5f87d5b66 100644
--- a/airflow/providers/microsoft/winrm/.latest-doc-only-change.txt
+++ b/airflow/providers/microsoft/winrm/.latest-doc-only-change.txt
@@ -1 +1 @@
-9ed9b5170c8dbb11469a88c41e323d8b61a1e7e6
+49e336ae0302b386a2f47269a6d13988382d975f
diff --git a/airflow/providers/oracle/.latest-doc-only-change.txt b/airflow/providers/oracle/.latest-doc-only-change.txt
index e7e8156d80..7d99544015 100644
--- a/airflow/providers/oracle/.latest-doc-only-change.txt
+++ b/airflow/providers/oracle/.latest-doc-only-change.txt
@@ -1 +1 @@
-b916b7507921129dc48d6add1bdc4b923b60c9b9
+4147aa1bdf1fdeddd86697d712151d56d54ecfac
diff --git a/airflow/providers/presto/.latest-doc-only-change.txt b/airflow/providers/presto/.latest-doc-only-change.txt
index 2812409864..b4e77ccb86 100644
--- a/airflow/providers/presto/.latest-doc-only-change.txt
+++ b/airflow/providers/presto/.latest-doc-only-change.txt
@@ -1 +1 @@
-6c3a67d4fccafe4ab6cd9ec8c7bacf2677f17038
+5164cdbe98ad63754d969b4b300a7a0167565e33
diff --git a/airflow/providers/qubole/.latest-doc-only-change.txt b/airflow/providers/qubole/.latest-doc-only-change.txt
index cda183acd3..f5f87d5b66 100644
--- a/airflow/providers/qubole/.latest-doc-only-change.txt
+++ b/airflow/providers/qubole/.latest-doc-only-change.txt
@@ -1 +1 @@
-cb73053211367e2c2dd76d5279cdc7dc7b190124
+49e336ae0302b386a2f47269a6d13988382d975f
diff --git a/airflow/providers/trino/.latest-doc-only-change.txt b/airflow/providers/trino/.latest-doc-only-change.txt
index 2812409864..b4e77ccb86 100644
--- a/airflow/providers/trino/.latest-doc-only-change.txt
+++ b/airflow/providers/trino/.latest-doc-only-change.txt
@@ -1 +1 @@
-6c3a67d4fccafe4ab6cd9ec8c7bacf2677f17038
+5164cdbe98ad63754d969b4b300a7a0167565e33
diff --git a/breeze-complete b/breeze-complete
index c57879cf17..173ce24029 100644
--- a/breeze-complete
+++ b/breeze-complete
@@ -23,10 +23,8 @@
# Those cannot be made read-only as the breeze-complete must be re-sourceable
_breeze_allowed_python_major_minor_versions="3.7 3.8 3.9 3.10"
-_breeze_allowed_debian_versions="bullseye buster"
_breeze_allowed_backends="sqlite mysql postgres mssql"
_breeze_allowed_integrations="cassandra kerberos mongo openldap pinot rabbitmq redis statsd trino all"
-_breeze_allowed_generate_constraints_modes="source-providers pypi-providers no-providers"
_breeze_allowed_kubernetes_modes="image"
_breeze_allowed_kubernetes_versions="v1.23.4 v1.22.7 v1.21.10 v1.20.15"
_breeze_allowed_helm_versions="v3.6.3"
@@ -37,16 +35,11 @@ _breeze_allowed_postgres_versions="10 11 12 13"
_breeze_allowed_kind_operations="start stop restart status deploy test shell k9s"
_breeze_allowed_executors="KubernetesExecutor CeleryExecutor LocalExecutor CeleryKubernetesExecutor LocalKubernetesExecutor"
_breeze_allowed_test_types="All Always Core Providers API CLI Integration Other WWW Postgres MySQL Helm Quarantined"
-_breeze_allowed_package_formats="both sdist wheel"
-_breeze_allowed_installation_methods=". apache-airflow"
-_breeze_allowed_platforms="linux/amd64 linux/arm64 linux/amd64,linux/arm64"
# shellcheck disable=SC2034
{
# Default values for the commands & flags used
- _breeze_default_debian_version=$(echo "${_breeze_allowed_debian_versions}" | awk '{print $1}')
_breeze_default_backend=$(echo "${_breeze_allowed_backends}" | awk '{print $1}')
- _breeze_default_generate_providers_mode=$(echo "${_breeze_allowed_generate_constraints_modes}" | awk '{print $1}')
_breeze_default_kubernetes_mode=$(echo "${_breeze_allowed_kubernetes_modes}" | awk '{print $1}')
_breeze_default_kubernetes_version=$(echo "${_breeze_allowed_kubernetes_versions}" | awk '{print $1}')
_breeze_default_helm_version=$(echo "${_breeze_allowed_helm_versions}" | awk '{print $1}')
@@ -56,21 +49,13 @@ _breeze_allowed_platforms="linux/amd64 linux/arm64 linux/amd64,linux/arm64"
_breeze_default_mysql_version=$(echo "${_breeze_allowed_mysql_versions}" | awk '{print $1}')
_breeze_default_mssql_version=$(echo "${_breeze_allowed_mssql_versions}" | awk '{print $1}')
_breeze_default_test_type=$(echo "${_breeze_allowed_test_types}" | awk '{print $1}')
- _breeze_default_package_format=$(echo "${_breeze_allowed_package_formats}" | awk '{print $1}')
- _breeze_default_platform=$(echo "${_breeze_allowed_platforms}" | awk '{print $1}')
}
-_breeze_allowed_install_airflow_versions=$(cat <<-EOF
-2.0.2
-2.0.1
-2.0.0
+_breeze_allowed_USE_AIRFLOW_VERSION=$(cat <<-EOF
+2.2.5
+2.1.4
wheel
sdist
-EOF
-)
-
-_breeze_allowed_USE_AIRFLOW_VERSION=$(cat <<-EOF
-${_breeze_allowed_install_airflow_versions}
none
EOF
)
@@ -176,7 +161,7 @@ _breeze_default_github_repository="apache/airflow"
_breeze_default_github_image_id="latest"
_breeze_short_options="
-h p: d: b: i:
+h p: b: i:
K: V:
l a: t: d:
v y n q f
@@ -187,38 +172,20 @@ S: N:
"
_breeze_long_options="
-help python: debian-version: backend: integration: platform:
+help python: backend: integration:
kubernetes-mode: kubernetes-version: helm-version: kind-version:
-skip-mounting-local-sources mount-all-local-sources install-airflow-version: install-airflow-reference: db-reset
+skip-mounting-local-sources mount-all-local-sources db-reset
verbose assume-yes assume-no assume-quit forward-credentials init-script:
-force-build-images force-pull-images check-if-base-python-image-updated production-image extras: force-clean-images skip-rebuild-check
-build-cache-local build-cache-pulled build-cache-disabled disable-pip-cache
-github-repository: github-image-id: generate-constraints-mode:
+github-repository: github-image-id:
postgres-version: mysql-version: mssql-version:
-version-suffix-for-pypi: version-suffix-for-svn:
-additional-extras: additional-python-deps: additional-dev-deps: additional-runtime-deps: image-tag:
-disable-mysql-client-installation disable-mssql-client-installation constraints-location: disable-pip-cache install-from-docker-context-files
-additional-extras: additional-python-deps: disable-pypi-when-building
-dev-apt-deps: additional-dev-apt-deps: dev-apt-command: additional-dev-apt-command: additional-dev-apt-env:
-runtime-apt-deps: additional-runtime-apt-deps: runtime-apt-command: additional-runtime-apt-command: additional-runtime-apt-env:
load-default-connections load-example-dags
-use-packages-from-dist no-rbac-ui package-format: upgrade-to-newer-dependencies installation-method: continue-on-pip-check-failure non-interactive generate-providers-issue
-use-airflow-version:
-cleanup-docker-context-files prepare-buildx-cache
-test-type: preserve-volumes dry-run-docker
-executor:
+use-airflow-version: test-type: preserve-volumes dry-run-docker executor:
"
_breeze_commands="
shell
-build-docs
-build-image
-prepare-build-cache
-cleanup-image
exec
-generate-constraints
initialize-local-virtualenv
-prepare-airflow-packages
setup-autocomplete
start-airflow
stop
@@ -229,8 +196,6 @@ toggle-suppress-asciiart"
_breeze_extra_arg_commands="
docker-compose
kind-cluster
-prepare-provider-documentation
-prepare-provider-packages
static-check
tests
"
@@ -266,9 +231,6 @@ function breeze_complete::get_known_values_breeze() {
-p | --python)
_breeze_known_values=${_breeze_allowed_python_major_minor_versions}
;;
- -d | --debian-version)
- _breeze_known_values=${_breeze_allowed_debian_versions}
- ;;
-b | --backend)
_breeze_known_values=${_breeze_allowed_backends}
;;
@@ -290,9 +252,6 @@ function breeze_complete::get_known_values_breeze() {
static-check)
_breeze_known_values=${_breeze_allowed_static_checks}
;;
- -a | --install-airflow-version)
- _breeze_known_values=${_breeze_allowed_install_airflow_versions}
- ;;
--use-airflow-version)
_breeze_known_values=${_breeze_allowed_USE_AIRFLOW_VERSION}
;;
@@ -330,15 +289,6 @@ function breeze_complete::get_known_values_breeze() {
--executor)
_breeze_known_values="${_breeze_allowed_executors}"
;;
- --package-format)
- _breeze_known_values="${_breeze_allowed_package_formats}"
- ;;
- --installation-method)
- _breeze_known_values="${_breeze_allowed_installation_methods}"
- ;;
- --generate-constraints-mode)
- _breeze_known_values="${_breeze_allowed_generate_constraints_modes}"
- ;;
*)
_breeze_known_values=""
;;
diff --git a/breeze-legacy b/breeze-legacy
index 2885ca87a6..362e327d19 100755
--- a/breeze-legacy
+++ b/breeze-legacy
@@ -500,7 +500,6 @@ EOF
Branch name: ${BRANCH_NAME}
Docker image: ${AIRFLOW_PROD_IMAGE}
Platform: ${PLATFORM}
- Airflow source version: $(build_images::get_airflow_version_from_production_image)
EOF
else
cat <<EOF
@@ -652,7 +651,7 @@ export AIRFLOW_PROD_IMAGE="${AIRFLOW_PROD_IMAGE}"
export AIRFLOW_IMAGE_KUBERNETES="${AIRFLOW_IMAGE_KUBERNETES}"
export SQLITE_URL="${SQLITE_URL}"
export USE_AIRFLOW_VERSION="${USE_AIRFLOW_VERSION}"
-export SKIP_TWINE_CHECK="${SKIP_TWINE_CHECK}"
+export SKIP_PACKAGE_VERIFICATION="${SKIP_PACKAGE_VERIFICATION}"
export USE_PACKAGES_FROM_DIST="${USE_PACKAGES_FROM_DIST}"
export EXECUTOR="${EXECUTOR}"
export START_AIRFLOW="${START_AIRFLOW}"
@@ -661,8 +660,6 @@ export ENABLED_SYSTEMS="${ENABLED_SYSTEMS}"
export GITHUB_ACTIONS="${GITHUB_ACTIONS}"
export ISSUE_ID="${ISSUE_ID}"
export NUM_RUNS="${NUM_RUNS}"
-export VERSION_SUFFIX_FOR_SVN="${VERSION_SUFFIX_FOR_SVN}"
-export VERSION_SUFFIX_FOR_PYPI="${VERSION_SUFFIX_FOR_PYPI}"
docker-compose ${command}
EOF
chmod u+x "${file}"
@@ -824,24 +821,12 @@ function breeze::parse_arguments() {
echo
shift 2
;;
- -d | --debian-version)
- export DEBIAN_VERSION="${2}"
- echo "Debian version: ${DEBIAN_VERSION}"
- echo
- shift 2
- ;;
-b | --backend)
export BACKEND="${2}"
echo "Backend: ${BACKEND}"
echo
shift 2
;;
- --platform)
- export PLATFORM="${2}"
- echo "Backend: ${PLATFORM}"
- echo
- shift 2
- ;;
-i | --integration)
local INTEGRATION=${2}
parameters::check_and_save_allowed_param "INTEGRATION" "integration" "--integration"
@@ -927,14 +912,6 @@ function breeze::parse_arguments() {
echo
shift
;;
- -a | --install-airflow-version)
- INSTALL_AIRFLOW_VERSION="${2}"
- # Reference is mutually exclusive with version
- INSTALL_AIRFLOW_REFERENCE=""
- echo "Installs version of Airflow: ${INSTALL_AIRFLOW_VERSION}"
- echo
- shift 2
- ;;
-t | --install-airflow-reference)
INSTALL_AIRFLOW_REFERENCE="${2}"
# Reference is mutually exclusive with version
@@ -967,182 +944,23 @@ function breeze::parse_arguments() {
shift
;;
-y | --assume-yes)
- export FORCE_ANSWER_TO_QUESTIONS="yes"
+ export ANSWER="yes"
echo "Assuming 'yes' answer to all questions."
echo
shift
;;
-n | --assume-no)
- export FORCE_ANSWER_TO_QUESTIONS="no"
+ export ANSWER="no"
echo "Assuming 'no' answer to all questions."
echo
shift
;;
-q | --assume-quit)
- export FORCE_ANSWER_TO_QUESTIONS="quit"
+ export ANSWER="quit"
echo "Assuming 'quit' answer to all questions."
echo
shift
;;
- -F | --force-build-images)
- echo "Force build images"
- echo
- export FORCE_BUILD_IMAGES="true"
- # if you want to force build an image - assume you want to build it :)
- export FORCE_ANSWER_TO_QUESTIONS="yes"
- shift
- ;;
- -C | --force-clean-images)
- echo "Clean build of images without cache"
- echo
- export DOCKER_CACHE="disabled"
- # if not set here, docker cached is determined later, depending on type of image to be built
- export FORCE_BUILD_IMAGES="true"
- shift
- ;;
- -r | --skip-rebuild-check)
- echo "Skips checking image for rebuilds"
- echo
- export CHECK_IMAGE_FOR_REBUILD="false"
- export SKIP_BUILDING_PROD_IMAGE="true"
- shift
- ;;
- -L | --build-cache-local)
- echo "Use local cache to build images"
- echo
- export DOCKER_CACHE="local"
- # if not set here, docker cached is determined later, depending on type of image to be built
- shift
- ;;
- -U | --build-cache-pulled)
- echo "Use pulled cache to build images"
- echo
- export DOCKER_CACHE="pulled"
- # if not set here, docker cached is determined later, depending on type of image to be built
- shift
- ;;
- -X | --build-cache-disabled)
- echo "Use disabled cache to build images"
- echo
- export DOCKER_CACHE="disabled"
- # if not set here, docker cached is determined later, depending on type of image to be built
- shift
- ;;
- -I | --production-image)
- export PRODUCTION_IMAGE="true"
- export SQLITE_URL=
- echo
- echo "*************** PRODUCTION IMAGE *************************"
- echo
- shift
- ;;
- --disable-pypi-when-building)
- export INSTALL_FROM_PYPI="false"
- export AIRFLOW_PRE_CACHED_PIP_PACKAGES="false"
- echo "Disable installing airflow from PYPI"
- shift
- ;;
- -E | --extras)
- export AIRFLOW_EXTRAS="${2}"
- echo "Extras : ${AIRFLOW_EXTRAS}"
- shift 2
- ;;
- --additional-extras)
- export ADDITIONAL_AIRFLOW_EXTRAS="${2}"
- echo "Additional extras : ${ADDITIONAL_AIRFLOW_EXTRAS}"
- shift 2
- ;;
- --additional-python-deps)
- export ADDITIONAL_PYTHON_DEPS="${2}"
- echo "Additional python dependencies: ${ADDITIONAL_PYTHON_DEPS}"
- shift 2
- ;;
- --dev-apt-deps)
- export DEV_APT_DEPS="${2}"
- echo "Apt dev dependencies: ${DEV_APT_DEPS}"
- shift 2
- ;;
- --additional-dev-apt-deps)
- export ADDITIONAL_DEV_APT_DEPS="${2}"
- echo "Additional apt dev dependencies: ${ADDITIONAL_DEV_APT_DEPS}"
- shift 2
- ;;
- --dev-apt-command)
- export DEV_APT_COMMAND="${2}"
- echo "Apt dev command: ${DEV_APT_COMMAND}"
- shift 2
- ;;
- --additional-dev-apt-command)
- export ADDITIONAL_DEV_APT_COMMAND="${2}"
- echo "Additional Apt dev command: ${ADDITIONAL_DEV_APT_COMMAND}"
- shift 2
- ;;
- --additional-dev-apt-env)
- export ADDITIONAL_DEV_APT_ENV="${2}"
- echo "Additional Apt dev environment variables: ${ADDITIONAL_DEV_APT_ENV}"
- shift 2
- ;;
- --runtime-apt-deps)
- export RUNTIME_APT_DEPS="${2}"
- echo "Apt runtime dependencies: ${RUNTIME_APT_DEPS}"
- shift 2
- ;;
- --additional-runtime-apt-deps)
- export ADDITIONAL_RUNTIME_APT_DEPS="${2}"
- echo "Additional apt runtime dependencies: ${ADDITIONAL_RUNTIME_APT_DEPS}"
- shift 2
- ;;
- --runtime-apt-command)
- export RUNTIME_APT_COMMAND="${2}"
- echo "Apt runtime command: ${RUNTIME_APT_COMMAND}"
- shift 2
- ;;
- --additional-runtime-apt-command)
- export ADDITIONAL_RUNTIME_APT_COMMAND="${2}"
- echo "Additional Apt runtime command: ${ADDITIONAL_RUNTIME_APT_COMMAND}"
- shift 2
- ;;
- --additional-runtime-apt-env)
- export ADDITIONAL_RUNTIME_APT_ENV="${2}"
- echo "Additional Apt runtime environment variables: ${ADDITIONAL_RUNTIME_APT_ENV}"
- shift 2
- ;;
- --disable-mysql-client-installation)
- export INSTALL_MYSQL_CLIENT="false"
- echo "Install MySQL client: ${INSTALL_MYSQL_CLIENT}"
- shift
- ;;
- --disable-mssql-client-installation)
- export INSTALL_MSSQL_CLIENT="false"
- echo "Install MsSQL client: ${INSTALL_MSSQL_CLIENT}"
- shift
- ;;
- --disable-postgres-client-installation)
- export INSTALL_POSTGRES_CLIENT="false"
- echo "Install Postgres client: ${INSTALL_POSTGRES_CLIENT}"
- shift
- ;;
- --constraints-location)
- export AIRFLOW_CONSTRAINTS_LOCATION="${2}"
- echo "Constraints location: ${AIRFLOW_CONSTRAINTS_LOCATION}"
- shift 2
- ;;
- --disable-pip-cache)
- echo "Disable PIP cache during build"
- echo
- export AIRFLOW_PRE_CACHED_PIP_PACKAGES="false"
- shift
- ;;
- --install-from-docker-context-files)
- export INSTALL_FROM_DOCKER_CONTEXT_FILES="true"
- echo "Install wheels from local docker-context-files when building image"
- shift
- ;;
- --image-tag)
- export IMAGE_TAG="${2}"
- echo "Tag to add to the image: ${IMAGE_TAG}"
- shift 2
- ;;
-f | --forward-credentials)
echo "Forwarding credentials. Be careful as your credentials are available in the container!"
echo
@@ -1178,12 +996,6 @@ function breeze::parse_arguments() {
echo
shift 2
;;
- -S | --version-suffix-for-pypi)
- export VERSION_SUFFIX_FOR_PYPI="${2}"
- echo "Version suffix for PyPI ${VERSION_SUFFIX_FOR_PYPI}"
- echo
- shift 2
- ;;
--load-example-dags)
export LOAD_EXAMPLES="true"
echo "Include Airflow sample dags"
@@ -1202,55 +1014,12 @@ function breeze::parse_arguments() {
echo
shift
;;
- --cleanup-docker-context-files)
- export CLEANUP_DOCKER_CONTEXT_FILES="true"
- echo "Preserves data volumes when stopping airflow"
- echo
- shift
- ;;
- --use-packages-from-dist)
- export USE_PACKAGES_FROM_DIST="true"
- echo "Install packages found in dist folder when entering breeze."
- echo
- shift
- ;;
- --upgrade-to-newer-dependencies)
- export UPGRADE_TO_NEWER_DEPENDENCIES="true"
- echo "Upgrade packages to latest versions."
- echo
- shift
- ;;
- --package-format)
- export PACKAGE_FORMAT="${2}"
- echo "Selected package type: ${PACKAGE_FORMAT}"
- echo
- shift 2
- ;;
- --non-interactive)
- export NON_INTERACTIVE="true"
- echo "Running in non-interactive mode"
- echo
- shift
- ;;
- --generate-providers-issue)
- export GENERATE_PROVIDERS_ISSUE="true"
- export FORWARD_CREDENTIALS="true"
- echo "Generating providers issue (includes forwarding credentials)"
- echo
- shift
- ;;
--installation-method)
export AIRFLOW_INSTALLATION_METHOD="${2}"
echo "Airflow installation method: ${AIRFLOW_INSTALLATION_METHOD}"
echo
shift 2
;;
- --generate-constraints-mode)
- export GENERATE_CONSTRAINTS_MODE="${2}"
- echo "Generate constraints mode: ${GENERATE_CONSTRAINTS_MODE}"
- echo
- shift 2
- ;;
--test-type)
export TEST_TYPE="${2}"
echo "Selected test type: ${TEST_TYPE}"
@@ -1289,42 +1058,6 @@ function breeze::parse_arguments() {
command_to_run="run_exec"
shift
;;
- build-docs)
- last_subcommand="${1}"
- command_to_run="build_docs"
- shift
- ;;
- build-image)
- last_subcommand="${1}"
- command_to_run="build_image"
- # if you want to build an image - assume you want to build it :)
- export FORCE_ANSWER_TO_QUESTIONS="yes"
- # and assume you want to build it no matter if it is needed
- export FORCE_BUILD_IMAGES="true"
- echo "Build image"
- echo
- shift
- ;;
- prepare-build-cache)
- last_subcommand="${1}"
- command_to_run="prepare_build_cache"
- export FORCE_ANSWER_TO_QUESTIONS="yes"
- # and assume you want to build it no matter if it is needed
- export FORCE_BUILD_IMAGES="true"
- export PREPARE_BUILDX_CACHE="true"
- export DOCKER_CACHE="pulled"
- export CLEANUP_DOCKER_CONTEXT_FILES="true"
- echo "Prepare buildx cache"
- echo
- shift
- ;;
- cleanup-image)
- last_subcommand="${1}"
- echo "Cleanup the image"
- echo
- command_to_run="cleanup_image"
- shift
- ;;
docker-compose)
last_subcommand="${1}"
if [[ $# -lt 2 ]]; then
@@ -1337,31 +1070,6 @@ function breeze::parse_arguments() {
fi
command_to_run="run_docker_compose"
;;
- generate-constraints)
- last_subcommand="${1}"
- command_to_run="perform_generate_constraints"
- export FORCE_ANSWER_TO_QUESTIONS="yes"
- export FORCE_BUILD_IMAGES="true"
- export SKIP_CHECK_REMOTE_IMAGE="true"
- export UPGRADE_TO_NEWER_DEPENDENCIES="true"
- shift
- ;;
- prepare-airflow-packages)
- last_subcommand="${1}"
- command_to_run="perform_prepare_airflow_packages"
- export INSTALL_PROVIDERS_FROM_SOURCES="false"
- shift
- ;;
- prepare-provider-packages)
- last_subcommand="${1}"
- command_to_run="perform_prepare_provider_packages"
- shift
- ;;
- prepare-provider-documentation)
- last_subcommand="${1}"
- command_to_run="perform_prepare_provider_documentation"
- shift
- ;;
initialize-local-virtualenv)
last_subcommand="${1}"
echo "Initializing local virtualenv"
@@ -1553,18 +1261,10 @@ function breeze::prepare_formatted_versions() {
fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
readonly FORMATTED_PYTHON_MAJOR_MINOR_VERSIONS
- FORMATTED_DEBIAN_VERSIONS=$(echo "${_breeze_allowed_debian_versions=""}" | tr '\n' ' ' |
- fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
- readonly FORMATTED_DEBIAN_VERSIONS
-
FORMATTED_BACKENDS=$(echo "${_breeze_allowed_backends=""}" | tr '\n' ' ' |
fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
readonly FORMATTED_BACKENDS
- FORMATTED_PLATFORMS=$(echo "${_breeze_allowed_platforms=""}" |
- tr '\n' ' ' | fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
- readonly FORMATTED_PLATFORMS
-
FORMATTED_STATIC_CHECKS=$(echo "${_breeze_allowed_static_checks=""}" | tr '\n' ' ' |
fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
readonly FORMATTED_STATIC_CHECKS
@@ -1597,18 +1297,6 @@ function breeze::prepare_formatted_versions() {
fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
readonly FORMATTED_KIND_OPERATIONS
- FORMATTED_INSTALL_AIRFLOW_VERSIONS=$(echo "${_breeze_allowed_install_airflow_versions=""}" |
- tr '\n' ' ' | fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
- readonly FORMATTED_INSTALL_AIRFLOW_VERSIONS
-
- FORMATTED_INSTALLATION_METHOD=$(echo "${_breeze_allowed_installation_methods=""}" |
- tr '\n' ' ' | fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
- readonly FORMATTED_INSTALLATION_METHOD
-
- FORMATTED_GENERATE_CONSTRAINTS_MODE=$(echo "${_breeze_allowed_generate_constraints_modes=""}" |
- tr '\n' ' ' | fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
- readonly FORMATTED_GENERATE_CONSTRAINTS_MODE
-
FORMATTED_POSTGRES_VERSIONS=$(echo "${_breeze_allowed_postgres_versions=""}" |
tr '\n' ' ' | fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
readonly FORMATTED_POSTGRES_VERSIONS
@@ -1621,14 +1309,6 @@ function breeze::prepare_formatted_versions() {
tr '\n' ' ' | fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
readonly FORMATTED_MSSQL_VERSIONS
- FORMATTED_DEFAULT_CI_EXTRAS=$(echo "${DEFAULT_CI_EXTRAS=}" |
- tr ',' ' ' | fold -w "${indented_screen_width}" -s | sed "s/ /,/g; s/^/${list_prefix}/")
- readonly FORMATTED_DEFAULT_CI_EXTRAS
-
- FORMATTED_DEFAULT_PROD_EXTRAS=$(echo "${DEFAULT_PROD_EXTRAS=}" |
- tr ',' ' ' | fold -w "${indented_screen_width}" -s | sed "s/ /,/g; s/^/${list_prefix}/")
- readonly FORMATTED_DEFAULT_PROD_EXTRAS
-
FORMATTED_TEST_TYPES=$(echo "${_breeze_allowed_test_types=""}" |
fold -w "${indented_screen_width}" -s | sed "s/ /,/g; s/^/${list_prefix}/")
readonly FORMATTED_TEST_TYPES
@@ -1672,26 +1352,12 @@ function breeze::prepare_usage() {
readonly USAGE_EXEC
export USAGE_BUILD_DOCS="Builds documentation in the container"
readonly USAGE_BUILD_DOCS
- export USAGE_BUILD_IMAGE="Builds CI or Production docker image"
- readonly USAGE_BUILD_DOCS
- export USAGE_PREPARE_BUILD_CACHE="Prepares CI or Production build cache"
- readonly USAGE_PREPARE_BUILD_CACHE
- export USAGE_CLEANUP_IMAGE="Cleans up the container image created"
- readonly USAGE_BUILD_DOCS
export USAGE_DOCKER_COMPOSE="Executes specified docker-compose command"
readonly USAGE_DOCKER_COMPOSE
export USAGE_FLAGS="Shows all breeze's flags"
readonly USAGE_FLAGS
- export USAGE_GENERATE_CONSTRAINTS="Generates pinned constraint files"
- readonly USAGE_GENERATE_CONSTRAINTS
export USAGE_INITIALIZE_LOCAL_VIRTUALENV="Initializes local virtualenv"
readonly USAGE_INITIALIZE_LOCAL_VIRTUALENV
- export USAGE_PREPARE_PROVIDER_DOCUMENTATION="Prepares provider packages documentation"
- readonly USAGE_PREPARE_PROVIDER_DOCUMENTATION
- export USAGE_PREPARE_AIRFLOW_PACKAGES="Prepares airflow packages"
- readonly USAGE_PREPARE_AIRFLOW_PACKAGES
- export USAGE_PREPARE_PROVIDER_PACKAGES="Prepares provider packages"
- readonly USAGE_PREPARE_PROVIDER_PACKAGES
export USAGE_PUSH_IMAGE="Pushes images to registry"
readonly USAGE_PUSH_IMAGE
export USAGE_KIND_CLUSTER="Manages KinD cluster on the host"
@@ -1750,7 +1416,6 @@ ${CMDNAME} shell [FLAGS] [-- <EXTRA_ARGS>]
'${CMDNAME} \\
--github-image-id 9a621eaa394c0a0a336f8e1b31b35eff4e4ee86e' - pull/use image with SHA
-Most flags are applicable to the shell command as it will run build when needed.
"
readonly DETAILED_USAGE_SHELL
export DETAILED_USAGE_EXEC="
@@ -1761,83 +1426,6 @@ ${CMDNAME} exec [-- <EXTRA_ARGS>]
way to run multiple processes in the same container at the same time for example scheduler,
webserver, workers, database console and interactive terminal.
"
- export DETAILED_USAGE_BUILD_DOCS="
-${CMDNAME} build-docs [-- <EXTRA_ARGS>]
-
- Builds Airflow documentation. The documentation is build inside docker container - to
- maintain the same build environment for everyone. Appropriate sources are mapped from
- the host to the container so that latest sources are used. The folders where documentation
- is generated ('docs/_build') are also mounted to the container - this way results of
- the documentation build is available in the host.
-
- The possible extra args are: --docs-only, --spellcheck-only, --package-filter, --help
-"
- readonly DETAILED_USAGE_BUILD_DOCS
- export DETAILED_USAGE_BUILD_IMAGE="
-${CMDNAME} build-image [FLAGS]
-
- Builds docker image (CI or production) without entering the container. You can pass
- additional options to this command, such as:
-
- Choosing python version:
- '--python'
-
- Choosing cache option:
- '--build-cache-local' or '-build-cache-pulled', or '--build-cache-none'
-
- Choosing whether to force pull images or force build the image:
- '--force-build-image'
-
- You can also pass '--production-image' flag to build production image rather than CI image.
-
- For GitHub repository, the '--github-repository' can be used to choose repository
- to pull/push images.
-
-Flags:
-$(breeze::flag_airflow_variants)
-$(breeze::flag_build_different_airflow_version)
-$(breeze::flag_production_image)
-$(breeze::flag_build_docker_images)
-$(breeze::flag_pull_push_docker_images "no_show_sha")
-$(breeze::flag_verbosity)
-"
- readonly DETAILED_USAGE_BUILD_IMAGE
- export DETAILED_USAGE_PREPARE_BUILD_CACHE="
-${CMDNAME} prepare-build-cache [FLAGS]
-
- Prepares build cache (CI or production) without entering the container. You can pass
- additional options to this command, such as:
-
- Choosing python version:
- '--python'
-
- You can also pass '--production-image' flag to build production image rather than CI image.
-
- For GitHub repository, the '--github-repository' can be used to choose repository
- to pull/push images. Cleanup docker context files and pull cache are forced. This command
- requires buildx to be installed.
-
-Flags:
-$(breeze::flag_airflow_variants)
-$(breeze::flag_build_different_airflow_version)
-$(breeze::flag_production_image)
-$(breeze::flag_pull_push_docker_images "no_show_sha")
-$(breeze::flag_verbosity)
-"
- readonly DETAILED_USAGE_PREPARE_BUILD_CACHE
- export DETAILED_USAGE_CLEANUP_IMAGE="
-${CMDNAME} cleanup-image [FLAGS]
-
- Removes the breeze-related images created in your local docker image cache. This will
- not reclaim space in docker cache. You need to 'docker system prune' (optionally
- with --all) to reclaim that space.
-
-Flags:
-$(breeze::flag_airflow_variants)
-$(breeze::flag_production_image)
-$(breeze::flag_verbosity)
-"
- readonly DETAILED_USAGE_CLEANUP_IMAGE
export DETAILED_USAGE_DOCKER_COMPOSE="
${CMDNAME} docker-compose [FLAGS] COMMAND [-- <EXTRA_ARGS>]
@@ -1857,128 +1445,6 @@ $(breeze::flag_verbosity)
Explains in detail all the flags that can be used with breeze.
"
readonly DETAILED_USAGE_FLAGS
- export DETAILED_USAGE_PREPARE_PROVIDER_DOCUMENTATION="
-${CMDNAME} prepare-provider-documentation [FLAGS] [PACKAGE_ID ...]
-
- Prepares documentation files for provider packages.
-
- The command is optionally followed by the list of packages to generate readme for.
- If the first parameter is not formatted as a date, then today is regenerated.
- If no packages are specified, readme for all packages are generated.
- If no date is specified, current date + 3 days is used (allowing for PMC votes to pass).
-
- Examples:
-
- '${CMDNAME} prepare-provider-documentation' or
- '${CMDNAME} prepare-provider-documentation --version-suffix-for-pypi rc1'
-
- General form:
-
- '${CMDNAME} prepare-provider-documentation <PACKAGE_ID> ...'
-
- * <PACKAGE_ID> is usually directory in the airflow/providers folder (for example
- 'google' but in several cases, it might be one level deeper separated with
- '.' for example 'apache.hive'
-
-Flags:
-$(breeze::flag_version_suffix)
-$(breeze::flag_packages)
-$(breeze::prepare_providers_documentation)
-$(breeze::flag_verbosity)
-"
- readonly DETAILED_USAGE_PREPARE_PROVIDER_DOCUMENTATION
- export DETAILED_USAGE_GENERATE_CONSTRAINTS="
-${CMDNAME} generate-constraints [FLAGS]
-
- Generates pinned constraint files with all extras from setup.py. Those files are generated in
- files folder - separate files for different python version. Those constraint files when
- pushed to orphan constraints-main, constraints-2-0 branches are used
- to generate repeatable CI test runs as well as run repeatable production image builds and
- upgrades when you want to include installing or updating some of the released providers
- released at the time particular airflow version was released. You can use those
- constraints to predictably install released Airflow versions. This is mainly used to test
- the constraint generation or manually fix them - constraints are pushed to the orphan
- branches by a successful scheduled CRON job in CI automatically, but sometimes manual fix
- might be needed.
-
-Flags:
-$(breeze::flag_generate_constraints)
-$(breeze::flag_airflow_variants)
-$(breeze::flag_verbosity)
-"
- readonly DETAILED_USAGE_GENERATE_CONSTRAINTS
- export DETAILED_USAGE_INITIALIZE_LOCAL_VIRTUALENV="
-${CMDNAME} initialize-local-virtualenv [FLAGS]
-
- Initializes locally created virtualenv installing all dependencies of Airflow
- taking into account the constraints for the version specified.
- This local virtualenv can be used to aid auto-completion and IDE support as
- well as run unit tests directly from the IDE. You need to have virtualenv
- activated before running this command.
-
-Flags:
-$(breeze::flag_airflow_variants)
-"
- readonly DETAILED_USAGE_INITIALIZE_LOCAL_VIRTUALENV
- export DETAILED_USAGE_PREPARE_AIRFLOW_PACKAGES="
-${CMDNAME} prepare-airflow-packages [FLAGS]
-
- Prepares airflow packages (sdist and wheel) in dist folder. Note that
- prepare-provider-packages command cleans up the dist folder, so if you want also
- to generate provider packages, make sure you run prepare-provider-packages first,
- and prepare-airflow-packages second. You can specify optional
- --version-suffix-for-pypi flag to generate rc candidates for PyPI packages.
- The packages are prepared in dist folder
-
- Examples:
-
- '${CMDNAME} prepare-airflow-packages --package-format wheel' or
- '${CMDNAME} prepare-airflow-packages --version-suffix-for-pypi rc1'
-
-Flags:
-$(breeze::flag_packages)
-$(breeze::flag_version_suffix)
-$(breeze::flag_verbosity)
-"
- readonly DETAILED_USAGE_PREPARE_AIRFLOW_PACKAGES
- export DETAILED_USAGE_PREPARE_PROVIDER_PACKAGES="
-${CMDNAME} prepare-provider-packages [FLAGS] [PACKAGE_ID ...]
-
- Prepares provider packages. You can provide (after --) optional list of packages to prepare.
- If no packages are specified, readme for all packages are generated. You can specify optional
- --version-suffix-for-svn flag to generate rc candidate packages to upload to SVN or
- --version-suffix-for-pypi flag to generate rc candidates for PyPI packages. You can also
- provide both suffixes in case you prepare alpha/beta versions. The packages are prepared in
- dist folder. Note that this command also cleans up dist folder before generating the packages
- so that you do not have accidental files there. This will delete airflow package if it is
- prepared there so make sure you run prepare-provider-packages first,
- and prepare-airflow-packages second.
-
- Examples:
-
- '${CMDNAME} prepare-provider-packages' or
- '${CMDNAME} prepare-provider-packages google' or
- '${CMDNAME} prepare-provider-packages --package-format wheel google' or
- '${CMDNAME} prepare-provider-packages --version-suffix-for-svn rc1 http google amazon' or
- '${CMDNAME} prepare-provider-packages --version-suffix-for-pypi rc1 http google amazon'
- '${CMDNAME} prepare-provider-packages --version-suffix-for-pypi a1
- --version-suffix-for-svn a1 http google amazon'
-
- General form:
-
- '${CMDNAME} prepare-provider-packages [--package-format PACKAGE_FORMAT] \\
- [--version-suffix-for-svn|--version-suffix-for-pypi] <PACKAGE_ID> ...'
-
- * <PACKAGE_ID> is usually directory in the airflow/providers folder (for example
- 'google'), but in several cases, it might be one level deeper separated with '.'
- for example 'apache.hive'
-
-Flags:
-$(breeze::flag_packages)
-$(breeze::flag_version_suffix)
-$(breeze::flag_verbosity)
-"
- readonly DETAILED_USAGE_PREPARE_PROVIDER_PACKAGES
export DETAILED_USAGE_KIND_CLUSTER="
${CMDNAME} kind-cluster [FLAGS] OPERATION
@@ -2265,22 +1731,6 @@ function breeze::flag_airflow_variants() {
One of:
${FORMATTED_PYTHON_MAJOR_MINOR_VERSIONS}
-
---platform PLATFORM
- Builds image for the platform specified.
-
- One of:
-
-${FORMATTED_PLATFORMS}
-
-
--d, --debian DEBIAN_VERSION
- Debian version used for the image. This is always name of the debian distribution version.
-
- One of:
-
-${FORMATTED_DEBIAN_VERSIONS}
-
"
}
@@ -2453,43 +1903,7 @@ function breeze::flag_local_file_mounting() {
#######################################################################################################
#
-# Prints flags that allow to build different airflow variants
-#
-# Global constants used:
-# FORMATTED_INSTALL_AIRFLOW_VERSIONS
-#
-#
-# Outputs:
-# Flag information.
-#######################################################################################################
-function breeze::flag_build_different_airflow_version() {
- echo "
--a, --install-airflow-version INSTALL_AIRFLOW_VERSION
- Uses different version of Airflow when building PROD image.
-
-${FORMATTED_INSTALL_AIRFLOW_VERSIONS}
-
--t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
- Installs Airflow directly from reference in GitHub when building PROD image.
- This can be a GitHub branch like main or v2-2-test, or a tag like 2.2.0rc1.
-
---installation-method INSTALLATION_METHOD
- Method of installing Airflow in PROD image - either from the sources ('.')
- or from package 'apache-airflow' to install from PyPI.
- Default in Breeze is to install from sources. One of:
-
-${FORMATTED_INSTALLATION_METHOD}
-
---upgrade-to-newer-dependencies
- Upgrades PIP packages to latest versions available without looking at the constraints.
-
-"
-}
-
-
-#######################################################################################################
-#
-# Prints flags that allow to build different airflow variants
+# Prints flags that allow to use different airflow variants
#
# Global constants used:
# FORMATTED_INSTALL_AIRFLOW_VERSIONS
@@ -2516,32 +1930,6 @@ ${FORMATTED_USE_AIRFLOW_VERSION}
"
}
-
-#######################################################################################################
-#
-# Prints flags that allow to choose variants of constraint generation
-#
-# Global constants used:
-# GENERATE_CONSTRAINTS_MODES
-#
-# Outputs:
-# Flag information.
-#######################################################################################################
-function breeze::flag_generate_constraints() {
- echo "
---generate-constraints-mode GENERATE_CONSTRAINTS_MODE
- Mode of generating constraints - determines whether providers are installed when generating
- constraints and which version of them (either the ones from sources are used or the ones
- from pypi.
-
- One of:
-
-${FORMATTED_GENERATE_CONSTRAINTS_MODE}
-
-"
-}
-
-
#######################################################################################################
#
# Prints flags that allow to set assumed answers to questions
@@ -2599,26 +1987,6 @@ ${FORMATTED_PACKAGE_FORMATS}
"
}
-#######################################################################################################
-#
-# Prints flags for prepare-providers-documentation
-#
-# Outputs:
-# Flag information.
-#######################################################################################################
-function breeze::prepare_providers_documentation() {
- echo "
---non-interactive
-
- Runs the command in non-interactive mode.
-
---generate-providers-issue
-
- Generate providers issue that should be created.
-
-"
-}
-
#######################################################################################################
#
# Prints flags that control verbosity
@@ -2657,197 +2025,6 @@ function breeze::flag_help() {
"
}
-#######################################################################################################
-#
-# Prints flags controlling docker build process
-#
-# Global constants used:
-# FORMATTED_DEFAULT_CI_EXTRAS
-# FORMATTED_DEFAULT_PROD_EXTRAS
-#
-# Outputs:
-# Flag information.
-#######################################################################################################
-function breeze::flag_build_docker_images() {
- echo "
--F, --force-build-images
- Forces building of the local docker images. The images are rebuilt
- automatically for the first time or when changes are detected in
- package-related files, but you can force it using this flag.
-
---cleanup-docker-context-files
- Removes whl and tar.gz files created in docker-context-files before running the command.
- In case there are some files there it unnecessarily increases the context size and
- makes the COPY . always invalidated - if you happen to have those files when you build your
- image.
-
-Customization options:
-
--E, --extras EXTRAS
- Extras to pass to build images The default are different for CI and production images:
-
- CI image:
-${FORMATTED_DEFAULT_CI_EXTRAS}
-
- Production image:
-${FORMATTED_DEFAULT_PROD_EXTRAS}
-
---image-tag TAG
- Additional tag in the image.
-
---disable-pypi-when-building
- Disable installing Airflow from pypi when building. If you use this flag and want
- to install Airflow, you have to install it from packages placed in
- 'docker-context-files' and use --install-from-docker-context-files flag.
-
---additional-extras ADDITIONAL_EXTRAS
- Additional extras to pass to build images The default is no additional extras.
-
---additional-python-deps ADDITIONAL_PYTHON_DEPS
- Additional python dependencies to use when building the images.
-
---dev-apt-command DEV_APT_COMMAND
- The basic command executed before dev apt deps are installed.
-
---additional-dev-apt-command ADDITIONAL_DEV_APT_COMMAND
- Additional command executed before dev apt deps are installed.
-
---additional-dev-apt-deps ADDITIONAL_DEV_APT_DEPS
- Additional apt dev dependencies to use when building the images.
-
---dev-apt-deps DEV_APT_DEPS
- The basic apt dev dependencies to use when building the images.
-
---additional-dev-apt-deps ADDITIONAL_DEV_DEPS
- Additional apt dev dependencies to use when building the images.
-
---additional-dev-apt-envs ADDITIONAL_DEV_APT_ENVS
- Additional environment variables set when adding dev dependencies.
-
---runtime-apt-command RUNTIME_APT_COMMAND
- The basic command executed before runtime apt deps are installed.
-
---additional-runtime-apt-command ADDITIONAL_RUNTIME_APT_COMMAND
- Additional command executed before runtime apt deps are installed.
-
---runtime-apt-deps ADDITIONAL_RUNTIME_APT_DEPS
- The basic apt runtime dependencies to use when building the images.
-
---additional-runtime-apt-deps ADDITIONAL_RUNTIME_DEPS
- Additional apt runtime dependencies to use when building the images.
-
---additional-runtime-apt-envs ADDITIONAL_RUNTIME_APT_DEPS
- Additional environment variables set when adding runtime dependencies.
-
-Build options:
-
---disable-mysql-client-installation
- Disables installation of the mysql client which might be problematic if you are building
- image in controlled environment. Only valid for production image.
-
---disable-mssql-client-installation
- Disables installation of the mssql client which might be problematic if you are building
- image in controlled environment. Only valid for production image.
-
---constraints-location
- Url to the constraints file. In case of the production image it can also be a path to the
- constraint file placed in 'docker-context-files' folder, in which case it has to be
- in the form of '/docker-context-files/<NAME_OF_THE_FILE>'
-
---disable-pip-cache
- Disables GitHub PIP cache during the build. Useful if GitHub is not reachable during build.
-
---install-from-docker-context-files
- This flag is used during image building. If it is used additionally to installing
- Airflow from PyPI, the packages are installed from the .whl and .tar.gz packages placed
- in the 'docker-context-files' folder. The same flag can be used during entering the image in
- the CI image - in this case also the .whl and .tar.gz files will be installed automatically
-
--C, --force-clean-images
- Force build images with cache disabled. This will remove the pulled or build images
- and start building images from scratch. This might take a long time.
-
--r, --skip-rebuild-check
- Skips checking image for rebuilds. It will use whatever image is available locally/pulled.
-
--L, --build-cache-local
- Uses local cache to build images. No pulled images will be used, but results of local
- builds in the Docker cache are used instead. This will take longer than when the pulled
- cache is used for the first time, but subsequent '--build-cache-local' builds will be
- faster as they will use mostly the locally build cache.
-
- This is default strategy used by the Production image builds.
-
--U, --build-cache-pulled
- Uses images pulled from GitHub Container Registry to build images.
- Those builds are usually faster than when ''--build-cache-local'' with the exception if
- the registry images are not yet updated. The images are updated after successful merges
- to main.
-
- This is default strategy used by the CI image builds.
-
--X, --build-cache-disabled
- Disables cache during docker builds. This is useful if you want to make sure you want to
- rebuild everything from scratch.
-
- This strategy is used by default for both Production and CI images for the scheduled
- (nightly) builds in CI.
-
-"
-}
-
-#######################################################################################################
-#
-# Prints flags controlling docker pull and push process
-#
-# Global constants used:
-# _breeze_default_github_repository
-# _breeze_default_github_image_id
-#
-# Outputs:
-# Flag information.
-#######################################################################################################
-function breeze::flag_pull_push_docker_images() {
- local show_sha="${1-'show_sha'}"
- echo "
--g, --github-repository GITHUB_REPOSITORY
- GitHub repository used to pull, push images.
- Default: ${_breeze_default_github_repository:=}.
-
-"
-
- [[ "$show_sha" != "no_show_sha" ]] && echo "
-
--s, --github-image-id COMMIT_SHA
- <COMMIT_SHA> of the image. Images in GitHub registry are stored with those
- to be able to easily find the image for particular CI runs. Once you know the
- <COMMIT_SHA>, you can specify it in github-image-id flag and Breeze will
- automatically pull and use that image so that you can easily reproduce a problem
- that occurred in CI.
-
- Default: ${_breeze_default_github_image_id:=}.
-
-"
-}
-
-#######################################################################################################
-#
-# Prints flags that control version of generated packages
-#
-# Outputs:
-# Flag information.
-#######################################################################################################
-function breeze::flag_version_suffix() {
- echo "
--S, --version-suffix-for-pypi SUFFIX
- Adds optional suffix to the version in the generated provider package. It can be used
- to generate rc1/rc2 ... versions of the packages to be uploaded to PyPI.
-
--N, --version-suffix-for-svn SUFFIX
- Adds optional suffix to the generated names of package. It can be used to generate
- rc1/rc2 ... versions of the packages to be uploaded to SVN.
-"
-}
#####################################################################################################
#
# Prints flags that control how Airflow should be populated with the command start-airflow
@@ -2954,10 +2131,6 @@ $(breeze::print_star_line)
Assume answers to questions
$(breeze::flag_assume_answers_to_questions)
-$(breeze::print_star_line)
- Install different Airflow version during PROD image build
-$(breeze::flag_build_different_airflow_version)
-
$(breeze::print_star_line)
Use different Airflow version at runtime in CI image
$(breeze::flag_use_different_airflow_version)
@@ -2966,22 +2139,10 @@ $(breeze::print_star_line)
Credentials
$(breeze::flag_credentials)
-$(breeze::print_star_line)
- Flags for building Docker images (both CI and production)
-$(breeze::flag_build_docker_images)
-
-$(breeze::print_star_line)
- Flags for pulling/pushing Docker images (both CI and production)
-$(breeze::flag_pull_push_docker_images "show_sha")
-
$(breeze::print_star_line)
Flags for running tests
$(breeze::flag_tests)
-$(breeze::print_star_line)
- Flags for generation of the provider packages
-$(breeze::flag_version_suffix)
-
$(breeze::print_star_line)
Increase verbosity of the scripts
$(breeze::flag_verbosity)
@@ -3260,36 +2421,6 @@ function breeze::make_sure_precommit_is_installed() {
export PATH="${PATH}":~/.local/bin
}
-#######################################################################################################
-#
-# Removes CI and PROD images and cleans up the flag that indicates that the image was already built
-#
-# Used global constants:
-# PYTHON_BASE_IMAGE
-# AIRFLOW_CI_IMAGE
-# AIRFLOW_PROD_IMAGE
-# BUILT_CI_IMAGE_FLAG_FILE
-#
-#######################################################################################################
-function breeze::remove_images() {
- # shellcheck disable=SC2086
- docker rmi --force ${PYTHON_BASE_IMAGE} \
- ${AIRFLOW_PYTHON_BASE_IMAGE} \
- ${AIRFLOW_CI_LOCAL_MANIFEST_IMAGE} \
- ${AIRFLOW_CI_REMOTE_MANIFEST_IMAGE} \
- ${AIRFLOW_CI_IMAGE} \
- ${AIRFLOW_PROD_IMAGE} \
- ${AIRFLOW_PROD_BUILD_IMAGE} \
- 2>/dev/null >/dev/null || true
- docker system prune || true
- echo
- echo "#########################################################################################"
- echo "NOTE!! Removed Airflow images for Python version ${PYTHON_MAJOR_MINOR_VERSION} and pruned cache."
- echo "#########################################################################################"
- echo
- rm -f "${BUILT_CI_IMAGE_FLAG_FILE}"
-}
-
#######################################################################################################
#
# Runs chosen static checks
@@ -3335,57 +2466,10 @@ function breeze::run_static_checks() {
function breeze::run_build_command() {
case "${command_to_run}" in
run_tests | run_docker_compose)
- build_images::prepare_ci_build
- build_images::rebuild_ci_image_if_needed
;;
enter_breeze)
- if [[ ${PRODUCTION_IMAGE} == "true" ]]; then
- build_images::prepare_prod_build
- else
- build_images::prepare_ci_build
- build_images::rebuild_ci_image_if_needed
- fi
;;
- build_docs | perform_static_checks | perform_generate_constraints | \
- perform_prepare_provider_documentation | perform_prepare_provider_packages | \
- perform_prepare_airflow_packages)
- build_images::prepare_ci_build
- build_images::rebuild_ci_image_if_needed
- ;;
- build_image)
- if [[ ${CLEANUP_DOCKER_CONTEXT_FILES} == "true" ]]; then
- build_images::cleanup_docker_context_files
- fi
- build_images::check_for_docker_context_files
- if [[ ${PRODUCTION_IMAGE} == "true" ]]; then
- build_images::prepare_prod_build
- build_images::build_prod_images
- else
-
- build_images::prepare_ci_build
- build_images::rebuild_ci_image_if_needed
- fi
- ;;
- prepare_build_cache)
- if [[ ${CLEANUP_DOCKER_CONTEXT_FILES} == "true" ]]; then
- build_images::cleanup_docker_context_files
- fi
- build_images::check_for_docker_context_files
-
- if [[ ${PRODUCTION_IMAGE} == "true" ]]; then
- build_images::prepare_prod_build
- build_images::build_prod_images
- else
- build_images::prepare_ci_build
- build_images::rebuild_ci_image_if_needed
- fi
- ;;
- cleanup_image | run_exec)
- if [[ ${PRODUCTION_IMAGE} == "true" ]]; then
- build_images::prepare_prod_build
- else
- build_images::prepare_ci_build
- fi
+ build_docs | perform_static_checks )
;;
perform_initialize_local_virtualenv | perform_setup_autocomplete | \
toggle_suppress_cheatsheet | toggle_suppress_asciiart )
@@ -3403,8 +2487,6 @@ function breeze::run_build_command() {
echo "Checks status of KinD cluster"
elif [[ ${KIND_CLUSTER_OPERATION} == "deploy" ]]; then
echo "Deploys Airflow to KinD cluster"
- build_images::prepare_prod_build
- build_images::build_prod_images
elif [[ ${KIND_CLUSTER_OPERATION} == "test" ]]; then
echo "Run Kubernetes tests with the KinD cluster "
elif [[ ${KIND_CLUSTER_OPERATION} == "shell" ]]; then
@@ -3534,27 +2616,6 @@ function breeze::run_breeze_command() {
breeze::make_sure_precommit_is_installed
breeze::run_static_checks "${@}"
;;
- build_image) ;;
- prepare_build_cache) ;;
- cleanup_image)
- breeze::remove_images
- ;;
- perform_generate_constraints)
- docker_engine_resources::check_all_resources
- runs::run_generate_constraints
- ;;
- perform_prepare_airflow_packages)
- docker_engine_resources::check_all_resources
- runs::run_prepare_airflow_packages
- ;;
- perform_prepare_provider_packages)
- docker_engine_resources::check_all_resources
- runs::run_prepare_provider_packages "${@}"
- ;;
- perform_prepare_provider_documentation)
- docker_engine_resources::check_all_resources
- runs::run_prepare_provider_documentation "${@}"
- ;;
perform_initialize_local_virtualenv)
breeze::initialize_virtualenv
;;
@@ -3644,6 +2705,8 @@ initialization::get_environment_for_builds_on_ci
breeze::determine_python_version_to_use_in_breeze
+initialization::get_docker_cache_image_names
+
sanity_checks::basic_sanity_checks
start_end::script_start
@@ -3657,10 +2720,6 @@ breeze::print_header_line
breeze::check_and_save_all_params
-build_images::determine_docker_cache_strategy
-
-build_images::get_docker_cache_image_names
-
initialization::make_constants_read_only
sanity_checks::sanitize_mounted_files
diff --git a/dev/PROVIDER_PACKAGE_DETAILS.md b/dev/PROVIDER_PACKAGE_DETAILS.md
index 8717e86836..1137f9dead 100644
--- a/dev/PROVIDER_PACKAGE_DETAILS.md
+++ b/dev/PROVIDER_PACKAGE_DETAILS.md
@@ -37,7 +37,7 @@ The provider packages are packages (per `provider`) that are not part of the cor
Once you release the packages, you can simply install them with:
-```
+```bash
pip install apache-airflow-providers-<PROVIDER>[<EXTRAS>]
```
@@ -61,8 +61,8 @@ Details to be hashed out in [the related issue](https://github.com/apache/airflo
When you want to prepare release notes for a package, you need to run:
-```
-./breeze-legacy prepare-provider-documentation <PACKAGE_ID> ...
+```bash
+breeze prepare-provider-documentation <PACKAGE_ID> ...
```
The version for each package is going to be updated separately for each package when we agree to the
@@ -79,8 +79,8 @@ If you do not change version number, you can iterate with merges and release can
release date without providing
the date (to update the existing release notes)
-```
-./breeze-legacy prepare-provider-documentation google
+```bash
+breeze prepare-provider-documentation google
```
@@ -119,47 +119,34 @@ the folders (for example Apache Hive's PACKAGE_ID is `apache.hive` ). You can se
providers by running:
```bash
-./breeze-legacy prepare-provider-packages -- --help
+breeze prepare-provider-packages --help
```
The examples below show how you can build selected packages, but you can also build all packages by
omitting the package ids altogether.
-* To build the release candidate packages for SVN Apache upload run the following command:
-
-```bash
-./breeze-legacy prepare-provider-packages package-format both --version-suffix-for-svn=rc1 [PACKAGE_ID] ...
-```
-
-for example:
-
-```bash
-./breeze-legacy prepare-provider-packages package-format both --version-suffix-for-svn=rc1 http ...
-```
-
* To build the release candidate packages for PyPI upload run the following command:
```bash
-./breeze-legacy prepare-provider-packages package-format both --version-suffix-for-pypi=rc1 [PACKAGE_ID] ...
+breeze prepare-provider-packages --package-format both --version-suffix-for-pypi=rc1 [PACKAGE_ID] ...
```
for example:
```bash
-./breeze-legacy prepare-provider-packages package-format both --version-suffix-for-pypi=rc1 http ...
+breeze prepare-provider-packages --package-format both --version-suffix-for-pypi=rc1 http ...
```
-
* To build the final release packages run the following command:
```bash
-./breeze-legacy prepare-provider-packages package-format both [PACKAGE_ID] ...
+breeze prepare-provider-packages --package-format both [PACKAGE_ID] ...
```
for example:
```bash
-./breeze-legacy prepare-provider-packages package-format both http ...
+breeze prepare-provider-packages --package-format both http ...
```
* For each package, this creates a wheel package and source distribution package in your `dist` folder with
diff --git a/dev/README_RELEASE_AIRFLOW.md b/dev/README_RELEASE_AIRFLOW.md
index 5237589d0b..6e10030f7c 100644
--- a/dev/README_RELEASE_AIRFLOW.md
+++ b/dev/README_RELEASE_AIRFLOW.md
@@ -49,7 +49,7 @@
- [Close the milestone](#close-the-milestone)
- [Announce the release on the community slack](#announce-the-release-on-the-community-slack)
- [Tweet about the release](#tweet-about-the-release)
- - [Update `main` with latest release details](#update-main-with-latest-release-details)
+ - [Update `main` with the latest release details](#update-main-with-the-latest-release-details)
- [Update default Airflow version in the helm chart](#update-default-airflow-version-in-the-helm-chart)
- [Update airflow/config_templates/config.yml file](#update-airflowconfig_templatesconfigyml-file)
@@ -248,7 +248,7 @@ The Release Candidate artifacts we vote upon should be the exact ones we vote ag
- Generate SHA512/ASC (If you have not generated a key yet, generate it by following instructions on http://www.apache.org/dev/openpgp.html#key-gen-generate-key)
```shell script
- ./breeze-legacy prepare-airflow-packages --package-format both
+ breeze prepare-airflow-package --package-format both
pushd dist
${AIRFLOW_REPO_ROOT}/dev/sign.sh *
popd
@@ -423,7 +423,7 @@ To do this we need to
- Build the package:
```shell script
- ./breeze-legacy prepare-airflow-packages --version-suffix-for-pypi "${VERSION_SUFFIX}" --package-format both
+ breeze prepare-airflow-package --version-suffix-for-pypi "${VERSION_SUFFIX}" --package-format both
```
- Verify the artifacts that would be uploaded:
@@ -1076,11 +1076,11 @@ Thanks to all the contributors who made this possible.
EOF
```
-## Update `main` with latest release details
+## Update `main` with the latest release details
This includes:
-- Modify `./scripts/ci/pre-commit/supported_versions.py` and let pre-commit do the job
+- Modify `./scripts/ci/pre_commit/pre_commit_supported_versions.py` and let pre-commit do the job
- Sync `RELEASE_NOTES.rst` (including deleting relevant `newsfragments`) and `README.md` changes
- Updating issue templates in `.github/ISSUE_TEMPLATE/` with the new version
- Updating `Dockerfile` with the new version
diff --git a/dev/README_RELEASE_PROVIDER_PACKAGES.md b/dev/README_RELEASE_PROVIDER_PACKAGES.md
index edb1647fe2..7669f15345 100644
--- a/dev/README_RELEASE_PROVIDER_PACKAGES.md
+++ b/dev/README_RELEASE_PROVIDER_PACKAGES.md
@@ -86,7 +86,7 @@ Details about maintaining the SEMVER version are going to be discussed and imple
```shell script
-./breeze-legacy prepare-provider-documentation [packages]
+breeze prepare-provider-documentation [packages]
```
This command will not only prepare documentation but will also help the release manager to review
@@ -107,7 +107,7 @@ When you want to regenerate the changes before the release and make sure all cha
are updated, run it in non-interactive mode:
```shell script
-./breeze-legacy --non-interactive prepare-provider-documentation [packages]
+breeze prepare-provider-documentation --answer yes [packages]
```
## Build provider packages for SVN apache upload
@@ -132,13 +132,13 @@ rm -rf ${AIRFLOW_REPO_ROOT}/dist/*
* Release candidate packages:
```shell script
-./breeze-legacy prepare-provider-packages --package-format both
+breeze prepare-provider-packages --package-format both
```
if you only build few packages, run:
```shell script
-./breeze-legacy prepare-provider-packages --package-format both PACKAGE PACKAGE ....
+breeze prepare-provider-packages --package-format both PACKAGE PACKAGE ....
```
* Sign all your packages
@@ -197,14 +197,13 @@ this will clean up dist folder before generating the packages, so you will only
```shell script
rm -rf ${AIRFLOW_REPO_ROOT}/dist/*
-./breeze-legacy prepare-provider-packages --version-suffix-for-pypi rc1 --package-format both
+breeze prepare-provider-packages --version-suffix-for-pypi rc1 --package-format both
```
if you only build few packages, run:
```shell script
-./breeze-legacy prepare-provider-packages --version-suffix-for-pypi rc1 --package-format both \
- PACKAGE PACKAGE ....
+breeze prepare-provider-packages --version-suffix-for-pypi rc1 --package-format both PACKAGE PACKAGE ....
```
* Verify the artifacts that would be uploaded:
diff --git a/dev/REFRESHING_CI_CACHE.md b/dev/REFRESHING_CI_CACHE.md
index b57d30130c..a1b5c45122 100644
--- a/dev/REFRESHING_CI_CACHE.md
+++ b/dev/REFRESHING_CI_CACHE.md
@@ -24,6 +24,9 @@
- [Automated cache refreshing in CI](#automated-cache-refreshing-in-ci)
- [Manually generating constraint files](#manually-generating-constraint-files)
- [Manually refreshing the images](#manually-refreshing-the-images)
+ - [Setting up cache refreshing with emulation](#setting-up-cache-refreshing-with-emulation)
+ - [Setting up cache refreshing with hardware ARM/AMD support](#setting-up-cache-refreshing-with-hardware-armamd-support)
+ - [When to rebuild the image](#when-to-rebuild-the-image)
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
@@ -48,15 +51,10 @@ manual refresh might be needed.
# Manually generating constraint files
```bash
-export CURRENT_PYTHON_MAJOR_MINOR_VERSIONS_AS_STRING="3.7 3.8 3.9 3.10"
-for python_version in $(echo "${CURRENT_PYTHON_MAJOR_MINOR_VERSIONS_AS_STRING}")
-do
- breeze build-image --upgrade-to-newer-dependencies --python ${python_version}
-done
-
-GENERATE_CONSTRAINTS_MODE="pypi-providers" ./scripts/ci/constraints/ci_generate_all_constraints.sh
-GENERATE_CONSTRAINTS_MODE="source-providers" ./scripts/ci/constraints/ci_generate_all_constraints.sh
-GENERATE_CONSTRAINTS_MODE="no-providers" ./scripts/ci/constraints/ci_generate_all_constraints.sh
+breeze build-image --build-multiple-images --upgrade-to-newer-dependencies
+breeze generate-constraints --generate-constraints-mode pypi-providers --run-in-parallel
+breeze generate-constraints --generate-constraints-mode source-providers --run-in-parallel
+breeze generate-constraints --generate-constraints-mode no-providers --run-in-parallel
AIRFLOW_SOURCES=$(pwd)
```
@@ -79,8 +77,10 @@ git push
Note that in order to refresh images you have to not only have `buildx` command installed for docker,
but you should also make sure that you have the buildkit builder configured and set. Since we also build
-multi-platform images (for both AMD and ARM), you need to have support for qemu installed with appropriate
-flags.
+multi-platform images (for both AMD and ARM), you need to have support for qemu or hardware ARM/AMD builders
+configured.
+
+## Setting up cache refreshing with emulation
According to the [official installation instructions](https://docs.docker.com/buildx/working-with-buildx/#build-multi-platform-images)
this can be achieved via:
@@ -91,18 +91,57 @@ docker run --privileged --rm tonistiigi/binfmt --install all
More information can be found [here](https://docs.docker.com/engine/reference/commandline/buildx_create/)
-The images can be rebuilt and refreshed after the constraints are pushed. Refreshing image for particular
-python version is a simple as running the [refresh_images.sh](refresh_images.sh) script with python version
-as parameter:
+However, emulation is very slow - more than 10x slower than hardware-backed builds.
+
+## Setting up cache refreshing with hardware ARM/AMD support
+
+If you plan to build a number of images, probably better solution is to set up a hardware remote builder
+for your ARM or AMD builds (depending which platform you build images on - the "other" platform should be
+remote.
+
+This can be achieved by settings build as described in
+[this guideline](https://www.docker.com/blog/speed-up-building-with-docker-buildx-and-graviton2-ec2/) and
+adding it to docker buildx `airflow_cache` builder.
+
+This usually can be done with those two commands:
+
+```bash
+docker buildx create --name airflow_cache # your local builder
+docker buildx create --name airflow_cache --append HOST:PORT # your remote builder
+```
+
+When everything is fine you should see both local and remote builder configured and reporting status:
```bash
-./dev/refresh_images.sh 3.10
+docker buildx ls
+
+ airflow_cache docker-container
+ airflow_cache0 unix:///var/run/docker.sock
+ airflow_cache1 tcp://127.0.0.1:2375
```
-If you have fast network and powerful computer, you can refresh the images in parallel running the
-[refresh_images.sh](refresh_images.sh) with all python versions. You might do it with `tmux` manually
-or with gnu parallel:
+## When to rebuild the image
+
+The images can be rebuilt and refreshed after the constraints are pushed. Refreshing image for all
+python version sis a simple as running the [refresh_images.sh](refresh_images.sh) script which will
+sequentially rebuild all the images. Usually building several images in parallel on one machine does not
+speed up the build significantly, that's why the images are build sequentially.
```bash
-parallel -j 4 --linebuffer --tagstring '{}' ./dev/refresh_images.sh ::: 3.7 3.8 3.9 3.10
+./dev/refresh_images.sh
+```
+
+You can also individually refresh the images by running commands like this:
+
+```bash
+rm -rf docker-context-files/*.whl
+rm -rf docker-context-files/*.tgz
+export ANSWER="yes"
+export CI="true"
+
+breeze build-image --python-version 3.7 \
+ --prepare-buildx-cache --platform linux/amd64,linux/arm64 --verbose
+
+breeze build-prod-image --python-version 3.7 \
+ --prepare-buildx-cache --platform linux/amd64,linux/arm64 --verbose
```
diff --git a/dev/breeze/src/airflow_breeze/breeze.py b/dev/breeze/src/airflow_breeze/breeze.py
index 0cfa4cc77b..b743cd47ab 100755
--- a/dev/breeze/src/airflow_breeze/breeze.py
+++ b/dev/breeze/src/airflow_breeze/breeze.py
@@ -16,26 +16,37 @@
# specific language governing permissions and limitations
# under the License.
import atexit
+import multiprocessing as mp
import os
import shutil
import subprocess
import sys
from dataclasses import dataclass
from pathlib import Path
-from typing import List, Optional, Tuple
+from typing import IO, List, Optional, Tuple
import rich
from airflow_breeze import NAME, VERSION
+from airflow_breeze.build_image.prod.build_prod_params import BuildProdParams
from airflow_breeze.shell.shell_params import ShellParams
+from airflow_breeze.utils.ci_group import ci_group
from airflow_breeze.utils.confirm import Answer, set_forced_answer, user_confirm
+from airflow_breeze.utils.constraints import run_generate_constraints, run_generate_constraints_in_parallel
+from airflow_breeze.utils.pulll_image import run_pull_image, run_pull_in_parallel
from airflow_breeze.utils.reinstall import ask_to_reinstall_breeze, reinstall_breeze, warn_non_editable
+from airflow_breeze.utils.run_tests import run_docker_compose_tests, verify_an_image
try:
# We handle ImportError so that click autocomplete works
import rich_click as click
+ try:
+ click.formatting.FORCED_WIDTH = os.get_terminal_size().columns - 2
+ except OSError:
+ pass
click.rich_click.SHOW_METAVARS_COLUMN = False
+ click.rich_click.SHOW_ARGUMENTS = False
click.rich_click.APPEND_METAVARS_HELP = True
click.rich_click.STYLE_ERRORS_SUGGESTION = "bright_blue italic"
click.rich_click.ERRORS_SUGGESTION = "\nTry running the '--help' flag for more information.\n"
@@ -123,25 +134,32 @@ try:
"--debian-version",
"--image-tag",
"--docker-cache",
- "--github-repository",
+ "--force-build",
+ ],
+ },
+ {
+ "name": "Building multiple images",
+ "options": [
+ "--build_multiple_images",
+ "--python-versions",
],
},
{
"name": "Advanced options (for power users)",
"options": [
"--install-providers-from-sources",
- "--additional-extras",
- "--additional-dev-apt-deps",
- "--additional-runtime-apt-deps",
"--additional-python-deps",
- "--additional-dev-apt-command",
+ "--runtime-apt-deps",
"--runtime-apt-command",
- "--additional-dev-apt-env",
+ "--additional-extras",
+ "--additional-runtime-apt-deps",
"--additional-runtime-apt-env",
"--additional-runtime-apt-command",
- "--dev-apt-command",
+ "--additional-dev-apt-deps",
+ "--additional-dev-apt-env",
+ "--additional-dev-apt-command",
"--dev-apt-deps",
- "--runtime-apt-deps",
+ "--dev-apt-command",
],
},
{
@@ -157,6 +175,36 @@ try:
],
},
],
+ "breeze pull-image": [
+ {
+ "name": "Pull image flags",
+ "options": [
+ "--image-tag",
+ "--python",
+ "--verify-image",
+ "--wait-for-image",
+ "--tag-as-latest",
+ ],
+ },
+ {
+ "name": "Parallel running",
+ "options": [
+ "--run-in-parallel",
+ "--parallelism",
+ "--python-versions",
+ ],
+ },
+ ],
+ "breeze verify-image": [
+ {
+ "name": "Verify image flags",
+ "options": [
+ "--image-name",
+ "--python",
+ "--image-tag",
+ ],
+ }
+ ],
"breeze build-prod-image": [
{
"name": "Basic usage",
@@ -167,33 +215,39 @@ try:
"--debian-version",
"--image-tag",
"--docker-cache",
- "--github-repository",
+ ],
+ },
+ {
+ "name": "Building multiple images",
+ "options": [
+ "--build_multiple_images",
+ "--python-versions",
],
},
{
"name": "Options for customizing images",
"options": [
"--install-providers-from-sources",
- "--extras",
+ "--additional-python-deps",
"--additional-extras",
- "--additional-dev-apt-deps",
"--additional-runtime-apt-deps",
- "--additional-python-deps",
- "--additional-dev-apt-command",
- "--runtime-apt-command",
- "--additional-dev-apt-env",
"--additional-runtime-apt-env",
"--additional-runtime-apt-command",
- "--dev-apt-command",
- "--dev-apt-deps",
+ "--additional-dev-apt-deps",
+ "--additional-dev-apt-env",
+ "--additional-dev-apt-command",
+ "--extras",
"--runtime-apt-deps",
+ "--runtime-apt-command",
+ "--dev-apt-deps",
+ "--dev-apt-command",
],
},
{
"name": "Customization options (for specific customization needs)",
"options": [
- "--install-from-docker-context-files",
"--cleanup-docker-context-files",
+ "--install-from-docker-context-files",
"--disable-mysql-client-installation",
"--disable-mssql-client-installation",
"--disable-postgres-client-installation",
@@ -206,16 +260,56 @@ try:
{
"name": "Preparing cache and push (for maintainers and CI)",
"options": [
- "--platform",
- "--prepare-buildx-cache",
- "--push-image",
- "--empty-image",
"--github-token",
"--github-username",
"--login-to-github-registry",
+ "--push-image",
+ "--prepare-buildx-cache",
+ "--platform",
+ "--empty-image",
],
},
],
+ "breeze pull-prod-image": [
+ {
+ "name": "Pull image flags",
+ "options": [
+ "--image-tag",
+ "--python",
+ "--verify-image",
+ "--wait-for-image",
+ "--tag-as-latest",
+ ],
+ },
+ {
+ "name": "Parallel running",
+ "options": [
+ "--run-in-parallel",
+ "--parallelism",
+ "--python-versions",
+ ],
+ },
+ ],
+ "breeze verify-prod-image": [
+ {
+ "name": "Verify image flags",
+ "options": [
+ "--image-name",
+ "--python",
+ "--image-tag",
+ ],
+ }
+ ],
+ "breeze docker-compose-tests": [
+ {
+ "name": "Docker-compose tests flag",
+ "options": [
+ "--image-name",
+ "--python",
+ "--image-tag",
+ ],
+ }
+ ],
"breeze static-checks": [
{
"name": "Pre-commit flags",
@@ -246,6 +340,14 @@ try:
],
},
],
+ "breeze cleanup": [
+ {
+ "name": "Cleanup flags",
+ "options": [
+ "--all",
+ ],
+ },
+ ],
"breeze setup-autocomplete": [
{
"name": "Setup autocomplete flags",
@@ -265,8 +367,50 @@ try:
],
},
],
+ "breeze prepare-airflow-package": [
+ {"name": "Package flags", "options": ["--package-format", "--version-suffix-for-pypi"]}
+ ],
+ "breeze prepare-provider-packages": [
+ {
+ "name": "Package flags",
+ "options": [
+ "--package-format",
+ "--version-suffix-for-pypi",
+ "--package-list-file",
+ ],
+ }
+ ],
+ "breeze prepare-provider-documentation": [
+ {"name": "Provider documentation preparation flags", "options": ["--skip-package-verification"]}
+ ],
+ "breeze generate-constraints": [
+ {
+ "name": "Generate constraints flags",
+ "options": [
+ "--image-tag",
+ "--python",
+ "--generate-constraints-mode",
+ ],
+ },
+ {
+ "name": "Parallel running",
+ "options": [
+ "--run-in-parallel",
+ "--parallelism",
+ "--python-versions",
+ ],
+ },
+ ],
+ "breeze self-upgrade": [
+ {
+ "name": "Self-upgrade flags",
+ "options": [
+ "--use-current-airflow-sources",
+ "--force",
+ ],
+ }
+ ],
}
-
click.rich_click.COMMAND_GROUPS = {
"breeze": [
{
@@ -275,16 +419,45 @@ try:
"shell",
"start-airflow",
"stop",
- "build-image",
- "build-prod-image",
"build-docs",
"static-checks",
],
},
+ {
+ "name": "Testing",
+ "commands": [
+ "docker-compose-tests",
+ ],
+ },
{
"name": "Configuration & maintenance",
"commands": ["cleanup", "self-upgrade", "setup-autocomplete", "config", "version"],
},
+ {
+ "name": "CI Image tools",
+ "commands": [
+ "build-image",
+ "pull-image",
+ "verify-image",
+ ],
+ },
+ {
+ "name": "Production Image tools",
+ "commands": [
+ "build-prod-image",
+ "pull-prod-image",
+ "verify-prod-image",
+ ],
+ },
+ {
+ "name": "Release management",
+ "commands": [
+ "prepare-provider-documentation",
+ "prepare-provider-packages",
+ "prepare-airflow-package",
+ "generate-constraints",
+ ],
+ },
]
}
@@ -292,24 +465,32 @@ try:
except ImportError:
import click # type: ignore[no-redef]
-from click import Context
+from click import Context, IntRange
-from airflow_breeze.build_image.ci.build_ci_image import build_image
+from airflow_breeze.build_image.ci.build_ci_image import build_ci_image, get_ci_image_build_params
from airflow_breeze.build_image.ci.build_ci_params import BuildCiParams
-from airflow_breeze.build_image.prod.build_prod_image import build_production_image
+from airflow_breeze.build_image.prod.build_prod_image import (
+ build_production_image,
+ get_prod_image_build_params,
+)
from airflow_breeze.global_constants import (
ALLOWED_BACKENDS,
ALLOWED_BUILD_CACHE,
ALLOWED_DEBIAN_VERSIONS,
ALLOWED_EXECUTORS,
+ ALLOWED_GENERATE_CONSTRAINTS_MODES,
ALLOWED_INSTALLATION_METHODS,
ALLOWED_INTEGRATIONS,
ALLOWED_MOUNT_OPTIONS,
ALLOWED_MSSQL_VERSIONS,
ALLOWED_MYSQL_VERSIONS,
+ ALLOWED_PACKAGE_FORMATS,
ALLOWED_PLATFORMS,
ALLOWED_POSTGRES_VERSIONS,
ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS,
+ DEFAULT_EXTRAS,
+ DEFAULT_PYTHON_MAJOR_MINOR_VERSION,
+ MOUNT_ALL,
MOUNT_SELECTED,
get_available_packages,
)
@@ -319,6 +500,7 @@ from airflow_breeze.utils.cache import (
check_if_cache_exists,
delete_cache,
read_from_cache_file,
+ synchronize_parameters_with_cache,
touch_cache_file,
write_to_cache_file,
)
@@ -340,13 +522,35 @@ from airflow_breeze.utils.path_utils import (
get_used_sources_setup_metadata_hash,
in_autocomplete,
)
-from airflow_breeze.utils.run_utils import check_pre_commit_installed, run_command
+from airflow_breeze.utils.run_utils import check_pre_commit_installed, filter_out_none, run_command
from airflow_breeze.utils.visuals import ASCIIART, ASCIIART_STYLE
find_airflow_sources_root_to_operate_on()
output_file_for_recording = os.environ.get('RECORD_BREEZE_OUTPUT_FILE')
+
+class BetterChoice(click.Choice):
+ """
+ Nicer formatted choice class for click. We have a lot of parameters sometimes, and formatting
+ them without spaces causes ugly artifacts as the words are broken. This one adds spaces so
+ that when the long list of choices does not wrap on words.
+ """
+
+ def get_metavar(self, param) -> str:
+ choices_str = " | ".join(self.choices)
+ # Use curly braces to indicate a required argument.
+ if param.required and param.param_type_name == "argument":
+ return f"{{{choices_str}}}"
+
+ if param.param_type_name == "argument" and param.nargs == -1:
+ # avoid double [[ for multiple args
+ return f"{choices_str}"
+
+ # Use square braces to indicate an option or optional argument.
+ return f"[{choices_str}]"
+
+
option_verbose = click.option(
"-v", "--verbose", is_flag=True, help="Print verbose information about performed steps.", envvar='VERBOSE'
)
@@ -362,16 +566,16 @@ option_dry_run = click.option(
option_answer = click.option(
"-a",
"--answer",
- type=click.Choice(['y', 'n', 'q', 'yes', 'no', 'quit']),
+ type=BetterChoice(['y', 'n', 'q', 'yes', 'no', 'quit']),
help="Force answer to questions.",
- envvar='FORCE_ANSWER_TO_QUESTIONS',
+ envvar='ANSWER',
)
option_python = click.option(
'-p',
'--python',
- type=click.Choice(ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS),
- help='Python version to use.',
+ type=BetterChoice(ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS),
+ help='Python major/minor version used in Airflow image for PROD/CI images.',
envvar='PYTHON_MAJOR_MINOR_VERSION',
)
@@ -379,32 +583,32 @@ option_backend = click.option(
'-b',
'--backend',
help="Database backend to use.",
- type=click.Choice(ALLOWED_BACKENDS),
+ type=BetterChoice(ALLOWED_BACKENDS),
)
option_integration = click.option(
'--integration',
help="Integration(s) to enable when running (can be more than one).",
- type=click.Choice(ALLOWED_INTEGRATIONS),
+ type=BetterChoice(ALLOWED_INTEGRATIONS),
multiple=True,
)
option_postgres_version = click.option(
- '-P', '--postgres-version', help="Version of Postgres.", type=click.Choice(ALLOWED_POSTGRES_VERSIONS)
+ '-P', '--postgres-version', help="Version of Postgres.", type=BetterChoice(ALLOWED_POSTGRES_VERSIONS)
)
option_mysql_version = click.option(
- '-M', '--mysql-version', help="Version of MySQL.", type=click.Choice(ALLOWED_MYSQL_VERSIONS)
+ '-M', '--mysql-version', help="Version of MySQL.", type=BetterChoice(ALLOWED_MYSQL_VERSIONS)
)
option_mssql_version = click.option(
- '-S', '--mssql-version', help="Version of MsSQL.", type=click.Choice(ALLOWED_MSSQL_VERSIONS)
+ '-S', '--mssql-version', help="Version of MsSQL.", type=BetterChoice(ALLOWED_MSSQL_VERSIONS)
)
option_executor = click.option(
'--executor',
help='Executor to use for a kubernetes cluster. Default is KubernetesExecutor.',
- type=click.Choice(ALLOWED_EXECUTORS),
+ type=BetterChoice(ALLOWED_EXECUTORS),
)
option_forward_credentials = click.option(
@@ -420,12 +624,15 @@ option_use_airflow_version = click.option(
option_mount_sources = click.option(
'--mount-sources',
- type=click.Choice(ALLOWED_MOUNT_OPTIONS),
+ type=BetterChoice(ALLOWED_MOUNT_OPTIONS),
default=ALLOWED_MOUNT_OPTIONS[0],
+ show_default=True,
help="Choose scope of local sources should be mounted (default = selected).",
)
-option_force_build = click.option('--force-build', help="Force image build before running.", is_flag=True)
+option_force_build = click.option(
+ '-f', '--force-build', help="Force image build no matter if it is " "determined as needed.", is_flag=True
+)
option_db_reset = click.option(
'-d',
@@ -435,11 +642,21 @@ option_db_reset = click.option(
envvar='DB_RESET',
)
+option_github_repository = click.option(
+ '-g',
+ '--github-repository',
+ help='GitHub repository used to pull, push run images.',
+ default="apache/airflow",
+ show_default=True,
+ envvar='GITHUB_REPOSITORY',
+)
+
@click.group(invoke_without_command=True, context_settings={'help_option_names': ['-h', '--help']})
@option_verbose
@option_dry_run
@option_python
+@option_github_repository
@option_backend
@option_postgres_version
@option_mysql_version
@@ -462,14 +679,9 @@ option_docker_cache = click.option(
'-c',
'--docker-cache',
help='Cache option for image used during the build.',
- type=click.Choice(ALLOWED_BUILD_CACHE),
-)
-
-option_github_repository = click.option(
- '-g',
- '--github-repository',
- help='GitHub repository used to pull, push images. Default: apache/airflow.',
- envvar='GITHUB_REPOSITORY',
+ default=ALLOWED_BUILD_CACHE[0],
+ show_default=True,
+ type=BetterChoice(ALLOWED_BUILD_CACHE),
)
option_login_to_github_registry = click.option(
@@ -499,27 +711,37 @@ option_github_image_id = click.option(
)
option_image_tag = click.option(
- '-t', '--image-tag', help='Set tag for the image (additionally to default Airflow convention).'
+ '-t',
+ '--image-tag',
+ help='Tag added to the default naming conventions of Airflow CI/PROD images.',
+ envvar='IMAGE_TAG',
+)
+
+option_image_name = click.option(
+ '-n', '--image-name', help='Name of the image to verify (overrides --python and --image-tag).'
)
option_platform = click.option(
'--platform',
help='Platform for Airflow image.',
envvar='PLATFORM',
- type=click.Choice(ALLOWED_PLATFORMS),
+ type=BetterChoice(ALLOWED_PLATFORMS),
)
option_debian_version = click.option(
'-d',
'--debian-version',
help='Debian version used for the image.',
- type=click.Choice(ALLOWED_DEBIAN_VERSIONS),
+ type=BetterChoice(ALLOWED_DEBIAN_VERSIONS),
+ default=ALLOWED_DEBIAN_VERSIONS[0],
+ show_default=True,
envvar='DEBIAN_VERSION',
)
option_upgrade_to_newer_dependencies = click.option(
"-u",
'--upgrade-to-newer-dependencies',
default="false",
+ show_default=True,
help='When other than "false", upgrade all PIP packages to latest.',
envvar='UPGRADE_TO_NEWER_DEPENDENCIES',
)
@@ -613,6 +835,26 @@ option_empty_image = click.option(
envvar='EMPTY_IMAGE',
)
+option_wait_for_image = click.option(
+ '--wait-for-image',
+ help='Wait until image is available.',
+ is_flag=True,
+ envvar='WAIT_FOR_IMAGE',
+)
+
+option_tag_as_latest = click.option(
+ '--tag-as-latest',
+ help='Tags the image as latest after pulling.',
+ is_flag=True,
+ envvar='TAG_AS_LATEST',
+)
+
+option_verify_image = click.option(
+ '--verify-image',
+ help='Verify image.',
+ is_flag=True,
+ envvar='VERIFY_IMAGE',
+)
option_install_providers_from_sources = click.option(
'--install-providers-from-sources',
@@ -637,11 +879,73 @@ option_load_default_connection = click.option(
envvar='LOAD_DEFAULT_CONNECTIONS',
)
+option_version_suffix_for_pypi = click.option(
+ '--version-suffix-for-pypi',
+ help='Version suffix used for PyPI packages (alpha, beta, rc1, etc.).',
+ default="",
+ envvar='VERSION_SUFFIX_FOR_PYPI',
+)
+
+option_package_format = click.option(
+ '--package-format',
+ type=BetterChoice(ALLOWED_PACKAGE_FORMATS),
+ help='Format of packages.',
+ default=ALLOWED_PACKAGE_FORMATS[0],
+ show_default=True,
+ envvar='PACKAGE_FORMAT',
+)
+
+option_python_versions = click.option(
+ '--python-versions',
+ help="Space separated list of python versions used for build with multiple versions.",
+ default=" ".join(ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS),
+ show_default=True,
+ envvar="PYTHON_VERSIONS",
+)
+
+option_run_in_parallel = click.option(
+ '--run-in-parallel',
+ help="Run the operation in parallel on all or selected subset of Python versions.",
+ is_flag=True,
+ envvar='RUN_IN_PARALLEL',
+)
+
+option_parallelism = click.option(
+ '--parallelism',
+ help="Maximum number of processes to use while running the operation in parallel.",
+ type=IntRange(1, mp.cpu_count() * 2 if not output_file_for_recording else 8),
+ default=mp.cpu_count() if not output_file_for_recording else 4,
+ envvar='PARALLELISM',
+ show_default=True,
+)
+
+option_build_multiple_images = click.option(
+ '--build_multiple_images',
+ help="Run the operation sequentially on all or selected subset of Python versions.",
+ is_flag=True,
+ envvar='BUILD_MULTIPLE_IMAGES',
+)
+
+option_with_ci_group = click.option(
+ '-g',
+ '--with-ci-group',
+ is_flag=True,
+ help="Uses CI group for the command to fold long logs in logical groups.",
+ envvar='WITH_CI_GROUP',
+)
+
+argument_packages = click.argument(
+ "packages",
+ nargs=-1,
+ required=False,
+ type=BetterChoice(get_available_packages(short_version=True)),
+)
+
@option_verbose
@main.command()
def version(verbose: bool):
- """Prints version of breeze.py."""
+ """Print information about version of apache-airflow-breeze."""
console.print(ASCIIART, style=ASCIIART_STYLE)
console.print(f"\n[bright_blue]Breeze version: {VERSION}[/]")
console.print(f"[bright_blue]Breeze installed from: {get_installation_airflow_sources()}[/]")
@@ -670,6 +974,7 @@ def version(verbose: bool):
@option_dry_run
@option_python
@option_backend
+@option_github_repository
@option_postgres_version
@option_mysql_version
@option_mssql_version
@@ -685,6 +990,7 @@ def shell(
verbose: bool,
dry_run: bool,
python: str,
+ github_repository: str,
backend: str,
integration: Tuple[str],
postgres_version: str,
@@ -700,13 +1006,14 @@ def shell(
):
"""Enter breeze.py environment. this is the default command use when no other is selected."""
set_forced_answer(answer)
- if verbose:
+ if verbose or dry_run:
console.print("\n[green]Welcome to breeze.py[/]\n")
console.print(f"\n[green]Root of Airflow Sources = {AIRFLOW_SOURCES_ROOT}[/]\n")
enter_shell(
verbose=verbose,
dry_run=dry_run,
python=python,
+ github_repository=github_repository,
backend=backend,
integration=integration,
postgres_version=postgres_version,
@@ -718,6 +1025,7 @@ def shell(
force_build=force_build,
db_reset=db_reset,
extra_args=extra_args,
+ answer=answer,
)
@@ -725,6 +1033,7 @@ def shell(
@main.command(name='start-airflow')
@option_dry_run
@option_python
+@option_github_repository
@option_backend
@option_postgres_version
@option_load_example_dags
@@ -743,6 +1052,7 @@ def start_airflow(
verbose: bool,
dry_run: bool,
python: str,
+ github_repository: str,
backend: str,
integration: Tuple[str],
postgres_version: str,
@@ -764,6 +1074,7 @@ def start_airflow(
verbose=verbose,
dry_run=dry_run,
python=python,
+ github_repository=github_repository,
backend=backend,
integration=integration,
postgres_version=postgres_version,
@@ -778,17 +1089,22 @@ def start_airflow(
db_reset=db_reset,
start_airflow=True,
extra_args=extra_args,
+ answer=answer,
)
@main.command(name='build-image')
+@option_github_repository
@option_verbose
@option_dry_run
+@option_with_ci_group
+@option_answer
@option_python
+@option_build_multiple_images
+@option_python_versions
@option_upgrade_to_newer_dependencies
@option_platform
@option_debian_version
-@option_github_repository
@option_github_token
@option_github_username
@option_login_to_github_registry
@@ -811,81 +1127,172 @@ def start_airflow(
@option_dev_apt_deps
@option_runtime_apt_command
@option_runtime_apt_deps
-@option_answer
-def build_ci_image(
+@option_force_build
+def build_image(
verbose: bool,
dry_run: bool,
- additional_extras: Optional[str],
- python: str,
- image_tag: Optional[str],
- additional_dev_apt_deps: Optional[str],
- additional_runtime_apt_deps: Optional[str],
- additional_python_deps: Optional[str],
- additional_dev_apt_command: Optional[str],
- additional_runtime_apt_command: Optional[str],
- additional_dev_apt_env: Optional[str],
- additional_runtime_apt_env: Optional[str],
- dev_apt_command: Optional[str],
- dev_apt_deps: Optional[str],
- install_providers_from_sources: bool,
- runtime_apt_command: Optional[str],
- runtime_apt_deps: Optional[str],
- github_repository: Optional[str],
- github_username: Optional[str],
- github_token: Optional[str],
- login_to_github_registry: bool,
- docker_cache: Optional[str],
- platform: Optional[str],
- debian_version: Optional[str],
- prepare_buildx_cache: bool,
- push_image: bool,
- empty_image: bool,
- answer: Optional[str],
- upgrade_to_newer_dependencies: str = "false",
+ with_ci_group: bool,
+ build_multiple_images: bool,
+ python_versions: str,
+ answer: str,
+ **kwargs,
):
- """Build CI image."""
+ """Build CI image. Include building multiple images for all python versions (sequentially)."""
+
+ def run_build(ci_image_params: BuildCiParams) -> None:
+ return_code, info = build_ci_image(
+ verbose=verbose, dry_run=dry_run, with_ci_group=with_ci_group, ci_image_params=ci_image_params
+ )
+ if return_code != 0:
+ console.print(f"[red]Error when building image! {info}")
+ sys.exit(return_code)
+
set_forced_answer(answer)
- if verbose:
+ parameters_passed = filter_out_none(**kwargs)
+ if build_multiple_images:
+ with ci_group(f"Building images sequentially {python_versions}", enabled=with_ci_group):
+ python_version_list = get_python_version_list(python_versions)
+ for python in python_version_list:
+ params = get_ci_image_build_params(parameters_passed)
+ params.python = python
+ params.answer = answer
+ run_build(ci_image_params=params)
+ else:
+ params = get_ci_image_build_params(parameters_passed)
+ synchronize_parameters_with_cache(params, parameters_passed)
+ run_build(ci_image_params=params)
+
+
+def get_python_version_list(python_versions: str) -> List[str]:
+ """
+ Retrieve and validate space-separated list of Python versions and return them in the form of list.
+ :param python_versions: space separated list of Python versions
+ :return: List of python versions
+ """
+ python_version_list = python_versions.split(" ")
+ errors = False
+ for python in python_version_list:
+ if python not in ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS:
+ console.print(f"[red]The Python version {python} passed in {python_versions} is wrong.[/]")
+ errors = True
+ if errors:
console.print(
- f"\n[bright_blue]Building image of airflow from {AIRFLOW_SOURCES_ROOT} "
- f"python version: {python}[/]\n"
+ f"\nSome of the Python versions passed are not in the "
+ f"list: {ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS}. Quitting.\n"
)
- build_image(
+ sys.exit(1)
+ return python_version_list
+
+
+@main.command(name='pull-image')
+@option_verbose
+@option_dry_run
+@option_python
+@option_github_repository
+@option_run_in_parallel
+@option_parallelism
+@option_python_versions
+@option_verify_image
+@option_wait_for_image
+@option_tag_as_latest
+@option_image_tag
+@click.argument('extra_pytest_args', nargs=-1, type=click.UNPROCESSED)
+def pull_image(
+ verbose: bool,
+ dry_run: bool,
+ python: str,
+ github_repository: str,
+ run_in_parallel: bool,
+ python_versions: str,
+ parallelism: int,
+ image_tag: Optional[str],
+ wait_for_image: bool,
+ tag_as_latest: bool,
+ verify_image: bool,
+ extra_pytest_args: Tuple,
+):
+ """Pull and optionally verify CI images - possibly in parallel for all Python versions."""
+ if run_in_parallel:
+ python_version_list = get_python_version_list(python_versions)
+ ci_image_params_list = [
+ BuildCiParams(image_tag=image_tag, python=python, github_repository=github_repository)
+ for python in python_version_list
+ ]
+ run_pull_in_parallel(
+ dry_run=dry_run,
+ parallelism=parallelism,
+ image_params_list=ci_image_params_list,
+ python_version_list=python_version_list,
+ verbose=verbose,
+ verify_image=verify_image,
+ wait_for_image=wait_for_image,
+ tag_as_latest=tag_as_latest,
+ extra_pytest_args=extra_pytest_args if extra_pytest_args is not None else (),
+ )
+ else:
+ image_params = BuildCiParams(image_tag=image_tag, python=python, github_repository=github_repository)
+ synchronize_parameters_with_cache(image_params, {"python": python})
+ return_code, info = run_pull_image(
+ image_params=image_params,
+ dry_run=dry_run,
+ verbose=verbose,
+ wait_for_image=wait_for_image,
+ tag_as_latest=tag_as_latest,
+ poll_time=10.0,
+ )
+ if return_code != 0:
+ console.print(f"[red]There was an error when pulling CI image: {info}[/]")
+ sys.exit(return_code)
+
+
+@main.command(
+ name='verify-image',
+ context_settings=dict(
+ ignore_unknown_options=True,
+ allow_extra_args=True,
+ ),
+)
+@option_verbose
+@option_dry_run
+@option_python
+@option_github_repository
+@option_image_tag
+@option_image_name
+@click.argument('extra_pytest_args', nargs=-1, type=click.UNPROCESSED)
+def verify_image(
+ verbose: bool,
+ dry_run: bool,
+ python: str,
+ github_repository: str,
+ image_name: str,
+ image_tag: str,
+ extra_pytest_args: Tuple,
+):
+ """Verify CI image."""
+ if image_name is None:
+ build_params = get_ci_image_build_params(
+ {"python": python, "image_tag": image_tag, "github_repository": github_repository}
+ )
+ image_name = build_params.airflow_image_name_with_tag
+ console.print(f"[bright_blue]Verifying CI image: {image_name}[/]")
+ return_code, info = verify_an_image(
+ image_name=image_name,
verbose=verbose,
dry_run=dry_run,
- additional_extras=additional_extras,
- python=python,
- image_tag=image_tag,
- additional_dev_apt_deps=additional_dev_apt_deps,
- additional_runtime_apt_deps=additional_runtime_apt_deps,
- additional_python_deps=additional_python_deps,
- additional_runtime_apt_command=additional_runtime_apt_command,
- additional_dev_apt_command=additional_dev_apt_command,
- additional_dev_apt_env=additional_dev_apt_env,
- additional_runtime_apt_env=additional_runtime_apt_env,
- install_providers_from_sources=install_providers_from_sources,
- dev_apt_command=dev_apt_command,
- dev_apt_deps=dev_apt_deps,
- runtime_apt_command=runtime_apt_command,
- runtime_apt_deps=runtime_apt_deps,
- github_repository=github_repository,
- github_token=github_token,
- login_to_github_registry=login_to_github_registry,
- github_username=github_username,
- docker_cache=docker_cache,
- platform=platform,
- debian_version=debian_version,
- prepare_buildx_cache=prepare_buildx_cache,
- push_image=push_image,
- empty_image=empty_image,
- upgrade_to_newer_dependencies=upgrade_to_newer_dependencies,
+ image_type='CI',
+ extra_pytest_args=extra_pytest_args,
)
+ sys.exit(return_code)
@option_verbose
@option_dry_run
+@option_answer
+@option_with_ci_group
@main.command(name='build-prod-image')
@option_python
+@option_build_multiple_images
+@option_python_versions
@option_upgrade_to_newer_dependencies
@option_platform
@option_debian_version
@@ -901,7 +1308,7 @@ def build_ci_image(
@click.option(
'--installation-method',
help="Install Airflow from: sources or PyPI.",
- type=click.Choice(ALLOWED_INSTALLATION_METHODS),
+ type=BetterChoice(ALLOWED_INSTALLATION_METHODS),
)
@option_install_providers_from_sources
@click.option(
@@ -914,7 +1321,9 @@ def build_ci_image(
help='Clean up docker context files before running build.',
is_flag=True,
)
-@click.option('--extras', help="Extras to install by default.")
+@click.option(
+ '--extras', default=",".join(DEFAULT_EXTRAS), show_default=True, help="Extras to install by default."
+)
@click.option('--disable-mysql-client-installation', help="Do not install MySQL client.", is_flag=True)
@click.option('--disable-mssql-client-installation', help="Do not install MsSQl client.", is_flag=True)
@click.option('--disable-postgres-client-installation', help="Do not install Postgres client.", is_flag=True)
@@ -941,97 +1350,183 @@ def build_ci_image(
@option_dev_apt_deps
@option_runtime_apt_command
@option_runtime_apt_deps
-@option_answer
def build_prod_image(
verbose: bool,
dry_run: bool,
- cleanup_docker_context_files: bool,
- disable_mysql_client_installation: bool,
- disable_mssql_client_installation: bool,
- disable_postgres_client_installation: bool,
- disable_airflow_repo_cache: bool,
- disable_pypi: bool,
- install_airflow_reference: Optional[str],
- install_airflow_version: Optional[str],
- docker_cache: str,
- additional_extras: Optional[str],
- python: str,
- image_tag: Optional[str],
- additional_dev_apt_deps: Optional[str],
- additional_runtime_apt_deps: Optional[str],
- additional_python_deps: Optional[str],
- additional_dev_apt_command: Optional[str],
- additional_runtime_apt_command: Optional[str],
- additional_dev_apt_env: Optional[str],
- additional_runtime_apt_env: Optional[str],
- dev_apt_command: Optional[str],
- dev_apt_deps: Optional[str],
- runtime_apt_command: Optional[str],
- runtime_apt_deps: Optional[str],
- github_repository: Optional[str],
- github_token: Optional[str],
- github_username: Optional[str],
- login_to_github_registry: bool,
- platform: Optional[str],
- debian_version: Optional[str],
- prepare_buildx_cache: bool,
- push_image: bool,
- empty_image: bool,
- install_providers_from_sources: bool,
- extras: Optional[str],
- installation_method: Optional[str],
- install_from_docker_context_files: bool,
+ with_ci_group: bool,
+ build_multiple_images: bool,
+ python_versions: str,
answer: Optional[str],
- upgrade_to_newer_dependencies: str = "false",
+ **kwargs,
):
- """Build Production image."""
+ """
+ Build Production image. Include building multiple images for all or selected Python versions sequentially.
+ """
+
+ def run_build(prod_image_params: BuildProdParams) -> None:
+ return_code, info = build_production_image(
+ verbose=verbose, dry_run=dry_run, with_ci_group=with_ci_group, prod_image_params=prod_image_params
+ )
+ if return_code != 0:
+ console.print(f"[red]Error when building image! {info}")
+ sys.exit(return_code)
+
set_forced_answer(answer)
- if verbose:
- console.print("\n[bright_blue]Building image[/]\n")
- if prepare_buildx_cache:
- docker_cache = "pulled"
- cleanup_docker_context_files = True
- build_production_image(
- verbose,
- dry_run,
- cleanup_docker_context_files=cleanup_docker_context_files,
- disable_mysql_client_installation=disable_mysql_client_installation,
- disable_mssql_client_installation=disable_mssql_client_installation,
- disable_postgres_client_installation=disable_postgres_client_installation,
- disable_airflow_repo_cache=disable_airflow_repo_cache,
- disable_pypi=disable_pypi,
- install_airflow_reference=install_airflow_reference,
- install_airflow_version=install_airflow_version,
- docker_cache=docker_cache,
- additional_extras=additional_extras,
- python=python,
- additional_dev_apt_deps=additional_dev_apt_deps,
- additional_runtime_apt_deps=additional_runtime_apt_deps,
- additional_python_deps=additional_python_deps,
- additional_runtime_apt_command=additional_runtime_apt_command,
- additional_dev_apt_command=additional_dev_apt_command,
- additional_dev_apt_env=additional_dev_apt_env,
- additional_runtime_apt_env=additional_runtime_apt_env,
- dev_apt_command=dev_apt_command,
- dev_apt_deps=dev_apt_deps,
- runtime_apt_command=runtime_apt_command,
- runtime_apt_deps=runtime_apt_deps,
- github_repository=github_repository,
- github_token=github_token,
- github_username=github_username,
- login_to_github_registry=login_to_github_registry,
- platform=platform,
- debian_version=debian_version,
- upgrade_to_newer_dependencies=upgrade_to_newer_dependencies,
- prepare_buildx_cache=prepare_buildx_cache,
- push_image=push_image,
- empty_image=empty_image,
- install_providers_from_sources=install_providers_from_sources,
- extras=extras,
- installation_method=installation_method,
- install_docker_context_files=install_from_docker_context_files,
- image_tag=image_tag,
+ parameters_passed = filter_out_none(**kwargs)
+ if build_multiple_images:
+ with ci_group(f"Building images sequentially {python_versions}", enabled=with_ci_group):
+ python_version_list = get_python_version_list(python_versions)
+ for python in python_version_list:
+ params = get_prod_image_build_params(parameters_passed)
+ params.python = python
+ params.answer = answer
+ run_build(prod_image_params=params)
+ else:
+ params = get_prod_image_build_params(parameters_passed)
+ synchronize_parameters_with_cache(params, parameters_passed)
+ run_build(prod_image_params=params)
+
+
+@main.command(name='pull-prod-image')
+@option_verbose
+@option_dry_run
+@option_python
+@option_github_repository
+@option_run_in_parallel
+@option_parallelism
+@option_python_versions
+@option_image_tag
+@option_wait_for_image
+@option_tag_as_latest
+@option_verify_image
+@click.argument('extra_pytest_args', nargs=-1, type=click.UNPROCESSED)
+def pull_prod_image(
+ verbose: bool,
+ dry_run: bool,
+ python: str,
+ github_repository: str,
+ run_in_parallel: bool,
+ parallelism: int,
+ python_versions: str,
+ image_tag: Optional[str],
+ wait_for_image: bool,
+ tag_as_latest: bool,
+ verify_image: bool,
+ extra_pytest_args: Tuple,
+):
+ """Pull and optionally verify Production images - possibly in parallel for all Python versions."""
+ if run_in_parallel:
+ python_version_list = get_python_version_list(python_versions)
+ prod_image_params_list = [
+ BuildProdParams(image_tag=image_tag, python=python, github_repository=github_repository)
+ for python in python_version_list
+ ]
+ run_pull_in_parallel(
+ dry_run=dry_run,
+ parallelism=parallelism,
+ image_params_list=prod_image_params_list,
+ python_version_list=python_version_list,
+ verbose=verbose,
+ verify_image=verify_image,
+ wait_for_image=wait_for_image,
+ tag_as_latest=tag_as_latest,
+ extra_pytest_args=extra_pytest_args if extra_pytest_args is not None else (),
+ )
+ else:
+ image_params = BuildProdParams(
+ image_tag=image_tag, python=python, github_repository=github_repository
+ )
+ synchronize_parameters_with_cache(image_params, {"python": python})
+ return_code, info = run_pull_image(
+ image_params=image_params,
+ dry_run=dry_run,
+ verbose=verbose,
+ wait_for_image=wait_for_image,
+ tag_as_latest=tag_as_latest,
+ poll_time=10.0,
+ )
+ if return_code != 0:
+ console.print(f"[red]There was an error when pulling PROD image: {info}[/]")
+ sys.exit(return_code)
+
+
+@main.command(
+ name='verify-prod-image',
+ context_settings=dict(
+ ignore_unknown_options=True,
+ allow_extra_args=True,
+ ),
+)
+@option_verbose
+@option_dry_run
+@option_python
+@option_github_repository
+@option_image_tag
+@option_image_name
+@click.argument('extra_pytest_args', nargs=-1, type=click.UNPROCESSED)
+def verify_prod_image(
+ verbose: bool,
+ dry_run: bool,
+ python: str,
+ github_repository: str,
+ image_name: str,
+ image_tag: str,
+ extra_pytest_args: Tuple,
+):
+ """Verify Production image."""
+ if image_name is None:
+ build_params = get_prod_image_build_params(
+ {"python": python, "image_tag": image_tag, "github_repository": github_repository}
+ )
+ image_name = build_params.airflow_image_name_with_tag
+ console.print(f"[bright_blue]Verifying PROD image: {image_name}[/]")
+ return_code, info = verify_an_image(
+ image_name=image_name,
+ verbose=verbose,
+ dry_run=dry_run,
+ image_type='PROD',
+ extra_pytest_args=extra_pytest_args,
+ )
+ sys.exit(return_code)
+
+
+@main.command(
+ name='docker-compose-tests',
+ context_settings=dict(
+ ignore_unknown_options=True,
+ allow_extra_args=True,
+ ),
+)
+@option_verbose
+@option_dry_run
+@option_python
+@option_github_repository
+@option_image_tag
+@option_image_name
+@click.argument('extra_pytest_args', nargs=-1, type=click.UNPROCESSED)
+def docker_compose_tests(
+ verbose: bool,
+ dry_run: bool,
+ python: str,
+ github_repository: str,
+ image_name: str,
+ image_tag: str,
+ extra_pytest_args: Tuple,
+):
+ """Run docker-compose tests."""
+ if image_name is None:
+ build_params = get_prod_image_build_params(
+ {"python": python, "image_tag": image_tag, "github_repository": github_repository}
+ )
+ image_name = build_params.airflow_image_name_with_tag
+ console.print(f"[bright_blue]Running docker-compose with PROD image: {image_name}[/]")
+ return_code, info = run_docker_compose_tests(
+ image_name=image_name,
+ verbose=verbose,
+ dry_run=dry_run,
+ extra_pytest_args=extra_pytest_args,
)
+ sys.exit(return_code)
BREEZE_COMMENT = "Added by Updated Airflow Breeze autocomplete setup"
@@ -1237,20 +1732,26 @@ class DocParams:
@main.command(name='build-docs')
@option_verbose
@option_dry_run
+@option_github_repository
@click.option('-d', '--docs-only', help="Only build documentation.", is_flag=True)
@click.option('-s', '--spellcheck-only', help="Only run spell checking.", is_flag=True)
@click.option(
'-p',
'--package-filter',
help="List of packages to consider.",
- type=click.Choice(get_available_packages()),
+ type=BetterChoice(get_available_packages()),
multiple=True,
)
def build_docs(
- verbose: bool, dry_run: bool, docs_only: bool, spellcheck_only: bool, package_filter: Tuple[str]
+ verbose: bool,
+ dry_run: bool,
+ github_repository: str,
+ docs_only: bool,
+ spellcheck_only: bool,
+ package_filter: Tuple[str],
):
"""Build documentation in the container."""
- params = BuildCiParams()
+ params = BuildCiParams(github_repository=github_repository, python=DEFAULT_PYTHON_MAJOR_MINOR_VERSION)
ci_image_name = params.airflow_image_name
check_docker_resources(verbose, ci_image_name)
doc_builder = DocParams(
@@ -1259,13 +1760,19 @@ def build_docs(
spellcheck_only=spellcheck_only,
)
extra_docker_flags = get_extra_docker_flags(MOUNT_SELECTED)
- cmd = []
- cmd.extend(["docker", "run"])
- cmd.extend(extra_docker_flags)
- cmd.extend(["-t", "-e", "GITHUB_ACTIONS="])
- cmd.extend(["--entrypoint", "/usr/local/bin/dumb-init", "--pull", "never"])
- cmd.extend([ci_image_name, "--", "/opt/airflow/scripts/in_container/run_docs_build.sh"])
- cmd.extend(doc_builder.args_doc_builder)
+ cmd = [
+ "docker",
+ "run",
+ "-t",
+ *extra_docker_flags,
+ "-e",
+ "GITHUB_ACTIONS=",
+ "--pull",
+ "never",
+ ci_image_name,
+ "/opt/airflow/scripts/in_container/run_docs_build.sh",
+ *doc_builder.args_doc_builder,
+ ]
run_command(cmd, verbose=verbose, dry_run=dry_run, text=True)
@@ -1281,7 +1788,7 @@ def build_docs(
'-t',
'--type',
help="Type(s) of the static checks to run (multiple can be added).",
- type=click.Choice(PRE_COMMIT_LIST),
+ type=BetterChoice(PRE_COMMIT_LIST),
multiple=True,
)
@click.option('-a', '--all-files', help="Run checks on all files.", is_flag=True)
@@ -1292,10 +1799,12 @@ def build_docs(
@click.option('-c', '--last-commit', help="Run checks for all files in last commit.", is_flag=True)
@option_verbose
@option_dry_run
+@option_github_repository
@click.argument('precommit_args', nargs=-1, type=click.UNPROCESSED)
def static_checks(
verbose: bool,
dry_run: bool,
+ github_repository: str,
all_files: bool,
show_diff_on_failure: bool,
last_commit: bool,
@@ -1315,10 +1824,12 @@ def static_checks(
command_to_execute.extend(["--from-ref", "HEAD^", "--to-ref", "HEAD"])
if files:
command_to_execute.append("--files")
- if verbose:
+ if verbose or dry_run:
command_to_execute.append("--verbose")
if precommit_args:
command_to_execute.extend(precommit_args)
+ env = os.environ.copy()
+ env['GITHUB_REPOSITORY'] = github_repository
run_command(
command_to_execute,
verbose=verbose,
@@ -1326,6 +1837,7 @@ def static_checks(
check=False,
no_output_dump_on_exception=True,
text=True,
+ env=env,
)
@@ -1342,7 +1854,7 @@ def stop(verbose: bool, dry_run: bool, preserve_volumes: bool):
command_to_execute = ['docker-compose', 'down', "--remove-orphans"]
if not preserve_volumes:
command_to_execute.append("--volumes")
- shell_params = ShellParams()
+ shell_params = ShellParams(verbose=verbose)
env_variables = construct_env_variables_docker_compose_command(shell_params)
run_command(command_to_execute, verbose=verbose, dry_run=dry_run, env=env_variables)
@@ -1354,6 +1866,7 @@ def stop(verbose: bool, dry_run: bool, preserve_volumes: bool):
help='Force upgrade without asking question to the user.',
)
@click.option(
+ '-a',
'--use-current-airflow-sources',
is_flag=True,
help='Use current workdir Airflow sources for upgrade'
@@ -1382,28 +1895,289 @@ def self_upgrade(force: bool, use_current_airflow_sources: bool):
sys.exit(1)
+@main.command(
+ name='prepare-airflow-package',
+ help="Prepare sdist/whl package of Airflow.",
+)
+@option_verbose
+@option_dry_run
+@option_github_repository
+@option_with_ci_group
+@option_package_format
+@option_version_suffix_for_pypi
+def prepare_airflow_packages(
+ verbose: bool,
+ dry_run: bool,
+ github_repository: str,
+ with_ci_group: bool,
+ package_format: str,
+ version_suffix_for_pypi: str,
+):
+ shell_params = ShellParams(
+ verbose=verbose, github_repository=github_repository, python=DEFAULT_PYTHON_MAJOR_MINOR_VERSION
+ )
+ env_variables = construct_env_variables_docker_compose_command(shell_params)
+ env_variables['INSTALL_PROVIDERS_FROM_SOURCES'] = "false"
+ extra_docker_flags = get_extra_docker_flags(MOUNT_ALL)
+ with ci_group("Prepare Airflow package", enabled=with_ci_group):
+ run_command(
+ [
+ "docker",
+ "run",
+ "-t",
+ *extra_docker_flags,
+ "-e",
+ "SKIP_ENVIRONMENT_INITIALIZATION=true",
+ "-e",
+ f"PACKAGE_FORMAT={package_format}",
+ "-e",
+ f"VERSION_SUFFIX_FOR_PYPI={version_suffix_for_pypi}",
+ "--pull",
+ "never",
+ shell_params.airflow_image_name_with_tag,
+ "/opt/airflow/scripts/in_container/run_prepare_airflow_packages.sh",
+ ],
+ verbose=verbose,
+ dry_run=dry_run,
+ env=env_variables,
+ )
+
+
+@main.command(
+ name='prepare-provider-documentation',
+ help="Prepare CHANGELOG, README and COMMITS information for providers.",
+)
+@option_verbose
+@option_dry_run
+@option_github_repository
+@option_with_ci_group
+@option_answer
+@click.option(
+ '--skip-package-verification',
+ help="Skip Provider package verification.",
+ is_flag=True,
+)
+@argument_packages
+def prepare_provider_documentation(
+ verbose: bool,
+ dry_run: bool,
+ github_repository: str,
+ with_ci_group: bool,
+ answer: Optional[str],
+ skip_package_verification: bool,
+ packages: List[str],
+):
+ set_forced_answer(answer)
+ shell_params = ShellParams(
+ verbose=verbose,
+ mount_sources=MOUNT_ALL,
+ skip_package_verification=skip_package_verification,
+ github_repository=github_repository,
+ python=DEFAULT_PYTHON_MAJOR_MINOR_VERSION,
+ answer=answer,
+ )
+ env_variables = construct_env_variables_docker_compose_command(shell_params)
+ extra_docker_flags = get_extra_docker_flags(shell_params.mount_sources)
+ if answer and answer.lower() in ['y', 'yes']:
+ term_flag = "-t"
+ else:
+ term_flag = "-it"
+ cmd_to_run = [
+ "docker",
+ "run",
+ term_flag,
+ *extra_docker_flags,
+ "-e",
+ "SKIP_ENVIRONMENT_INITIALIZATION=true",
+ "--pull",
+ "never",
+ shell_params.airflow_image_name_with_tag,
+ "/opt/airflow/scripts/in_container/run_prepare_provider_documentation.sh",
+ ]
+ if packages:
+ cmd_to_run.extend(packages)
+ with ci_group("Prepare provider documentation", enabled=with_ci_group):
+ run_command(cmd_to_run, verbose=verbose, dry_run=dry_run, env=env_variables)
+
+
+@main.command(
+ name='prepare-provider-packages',
+ help="Prepare sdist/whl packages of Airflow Providers.",
+)
+@option_verbose
+@option_dry_run
+@option_github_repository
+@option_with_ci_group
+@option_package_format
+@option_version_suffix_for_pypi
+@click.option(
+ '--package-list-file',
+ type=click.File('rt'),
+ help='Read list of packages from text file (one package per line)',
+)
+@argument_packages
+def prepare_provider_packages(
+ verbose: bool,
+ dry_run: bool,
+ github_repository: str,
+ with_ci_group: bool,
+ package_format: str,
+ version_suffix_for_pypi: str,
+ package_list_file: IO,
+ packages: Tuple[str, ...],
+):
+ packages_list = list(packages)
+ if package_list_file:
+ packages_list.extend([package.strip() for package in package_list_file.readlines()])
+ shell_params = ShellParams(
+ verbose=verbose,
+ mount_sources=MOUNT_ALL,
+ github_repository=github_repository,
+ python=DEFAULT_PYTHON_MAJOR_MINOR_VERSION,
+ )
+ env_variables = construct_env_variables_docker_compose_command(shell_params)
+ extra_docker_flags = get_extra_docker_flags(shell_params.mount_sources)
+ cmd_to_run = [
+ "docker",
+ "run",
+ "-t",
+ *extra_docker_flags,
+ "-e",
+ "SKIP_ENVIRONMENT_INITIALIZATION=true",
+ "-e",
+ f"PACKAGE_FORMAT={package_format}",
+ "-e",
+ f"VERSION_SUFFIX_FOR_PYPI={version_suffix_for_pypi}",
+ "--pull",
+ "never",
+ shell_params.airflow_image_name_with_tag,
+ "/opt/airflow/scripts/in_container/run_prepare_provider_packages.sh",
+ ]
+ cmd_to_run.extend(packages_list)
+ with ci_group("Prepare provider packages", enabled=with_ci_group):
+ run_command(cmd_to_run, verbose=verbose, dry_run=dry_run, env=env_variables)
+
+
+@main.command(
+ name='generate-constraints',
+ help="Generates pinned constraint files with all extras from setup.py in parallel.",
+)
+@click.option(
+ '--generate-constraints-mode',
+ type=BetterChoice(ALLOWED_GENERATE_CONSTRAINTS_MODES),
+ default=ALLOWED_GENERATE_CONSTRAINTS_MODES[0],
+ show_default=True,
+ help='Mode of generating constraints',
+)
+@option_verbose
+@option_dry_run
+@option_python
+@option_github_repository
+@option_run_in_parallel
+@option_parallelism
+@option_python_versions
+@option_image_tag
+@option_answer
+@option_with_ci_group
+def generate_constraints(
+ verbose: bool,
+ dry_run: bool,
+ python: str,
+ github_repository: str,
+ run_in_parallel: bool,
+ parallelism: int,
+ python_versions: str,
+ image_tag: str,
+ answer: Optional[str],
+ generate_constraints_mode: str,
+ with_ci_group: bool,
+):
+ set_forced_answer(answer)
+ if run_in_parallel:
+ given_answer = user_confirm(
+ f"Did you build all CI images {python_versions} with --upgrade-to-newer-dependencies "
+ f"true flag set?",
+ timeout=None,
+ )
+ else:
+ given_answer = user_confirm(
+ f"Did you build CI image {python} with --upgrade-to-newer-dependencies true flag set?",
+ timeout=None,
+ )
+ if given_answer != Answer.YES:
+ if run_in_parallel:
+ console.print("\n[yellow]Use this command to build the images:[/]\n")
+ console.print(
+ f" breeze build-image --run-in-parallel --python-versions '{python_versions}' "
+ f"--upgrade-to-newer-dependencies true\n"
+ )
+ else:
+ shell_params = ShellParams(
+ image_tag=image_tag, python=python, github_repository=github_repository, answer=answer
+ )
+ synchronize_parameters_with_cache(shell_params, {"python": python})
+ console.print("\n[yellow]Use this command to build the image:[/]\n")
+ console.print(
+ f" breeze build-image --python '{shell_params.python}' "
+ f"--upgrade-to-newer-dependencies true\n"
+ )
+ sys.exit(1)
+ if run_in_parallel:
+ python_version_list = get_python_version_list(python_versions)
+ shell_params_list = [
+ ShellParams(
+ image_tag=image_tag, python=python, github_repository=github_repository, answer=answer
+ )
+ for python in python_version_list
+ ]
+ with ci_group(f"Generating constraints with {generate_constraints_mode}", enabled=with_ci_group):
+ run_generate_constraints_in_parallel(
+ shell_params_list=shell_params_list,
+ parallelism=parallelism,
+ dry_run=dry_run,
+ verbose=verbose,
+ python_version_list=python_version_list,
+ generate_constraints_mode=generate_constraints_mode,
+ )
+ else:
+ with ci_group(f"Generating constraints with {generate_constraints_mode}", enabled=with_ci_group):
+ shell_params = ShellParams(
+ image_tag=image_tag, python=python, github_repository=github_repository, answer=answer
+ )
+ synchronize_parameters_with_cache(shell_params, {"python": python})
+ return_code, info = run_generate_constraints(
+ shell_params=shell_params,
+ dry_run=dry_run,
+ verbose=verbose,
+ generate_constraints_mode=generate_constraints_mode,
+ )
+ if return_code != 0:
+ console.print(f"[red]There was an error when generating constraints: {info}[/]")
+ sys.exit(return_code)
+
+
@main.command(
name="cleanup",
- help=" the cache of parameters, docker cache and optionally - currently downloaded images.",
+ help="Cleans the cache of parameters, docker cache and optionally - currently downloaded images.",
)
@click.option(
- '--also-remove-current-images',
+ '--all',
is_flag=True,
help='Also remove currently downloaded Breeze images.',
)
@option_verbose
@option_answer
@option_dry_run
-def cleanup(verbose: bool, dry_run: bool, also_remove_current_images: bool, answer: Optional[str]):
+def cleanup(verbose: bool, dry_run: bool, github_repository: str, all: bool, answer: Optional[str]):
set_forced_answer(answer)
- if also_remove_current_images:
+ if all:
console.print(
"\n[bright_yellow]Removing cache of parameters, clean up docker cache "
"and remove locally downloaded images[/]"
)
else:
console.print("\n[bright_yellow]Removing cache of parameters, and cleans up docker cache[/]")
- if also_remove_current_images:
+ if all:
docker_images_command_to_execute = [
'docker',
'images',
@@ -1473,7 +2247,7 @@ def enable_recording_of_help_output(path: str, title: Optional[str], width: Opti
atexit.register(save_ouput_as_svg)
click.rich_click.MAX_WIDTH = width_int
- click.formatting.FORCED_WIDTH = width_int
+ click.formatting.FORCED_WIDTH = width_int - 2
click.rich_click.COLOR_SYSTEM = "standard"
# monkeypatch rich_click console to record help (rich_click does not allow passing extra args to console)
click.rich_click.Console = RecordingConsole
diff --git a/dev/breeze/src/airflow_breeze/build_image/ci/build_ci_image.py b/dev/breeze/src/airflow_breeze/build_image/ci/build_ci_image.py
index a904c5d89d..124e0517b0 100644
--- a/dev/breeze/src/airflow_breeze/build_image/ci/build_ci_image.py
+++ b/dev/breeze/src/airflow_breeze/build_image/ci/build_ci_image.py
@@ -14,69 +14,98 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-"""Command to build CI image."""
-from typing import Dict
-from airflow_breeze.build_image.ci.build_ci_params import BuildCiParams
+import multiprocessing as mp
+import sys
+from typing import Dict, List, Tuple
+
+from airflow_breeze.build_image.ci.build_ci_params import (
+ OPTIONAL_CI_IMAGE_ARGS,
+ REQUIRED_CI_IMAGE_ARGS,
+ BuildCiParams,
+)
from airflow_breeze.utils.cache import synchronize_parameters_with_cache, touch_cache_file
+from airflow_breeze.utils.ci_group import ci_group
+from airflow_breeze.utils.confirm import Answer, user_confirm
from airflow_breeze.utils.console import console
from airflow_breeze.utils.docker_command_utils import (
construct_docker_build_command,
construct_empty_docker_build_command,
tag_and_push_image,
)
-from airflow_breeze.utils.md5_build_check import calculate_md5_checksum_for_files
+from airflow_breeze.utils.md5_build_check import (
+ calculate_md5_checksum_for_files,
+ md5sum_check_if_build_is_needed,
+)
+from airflow_breeze.utils.parallel import check_async_run_results
from airflow_breeze.utils.path_utils import AIRFLOW_SOURCES_ROOT, BUILD_CACHE_DIR
from airflow_breeze.utils.registry import login_to_docker_registry
-from airflow_breeze.utils.run_utils import filter_out_none, fix_group_permissions, run_command
-
-REQUIRED_CI_IMAGE_ARGS = [
- "python_base_image",
- "airflow_version",
- "airflow_branch",
- "airflow_extras",
- "airflow_pre_cached_pip_packages",
- "additional_airflow_extras",
- "additional_python_deps",
- "additional_dev_apt_command",
- "additional_dev_apt_deps",
- "additional_dev_apt_env",
- "additional_runtime_apt_command",
- "additional_runtime_apt_deps",
- "additional_runtime_apt_env",
- "upgrade_to_newer_dependencies",
- "constraints_github_repository",
- "airflow_constraints_reference",
- "airflow_constraints",
- "airflow_image_repository",
- "airflow_image_date_created",
- "build_id",
-]
-
-OPTIONAL_CI_IMAGE_ARGS = [
- "dev_apt_command",
- "dev_apt_deps",
- "runtime_apt_command",
- "runtime_apt_deps",
-]
+from airflow_breeze.utils.run_utils import (
+ fix_group_permissions,
+ get_return_code,
+ instruct_build_image,
+ is_repo_rebased,
+ run_command,
+)
-def get_ci_image_build_params(parameters_passed: Dict) -> BuildCiParams:
+def should_we_run_the_build(build_ci_params: BuildCiParams) -> bool:
"""
- Converts parameters received as dict into BuildCiParams. In case cacheable
- parameters are missing, it reads the last used value for that parameter
- from the cache and if it is not found, it uses default value for that parameter.
+ Check if we should run the build based on what files have been modified since last build and answer from
+ the user.
- This method updates cached based on parameters passed via Dict.
+ * If build is needed, the user is asked for confirmation
+ * If the branch is not rebased it warns the user to rebase (to make sure latest remote cache is useful)
+ * Builds Image/Skips/Quits depending on the answer
- :param parameters_passed: parameters to use when constructing BuildCiParams
+ :param build_ci_params: parameters for the build
"""
- ci_image_params = BuildCiParams(**parameters_passed)
- synchronize_parameters_with_cache(ci_image_params, parameters_passed)
- return ci_image_params
+ # We import those locally so that click autocomplete works
+ from inputimeout import TimeoutOccurred
+
+ if not md5sum_check_if_build_is_needed(md5sum_cache_dir=build_ci_params.md5sum_cache_dir):
+ return False
+ try:
+ answer = user_confirm(message="Do you want to build image?", timeout=5, default_answer=Answer.NO)
+ if answer == answer.YES:
+ if is_repo_rebased(build_ci_params.github_repository, build_ci_params.airflow_branch):
+ return True
+ else:
+ console.print(
+ "\n[bright_yellow]This might take a lot of time, w"
+ "e think you should rebase first.[/]\n"
+ )
+ answer = user_confirm(
+ "But if you really, really want - you can do it", timeout=5, default_answer=Answer.NO
+ )
+ if answer == Answer.YES:
+ return True
+ else:
+ console.print(
+ "[bright_blue]Please rebase your code before continuing.[/]\n"
+ "Check this link to know more "
+ "https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#id15\n"
+ )
+ console.print('[red]Exiting the process[/]\n')
+ sys.exit(1)
+ elif answer == Answer.NO:
+ instruct_build_image(build_ci_params.python)
+ return False
+ else: # users_status == Answer.QUIT:
+ console.print('\n[bright_yellow]Quitting the process[/]\n')
+ sys.exit()
+ except TimeoutOccurred:
+ console.print('\nTimeout. Considering your response as No\n')
+ instruct_build_image(build_ci_params.python)
+ return False
+ except Exception as e:
+ console.print(f'\nTerminating the process on {e}')
+ sys.exit(1)
-def build_image(verbose: bool, dry_run: bool, **kwargs) -> None:
+def build_ci_image(
+ verbose: bool, dry_run: bool, with_ci_group: bool, ci_image_params: BuildCiParams
+) -> Tuple[int, str]:
"""
Builds CI image:
@@ -92,44 +121,98 @@ def build_image(verbose: bool, dry_run: bool, **kwargs) -> None:
:param verbose: print commands when running
:param dry_run: do not execute "write" commands - just print what would happen
- :param kwargs: arguments passed from the command
+ :param with_ci_group: whether to wrap the build in CI logging group
+ :param ci_image_params: CI image parameters
"""
fix_group_permissions()
- parameters_passed = filter_out_none(**kwargs)
- ci_image_params = get_ci_image_build_params(parameters_passed)
- ci_image_params.print_info()
- run_command(
- ["docker", "rmi", "--no-prune", "--force", ci_image_params.airflow_image_name],
- verbose=verbose,
- dry_run=dry_run,
- cwd=AIRFLOW_SOURCES_ROOT,
- text=True,
- check=False,
- )
- if ci_image_params.prepare_buildx_cache:
- login_to_docker_registry(ci_image_params)
- cmd = construct_docker_build_command(
- image_params=ci_image_params,
- verbose=verbose,
- required_args=REQUIRED_CI_IMAGE_ARGS,
- optional_args=OPTIONAL_CI_IMAGE_ARGS,
- production_image=False,
- )
- if ci_image_params.empty_image:
- console.print(f"\n[blue]Building empty CI Image for Python {ci_image_params.python}\n")
- cmd = construct_empty_docker_build_command(image_params=ci_image_params)
+ if verbose or dry_run:
+ console.print(
+ f"\n[bright_blue]Building CI image of airflow from {AIRFLOW_SOURCES_ROOT} "
+ f"python version: {ci_image_params.python}[/]\n"
+ )
+ with ci_group(
+ f"Build CI image for Python {ci_image_params.python} " f"with tag: {ci_image_params.image_tag}",
+ enabled=with_ci_group,
+ ):
+ ci_image_params.print_info()
+ if not ci_image_params.force_build and not ci_image_params.upgrade_to_newer_dependencies:
+ if not should_we_run_the_build(build_ci_params=ci_image_params):
+ return 0, f"Image build: {ci_image_params.python}"
run_command(
- cmd, input="FROM scratch\n", verbose=verbose, dry_run=dry_run, cwd=AIRFLOW_SOURCES_ROOT, text=True
+ ["docker", "rmi", "--no-prune", "--force", ci_image_params.airflow_image_name],
+ verbose=verbose,
+ dry_run=dry_run,
+ cwd=AIRFLOW_SOURCES_ROOT,
+ text=True,
+ check=False,
+ )
+ if ci_image_params.prepare_buildx_cache:
+ login_to_docker_registry(ci_image_params, dry_run=dry_run)
+ cmd = construct_docker_build_command(
+ image_params=ci_image_params,
+ verbose=verbose,
+ required_args=REQUIRED_CI_IMAGE_ARGS,
+ optional_args=OPTIONAL_CI_IMAGE_ARGS,
+ production_image=False,
)
- else:
- console.print(f"\n[blue]Building CI Image for Python {ci_image_params.python}\n")
- run_command(cmd, verbose=verbose, dry_run=dry_run, cwd=AIRFLOW_SOURCES_ROOT, text=True)
- if not dry_run:
- ci_image_cache_dir = BUILD_CACHE_DIR / ci_image_params.airflow_branch
- ci_image_cache_dir.mkdir(parents=True, exist_ok=True)
- touch_cache_file(f"built_{ci_image_params.python}", root_dir=ci_image_cache_dir)
- calculate_md5_checksum_for_files(ci_image_params.md5sum_cache_dir, update=True)
- else:
- console.print("[blue]Not updating build cache because we are in `dry_run` mode.[/]")
- if ci_image_params.push_image:
- tag_and_push_image(image_params=ci_image_params, dry_run=dry_run, verbose=verbose)
+ if ci_image_params.empty_image:
+ console.print(f"\n[blue]Building empty CI Image for Python {ci_image_params.python}\n")
+ cmd = construct_empty_docker_build_command(image_params=ci_image_params)
+ run_command(
+ cmd,
+ input="FROM scratch\n",
+ verbose=verbose,
+ dry_run=dry_run,
+ cwd=AIRFLOW_SOURCES_ROOT,
+ text=True,
+ )
+ else:
+ console.print(f"\n[blue]Building CI Image for Python {ci_image_params.python}\n")
+ process = run_command(
+ cmd, verbose=verbose, dry_run=dry_run, cwd=AIRFLOW_SOURCES_ROOT, text=True, check=False
+ )
+ if not dry_run:
+ if process and process.returncode == 0:
+ ci_image_cache_dir = BUILD_CACHE_DIR / ci_image_params.airflow_branch
+ ci_image_cache_dir.mkdir(parents=True, exist_ok=True)
+ touch_cache_file(f"built_{ci_image_params.python}", root_dir=ci_image_cache_dir)
+ calculate_md5_checksum_for_files(ci_image_params.md5sum_cache_dir, update=True)
+ else:
+ console.print("[red]Error when building image![/]")
+ return (
+ get_return_code(process=process, dry_run=dry_run),
+ f"Image build: {ci_image_params.python}",
+ )
+ else:
+ console.print("[blue]Not updating build cache because we are in `dry_run` mode.[/]")
+ if ci_image_params.push_image:
+ return tag_and_push_image(image_params=ci_image_params, dry_run=dry_run, verbose=verbose)
+ return get_return_code(process=process, dry_run=dry_run), f"Image build: {ci_image_params.python}"
+
+
+def get_ci_image_build_params(parameters_passed: Dict) -> BuildCiParams:
+ """
+ Converts parameters received as dict into BuildCiParams. In case cacheable
+ parameters are missing, it reads the last used value for that parameter
+ from the cache and if it is not found, it uses default value for that parameter.
+
+ This method updates cached based on parameters passed via Dict.
+
+ :param parameters_passed: parameters to use when constructing BuildCiParams
+ """
+ ci_image_params = BuildCiParams(**parameters_passed)
+ synchronize_parameters_with_cache(ci_image_params, parameters_passed)
+ return ci_image_params
+
+
+def build_ci_image_in_parallel(
+ verbose: bool, dry_run: bool, parallelism: int, python_version_list: List[str], **kwargs
+):
+ """Run CI image builds in parallel."""
+ console.print(
+ f"\n[bright_blue]Running with parallelism = {parallelism} for the images: {python_version_list}:"
+ )
+ pool = mp.Pool(parallelism)
+ results = [pool.apply_async(build_ci_image, args=(verbose, dry_run, False), kwds=kwargs)]
+ check_async_run_results(results)
+ pool.close()
diff --git a/dev/breeze/src/airflow_breeze/build_image/ci/build_ci_params.py b/dev/breeze/src/airflow_breeze/build_image/ci/build_ci_params.py
index 519c13fefe..f726b000f7 100644
--- a/dev/breeze/src/airflow_breeze/build_image/ci/build_ci_params.py
+++ b/dev/breeze/src/airflow_breeze/build_image/ci/build_ci_params.py
@@ -39,7 +39,7 @@ class BuildCiParams:
build_id: int = 0
docker_cache: str = "pulled"
airflow_extras: str = "devel_ci"
- install_providers_from_sources: bool = False
+ install_providers_from_sources: bool = True
additional_airflow_extras: str = ""
additional_python_deps: str = ""
github_repository: str = "apache/airflow"
@@ -53,7 +53,7 @@ class BuildCiParams:
github_username: str = ""
dev_apt_command: str = ""
dev_apt_deps: str = ""
- image_tag: str = ""
+ image_tag: Optional[str] = None
github_token: str = ""
additional_dev_apt_command: str = ""
additional_dev_apt_deps: str = ""
@@ -68,7 +68,13 @@ class BuildCiParams:
prepare_buildx_cache: bool = False
push_image: bool = False
empty_image: bool = False
+ force_build: bool = False
skip_rebuild_check: bool = False
+ answer: Optional[str] = None
+
+ @property
+ def the_image_type(self) -> str:
+ return 'CI'
@property
def airflow_base_image_name(self):
@@ -85,7 +91,7 @@ class BuildCiParams:
def airflow_image_name_with_tag(self):
"""Construct CI image link"""
image = f'{self.airflow_base_image_name}/{self.airflow_branch}/ci/python{self.python}'
- return image if not self.image_tag else image + f":{self.image_tag}"
+ return image if self.image_tag is None else image + f":{self.image_tag}"
@property
def airflow_image_repository(self):
@@ -144,3 +150,33 @@ class BuildCiParams:
def print_info(self):
console.print(f"CI Image: {self.airflow_version} Python: {self.python}.")
+
+
+REQUIRED_CI_IMAGE_ARGS = [
+ "python_base_image",
+ "airflow_version",
+ "airflow_branch",
+ "airflow_extras",
+ "airflow_pre_cached_pip_packages",
+ "additional_airflow_extras",
+ "additional_python_deps",
+ "additional_dev_apt_command",
+ "additional_dev_apt_deps",
+ "additional_dev_apt_env",
+ "additional_runtime_apt_command",
+ "additional_runtime_apt_deps",
+ "additional_runtime_apt_env",
+ "upgrade_to_newer_dependencies",
+ "constraints_github_repository",
+ "airflow_constraints_reference",
+ "airflow_constraints",
+ "airflow_image_repository",
+ "airflow_image_date_created",
+ "build_id",
+]
+OPTIONAL_CI_IMAGE_ARGS = [
+ "dev_apt_command",
+ "dev_apt_deps",
+ "runtime_apt_command",
+ "runtime_apt_deps",
+]
diff --git a/dev/breeze/src/airflow_breeze/build_image/prod/build_prod_image.py b/dev/breeze/src/airflow_breeze/build_image/prod/build_prod_image.py
index 4fc511f35c..7883677eb5 100644
--- a/dev/breeze/src/airflow_breeze/build_image/prod/build_prod_image.py
+++ b/dev/breeze/src/airflow_breeze/build_image/prod/build_prod_image.py
@@ -17,10 +17,11 @@
"""Command to build PROD image."""
import contextlib
import sys
-from typing import Dict
+from typing import Dict, Tuple
from airflow_breeze.build_image.prod.build_prod_params import BuildProdParams
from airflow_breeze.utils.cache import synchronize_parameters_with_cache
+from airflow_breeze.utils.ci_group import ci_group
from airflow_breeze.utils.console import console
from airflow_breeze.utils.docker_command_utils import (
construct_docker_build_command,
@@ -29,7 +30,7 @@ from airflow_breeze.utils.docker_command_utils import (
)
from airflow_breeze.utils.path_utils import AIRFLOW_SOURCES_ROOT, DOCKER_CONTEXT_DIR
from airflow_breeze.utils.registry import login_to_docker_registry
-from airflow_breeze.utils.run_utils import filter_out_none, fix_group_permissions, run_command
+from airflow_breeze.utils.run_utils import fix_group_permissions, get_return_code, run_command
REQUIRED_PROD_IMAGE_ARGS = [
"python_base_image",
@@ -41,6 +42,7 @@ REQUIRED_PROD_IMAGE_ARGS = [
"airflow_extras",
"airflow_pre_cached_pip_packages",
"docker_context_files",
+ "extras",
"additional_airflow_extras",
"additional_python_deps",
"additional_dev_apt_command",
@@ -69,10 +71,14 @@ OPTIONAL_PROD_IMAGE_ARGS = [
]
-def clean_docker_context_files():
+def clean_docker_context_files(verbose: bool, dry_run: bool):
"""
Cleans up docker context files folder - leaving only README.md there.
"""
+ if verbose or dry_run:
+ console.print("[bright_blue]Cleaning docker-context-files[/]")
+ if dry_run:
+ return
with contextlib.suppress(FileNotFoundError):
context_files_to_delete = DOCKER_CONTEXT_DIR.glob('**/*')
for file_to_delete in context_files_to_delete:
@@ -129,7 +135,9 @@ def get_prod_image_build_params(parameters_passed: Dict) -> BuildProdParams:
return prod_image_params
-def build_production_image(verbose: bool, dry_run: bool, **kwargs):
+def build_production_image(
+ verbose: bool, dry_run: bool, with_ci_group: bool, prod_image_params: BuildProdParams
+) -> Tuple[int, str]:
"""
Builds PROD image:
@@ -147,40 +155,56 @@ def build_production_image(verbose: bool, dry_run: bool, **kwargs):
:param verbose: print commands when running
:param dry_run: do not execute "write" commands - just print what would happen
- :param kwargs: arguments passed from the command
+ :param with_ci_group: whether to wrap the build in CI logging group
+ :param prod_image_params: PROD image parameters
"""
fix_group_permissions()
- parameters_passed = filter_out_none(**kwargs)
- prod_image_params = get_prod_image_build_params(parameters_passed)
- prod_image_params.print_info()
- if prod_image_params.cleanup_docker_context_files:
- clean_docker_context_files()
- check_docker_context_files(prod_image_params.install_docker_context_files)
- if prod_image_params.prepare_buildx_cache:
- login_to_docker_registry(prod_image_params)
- run_command(
- ["docker", "rmi", "--no-prune", "--force", prod_image_params.airflow_image_name],
- verbose=verbose,
- dry_run=dry_run,
- cwd=AIRFLOW_SOURCES_ROOT,
- text=True,
- check=False,
- )
- console.print(f"\n[blue]Building PROD Image for Python {prod_image_params.python}\n")
- if prod_image_params.empty_image:
- console.print(f"\n[blue]Building empty PROD Image for Python {prod_image_params.python}\n")
- cmd = construct_empty_docker_build_command(image_params=prod_image_params)
- run_command(
- cmd, input="FROM scratch\n", verbose=verbose, dry_run=dry_run, cwd=AIRFLOW_SOURCES_ROOT, text=True
+ if verbose or dry_run:
+ console.print(
+ f"\n[bright_blue]Building PROD image of airflow from {AIRFLOW_SOURCES_ROOT} "
+ f"python version: {prod_image_params.python}[/]\n"
)
- else:
- cmd = construct_docker_build_command(
- image_params=prod_image_params,
+ with ci_group(
+ f"Build Production image for Python {prod_image_params.python} "
+ f"with tag: {prod_image_params.image_tag}",
+ enabled=with_ci_group,
+ ):
+ prod_image_params.print_info()
+ if prod_image_params.cleanup_docker_context_files:
+ clean_docker_context_files(verbose=verbose, dry_run=dry_run)
+ check_docker_context_files(prod_image_params.install_from_docker_context_files)
+ if prod_image_params.prepare_buildx_cache:
+ login_to_docker_registry(prod_image_params, dry_run=dry_run)
+ run_command(
+ ["docker", "rmi", "--no-prune", "--force", prod_image_params.airflow_image_name],
verbose=verbose,
- required_args=REQUIRED_PROD_IMAGE_ARGS,
- optional_args=OPTIONAL_PROD_IMAGE_ARGS,
- production_image=True,
+ dry_run=dry_run,
+ cwd=AIRFLOW_SOURCES_ROOT,
+ text=True,
+ check=False,
)
- run_command(cmd, verbose=verbose, dry_run=dry_run, cwd=AIRFLOW_SOURCES_ROOT, text=True)
- if prod_image_params.push_image:
- tag_and_push_image(image_params=prod_image_params, dry_run=dry_run, verbose=verbose)
+ console.print(f"\n[blue]Building PROD Image for Python {prod_image_params.python}\n")
+ if prod_image_params.empty_image:
+ console.print(f"\n[blue]Building empty PROD Image for Python {prod_image_params.python}\n")
+ cmd = construct_empty_docker_build_command(image_params=prod_image_params)
+ process = run_command(
+ cmd,
+ input="FROM scratch\n",
+ verbose=verbose,
+ dry_run=dry_run,
+ cwd=AIRFLOW_SOURCES_ROOT,
+ text=True,
+ )
+ else:
+ cmd = construct_docker_build_command(
+ image_params=prod_image_params,
+ verbose=verbose,
+ required_args=REQUIRED_PROD_IMAGE_ARGS,
+ optional_args=OPTIONAL_PROD_IMAGE_ARGS,
+ production_image=True,
+ )
+ process = run_command(cmd, verbose=verbose, dry_run=dry_run, cwd=AIRFLOW_SOURCES_ROOT, text=True)
+ if process and process.returncode == 0:
+ if prod_image_params.push_image:
+ return tag_and_push_image(image_params=prod_image_params, dry_run=dry_run, verbose=verbose)
+ return get_return_code(process=process, dry_run=dry_run), f"Image build: {prod_image_params.python}"
diff --git a/dev/breeze/src/airflow_breeze/build_image/prod/build_prod_params.py b/dev/breeze/src/airflow_breeze/build_image/prod/build_prod_params.py
index 2a398addc6..3eaa7192d3 100644
--- a/dev/breeze/src/airflow_breeze/build_image/prod/build_prod_params.py
+++ b/dev/breeze/src/airflow_breeze/build_image/prod/build_prod_params.py
@@ -20,7 +20,7 @@ import re
import sys
from dataclasses import dataclass
from datetime import datetime
-from typing import List
+from typing import List, Optional
from airflow_breeze.branch_defaults import AIRFLOW_BRANCH, DEFAULT_AIRFLOW_CONSTRAINTS_BRANCH
from airflow_breeze.global_constants import (
@@ -45,9 +45,9 @@ class BuildProdParams:
disable_mysql_client_installation: bool = False
disable_mssql_client_installation: bool = False
disable_postgres_client_installation: bool = False
- install_docker_context_files: bool = False
+ install_from_docker_context_files: bool = False
disable_airflow_repo_cache: bool = False
- install_providers_from_sources: bool = True
+ install_providers_from_sources: bool = False
cleanup_docker_context_files: bool = False
prepare_buildx_cache: bool = False
push_image: bool = False
@@ -75,7 +75,8 @@ class BuildProdParams:
additional_runtime_apt_deps: str = ""
additional_runtime_apt_env: str = ""
additional_python_deps: str = ""
- image_tag: str = ""
+ image_tag: Optional[str] = None
+ extras: str = ""
additional_airflow_extras: str = ""
github_token: str = ""
login_to_github_registry: str = "false"
@@ -85,6 +86,7 @@ class BuildProdParams:
airflow_constraints_location: str = ""
installation_method: str = "."
debian_version: str = "bullseye"
+ answer: Optional[str] = None
@property
def airflow_branch(self) -> str:
@@ -103,8 +105,7 @@ class BuildProdParams:
@property
def the_image_type(self) -> str:
- the_image_type = 'PROD'
- return the_image_type
+ return 'PROD'
@property
def args_for_remote_install(self) -> List:
@@ -276,13 +277,6 @@ class BuildProdParams:
install_postgres = 'false'
return install_postgres
- @property
- def install_from_docker_context_files(self) -> str:
- install_from_docker_context_files = 'false'
- if self.install_docker_context_files:
- install_from_docker_context_files = 'true'
- return install_from_docker_context_files
-
@property
def airflow_extras(self):
return get_airflow_extras()
@@ -295,4 +289,4 @@ class BuildProdParams:
def airflow_image_name_with_tag(self):
"""Construct PROD image link"""
image = f'{self.airflow_base_image_name}/{self.airflow_branch}/prod/python{self.python}'
- return image if not self.image_tag else image + f":{self.image_tag}"
+ return image if self.image_tag is None else image + f":{self.image_tag}"
diff --git a/dev/breeze/src/airflow_breeze/global_constants.py b/dev/breeze/src/airflow_breeze/global_constants.py
index e177df0d44..9a6a8acfb1 100644
--- a/dev/breeze/src/airflow_breeze/global_constants.py
+++ b/dev/breeze/src/airflow_breeze/global_constants.py
@@ -26,16 +26,16 @@ from airflow_breeze.utils.path_utils import AIRFLOW_SOURCES_ROOT
# FORCE_PULL_IMAGES = False
# CHECK_IF_BASE_PYTHON_IMAGE_UPDATED = False
FORCE_BUILD_IMAGES = False
-FORCE_ANSWER_TO_QUESTION = ""
+ANSWER = ""
SKIP_CHECK_REMOTE_IMAGE = False
# PUSH_PYTHON_BASE_IMAGE = False
-DEFAULT_PYTHON_MAJOR_MINOR_VERSION = '3.7'
-DEFAULT_BACKEND = 'sqlite'
# Checked before putting in build cache
ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS = ['3.7', '3.8', '3.9', '3.10']
+DEFAULT_PYTHON_MAJOR_MINOR_VERSION = ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS[0]
ALLOWED_BACKENDS = ['sqlite', 'mysql', 'postgres', 'mssql']
+DEFAULT_BACKEND = ALLOWED_BACKENDS[0]
ALLOWED_INTEGRATIONS = [
'cassandra',
'kerberos',
@@ -79,9 +79,9 @@ ALLOWED_TEST_TYPES = [
'Helm',
'Quarantined',
]
-ALLOWED_PACKAGE_FORMATS = ['both', 'sdist', 'wheel']
+ALLOWED_PACKAGE_FORMATS = ['wheel', 'sdist', 'both']
ALLOWED_INSTALLATION_METHODS = ['.', 'apache-airflow']
-ALLOWED_DEBIAN_VERSIONS = ['buster', 'bullseye']
+ALLOWED_DEBIAN_VERSIONS = ['bullseye', 'buster']
ALLOWED_BUILD_CACHE = ["pulled", "local", "disabled"]
ALLOWED_PLATFORMS = ["linux/amd64", "linux/arm64", "linux/amd64,linux/arm64"]
@@ -119,11 +119,16 @@ EXCLUDE_DOCS_PACKAGE_FOLDER = [
]
-def get_available_packages() -> List[str]:
+def get_available_packages(short_version=False) -> List[str]:
docs_path_content = (AIRFLOW_SOURCES_ROOT / 'docs').glob('*/')
available_packages = [x.name for x in docs_path_content if x.is_dir()]
package_list = list(set(available_packages) - set(EXCLUDE_DOCS_PACKAGE_FOLDER))
package_list.sort()
+ if short_version:
+ prefix_len = len("apache-airflow-providers-")
+ package_list = [
+ package[prefix_len:].replace("-", ".") for package in package_list if len(package) > prefix_len
+ ]
return package_list
@@ -228,18 +233,12 @@ DEFAULT_HELM_VERSIONS = CURRENT_HELM_VERSIONS[0]
DEFAULT_EXECUTOR = CURRENT_EXECUTORS[0]
# Initialize image build variables - Have to check if this has to go to ci dataclass
-SKIP_TWINE_CHECK = ""
USE_AIRFLOW_VERSION = ""
GITHUB_ACTIONS = ""
ISSUE_ID = ""
NUM_RUNS = ""
-# Initialize package variables
-PACKAGE_FORMAT = "wheel"
-VERSION_SUFFIX_FOR_SVN = ""
-VERSION_SUFFIX_FOR_PYPI = ""
-
MIN_DOCKER_VERSION = "20.10.0"
MIN_DOCKER_COMPOSE_VERSION = "1.29.0"
@@ -247,3 +246,34 @@ AIRFLOW_SOURCES_FROM = "."
AIRFLOW_SOURCES_TO = "/opt/airflow"
AIRFLOW_SOURCES_WWW_FROM = "./airflow/www"
AIRFLOW_SOURCES_WWW_TO = "/opt/airflow/airflow/www"
+
+DEFAULT_EXTRAS = [
+ # BEGINNING OF EXTRAS LIST UPDATED BY PRE COMMIT
+ "amazon",
+ "async",
+ "celery",
+ "cncf.kubernetes",
+ "dask",
+ "docker",
+ "elasticsearch",
+ "ftp",
+ "google",
+ "google_auth",
+ "grpc",
+ "hashicorp",
+ "http",
+ "ldap",
+ "microsoft.azure",
+ "mysql",
+ "odbc",
+ "pandas",
+ "postgres",
+ "redis",
+ "sendgrid",
+ "sftp",
+ "slack",
+ "ssh",
+ "statsd",
+ "virtualenv",
+ # END OF EXTRAS LIST UPDATED BY PRE COMMIT
+]
diff --git a/dev/breeze/src/airflow_breeze/shell/enter_shell.py b/dev/breeze/src/airflow_breeze/shell/enter_shell.py
index 183a999a52..0a426b3713 100644
--- a/dev/breeze/src/airflow_breeze/shell/enter_shell.py
+++ b/dev/breeze/src/airflow_breeze/shell/enter_shell.py
@@ -20,14 +20,13 @@ from pathlib import Path
from typing import Dict
from airflow_breeze import global_constants
-from airflow_breeze.build_image.ci.build_ci_image import build_image
+from airflow_breeze.build_image.ci.build_ci_image import build_ci_image, get_ci_image_build_params
from airflow_breeze.shell.shell_params import ShellParams
from airflow_breeze.utils.cache import (
check_cached_value_is_allowed,
read_from_cache_file,
write_to_cache_file,
)
-from airflow_breeze.utils.confirm import Answer, user_confirm
from airflow_breeze.utils.console import console
from airflow_breeze.utils.docker_command_utils import (
SOURCE_OF_DEFAULT_VALUES_FOR_VARIABLES,
@@ -38,121 +37,11 @@ from airflow_breeze.utils.docker_command_utils import (
check_docker_version,
construct_env_variables_docker_compose_command,
)
-from airflow_breeze.utils.md5_build_check import md5sum_check_if_build_is_needed
from airflow_breeze.utils.path_utils import BUILD_CACHE_DIR
-from airflow_breeze.utils.run_utils import filter_out_none, instruct_build_image, is_repo_rebased, run_command
+from airflow_breeze.utils.run_utils import filter_out_none, run_command
from airflow_breeze.utils.visuals import ASCIIART, ASCIIART_STYLE, CHEATSHEET, CHEATSHEET_STYLE
-def build_image_if_needed_steps(verbose: bool, dry_run: bool, shell_params: ShellParams) -> None:
- """
- Check if image build is needed based on what files have been modified since last build.
-
- * If build is needed, the user is asked for confirmation
- * If the branch is not rebased it warns the user to rebase (to make sure latest remote cache is useful)
- * Builds Image/Skips/Quits depending on the answer
-
- :param verbose: print commands when running
- :param dry_run: do not execute "write" commands - just print what would happen
- :param shell_params: parameters for the build
- """
- # We import those locally so that click autocomplete works
- from inputimeout import TimeoutOccurred
-
- build_needed = md5sum_check_if_build_is_needed(shell_params.md5sum_cache_dir, shell_params.the_image_type)
- if not build_needed:
- return
- try:
- answer = user_confirm(message="Do you want to build image?", timeout=5, default_answer=Answer.NO)
- if answer == answer.YES:
- if is_repo_rebased(shell_params.github_repository, shell_params.airflow_branch):
- build_image(
- verbose,
- dry_run=dry_run,
- python=shell_params.python,
- upgrade_to_newer_dependencies="false",
- )
- else:
- console.print(
- "\n[bright_yellow]This might take a lot of time, w"
- "e think you should rebase first.[/]\n"
- )
- answer = user_confirm(
- "But if you really, really want - you can do it", timeout=5, default_answer=Answer.NO
- )
- if answer == Answer.YES:
- build_image(
- verbose=verbose,
- dry_run=dry_run,
- python=shell_params.python,
- upgrade_to_newer_dependencies="false",
- )
- else:
- console.print(
- "[bright_blue]Please rebase your code before continuing.[/]\n"
- "Check this link to know more "
- "https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#id15\n"
- )
- console.print('[red]Exiting the process[/]\n')
- sys.exit(1)
- elif answer == Answer.NO:
- instruct_build_image(shell_params.python)
- else: # users_status == Answer.QUIT:
- console.print('\n[bright_yellow]Quitting the process[/]\n')
- sys.exit()
- except TimeoutOccurred:
- console.print('\nTimeout. Considering your response as No\n')
- instruct_build_image(shell_params.python)
- except Exception as e:
- console.print(f'\nTerminating the process on {e}')
- sys.exit(1)
-
-
-def run_shell_with_build_image_checks(verbose: bool, dry_run: bool, shell_params: ShellParams):
- """
- Executes shell command built from params passed, checking if build is not needed.
- * checks if there are enough resources to run shell
- * checks if image was built at least once (if not - forces the build)
- * if not forces, checks if build is needed and asks the user if so
- * builds the image if needed
- * prints information about the build
- * constructs docker compose command to enter shell
- * executes it
-
- :param verbose: print commands when running
- :param dry_run: do not execute "write" commands - just print what would happen
- :param shell_params: parameters of the execution
- """
- check_docker_resources(verbose, shell_params.airflow_image_name)
- build_ci_image_check_cache = Path(
- BUILD_CACHE_DIR, shell_params.airflow_branch, f".built_{shell_params.python}"
- )
- if build_ci_image_check_cache.exists():
- console.print(f'[bright_blue]{shell_params.the_image_type} image already built locally.[/]')
- else:
- console.print(
- f'[bright_yellow]{shell_params.the_image_type} image not built locally. ' f'Forcing build.[/]'
- )
- shell_params.force_build = True
-
- if not shell_params.force_build:
- build_image_if_needed_steps(verbose, dry_run, shell_params)
- else:
- build_image(
- verbose,
- dry_run=dry_run,
- python=shell_params.python,
- upgrade_to_newer_dependencies="false",
- )
- shell_params.print_badge_info()
- cmd = ['docker-compose', 'run', '--service-ports', "-e", "BREEZE", '--rm', 'airflow']
- cmd_added = shell_params.command_passed
- env_variables = construct_env_variables_docker_compose_command(shell_params)
- if cmd_added is not None:
- cmd.extend(['-c', cmd_added])
- run_command(cmd, verbose=verbose, dry_run=dry_run, env=env_variables, text=True)
-
-
def synchronize_cached_params(parameters_passed_by_the_user: Dict[str, str]) -> Dict[str, str]:
"""
Synchronizes cached params with arguments passed via dictionary.
@@ -206,3 +95,44 @@ def enter_shell(**kwargs):
console.print(CHEATSHEET, style=CHEATSHEET_STYLE)
enter_shell_params = ShellParams(**filter_out_none(**updated_kwargs))
run_shell_with_build_image_checks(verbose, dry_run, enter_shell_params)
+
+
+def run_shell_with_build_image_checks(verbose: bool, dry_run: bool, shell_params: ShellParams):
+ """
+ Executes a shell command built from params passed, checking if build is not needed.
+ * checks if there are enough resources to run shell
+ * checks if image was built at least once (if not - forces the build)
+ * if not forces, checks if build is needed and asks the user if so
+ * builds the image if needed
+ * prints information about the build
+ * constructs docker compose command to enter shell
+ * executes it
+
+ :param verbose: print commands when running
+ :param dry_run: do not execute "write" commands - just print what would happen
+ :param shell_params: parameters of the execution
+ """
+ check_docker_resources(verbose, shell_params.airflow_image_name)
+ build_ci_image_check_cache = Path(
+ BUILD_CACHE_DIR, shell_params.airflow_branch, f".built_{shell_params.python}"
+ )
+
+ ci_image_params = get_ci_image_build_params(
+ {"python": shell_params.python, "upgrade_to_newer_dependencies": "false"}
+ )
+ if build_ci_image_check_cache.exists():
+ console.print(f'[bright_blue]{shell_params.the_image_type} image already built locally.[/]')
+ else:
+ console.print(
+ f'[bright_yellow]{shell_params.the_image_type} image not built locally. ' f'Forcing build.[/]'
+ )
+ ci_image_params.force_build = True
+
+ build_ci_image(verbose, dry_run=dry_run, with_ci_group=False, ci_image_params=ci_image_params)
+ shell_params.print_badge_info()
+ cmd = ['docker-compose', 'run', '--service-ports', "-e", "BREEZE", '--rm', 'airflow']
+ cmd_added = shell_params.command_passed
+ env_variables = construct_env_variables_docker_compose_command(shell_params)
+ if cmd_added is not None:
+ cmd.extend(['-c', cmd_added])
+ run_command(cmd, verbose=verbose, dry_run=dry_run, env=env_variables, text=True)
diff --git a/dev/breeze/src/airflow_breeze/shell/shell_params.py b/dev/breeze/src/airflow_breeze/shell/shell_params.py
index ebc991d90e..45acd83085 100644
--- a/dev/breeze/src/airflow_breeze/shell/shell_params.py
+++ b/dev/breeze/src/airflow_breeze/shell/shell_params.py
@@ -17,7 +17,7 @@
"""Breeze shell paameters."""
from dataclasses import dataclass
from pathlib import Path
-from typing import Tuple
+from typing import Optional, Tuple
from airflow_breeze.branch_defaults import AIRFLOW_BRANCH
from airflow_breeze.global_constants import (
@@ -57,18 +57,17 @@ class ShellParams:
load_default_connections: bool = False
use_airflow_version: str = ""
install_airflow_version: str = ""
- tag: str = "latest"
+ image_tag: str = "latest"
github_repository: str = "apache/airflow"
mount_sources: str = MOUNT_SELECTED
forward_credentials: str = "false"
airflow_branch: str = AIRFLOW_BRANCH
start_airflow: str = "false"
- skip_twine_check: str = ""
+ skip_package_verification: bool = False
github_actions: str = ""
issue_id: str = ""
num_runs: str = ""
- version_suffix_for_pypi: str = ""
- version_suffix_for_svn: str = ""
+ answer: Optional[str] = None
db_reset: bool = False
ci: bool = False
@@ -105,7 +104,7 @@ class ShellParams:
@property
def airflow_image_name_with_tag(self) -> str:
image = self.airflow_image_name
- return image if not self.tag else image + f":{self.tag}"
+ return image if not self.image_tag else image + f":{self.image_tag}"
@property
def airflow_image_kubernetes(self) -> str:
diff --git a/scripts/ci/images/ci_pull_prod_image_on_ci.sh b/dev/breeze/src/airflow_breeze/utils/ci_group.py
old mode 100755
new mode 100644
similarity index 53%
rename from scripts/ci/images/ci_pull_prod_image_on_ci.sh
rename to dev/breeze/src/airflow_breeze/utils/ci_group.py
index 7b6f9cb79c..4a0508334b
--- a/scripts/ci/images/ci_pull_prod_image_on_ci.sh
+++ b/dev/breeze/src/airflow_breeze/utils/ci_group.py
@@ -1,4 +1,3 @@
-#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -16,20 +15,28 @@
# specific language governing permissions and limitations
# under the License.
-export VERBOSE="true"
+import os
+from contextlib import contextmanager
-# shellcheck source=scripts/ci/libraries/_script_init.sh
-. "$( dirname "${BASH_SOURCE[0]}" )/../libraries/_script_init.sh"
-# Pulls prepared PROD image in the CI environment
-function pull_prod_images_on_ci() {
- build_images::prepare_prod_build
- start_end::group_start "Pull PROD image ${AIRFLOW_PROD_IMAGE}"
- build_images::clean_build_cache
- local image_name_with_tag="${AIRFLOW_PROD_IMAGE}:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
- push_pull_remove_images::wait_for_image "${image_name_with_tag}"
- docker_v tag "${image_name_with_tag}" "${AIRFLOW_PROD_IMAGE}"
- start_end::group_end
-}
+@contextmanager
+def ci_group(title, enabled: bool = False):
+ """
+ If used in GitHub Action, creates an expandable group in the GitHub Action log.
+ Otherwise, display simple text groups.
-pull_prod_images_on_ci
+ For more information, see:
+ https://docs.github.com/en/free-pro-team@latest/actions/reference/workflow-commands-for-github-actions#grouping-log-lines
+ """
+ if not enabled:
+ yield
+ return
+ if os.environ.get('GITHUB_ACTIONS', 'false') != "true":
+ print("#" * 20, title, "#" * 20)
+ yield
+ return
+ print(f"::group::{title}")
+ print()
+ yield
+ print("\033[0m")
+ print("::endgroup::")
diff --git a/dev/breeze/src/airflow_breeze/utils/confirm.py b/dev/breeze/src/airflow_breeze/utils/confirm.py
index 620608d295..5fcfe217c1 100644
--- a/dev/breeze/src/airflow_breeze/utils/confirm.py
+++ b/dev/breeze/src/airflow_breeze/utils/confirm.py
@@ -56,12 +56,13 @@ def user_confirm(
allowed_answers = "y/n/q" if quit_allowed else "y/n"
while True:
try:
- force = forced_answer or os.environ.get('FORCE_ANSWER_TO_QUESTIONS')
+ force = forced_answer or os.environ.get('ANSWER')
if force:
user_status = force
else:
user_status = inputimeout(
- prompt=f'\n{message} \nPress {allowed_answers} in {timeout} seconds: ',
+ prompt=f'\n{message} \nPress {allowed_answers}'
+ + (f" in {timeout} seconds: " if timeout is not None else ": "),
timeout=timeout,
)
if user_status.upper() in ['Y', 'YES']:
diff --git a/dev/breeze/src/airflow_breeze/utils/constraints.py b/dev/breeze/src/airflow_breeze/utils/constraints.py
new file mode 100644
index 0000000000..77605678ae
--- /dev/null
+++ b/dev/breeze/src/airflow_breeze/utils/constraints.py
@@ -0,0 +1,79 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import multiprocessing as mp
+from typing import List, Tuple
+
+from airflow_breeze.shell.shell_params import ShellParams
+from airflow_breeze.utils.console import console
+from airflow_breeze.utils.docker_command_utils import (
+ construct_env_variables_docker_compose_command,
+ get_extra_docker_flags,
+)
+from airflow_breeze.utils.parallel import check_async_run_results
+from airflow_breeze.utils.run_utils import get_return_code, run_command
+
+
+def run_generate_constraints(
+ shell_params: ShellParams, dry_run: bool, verbose: bool, generate_constraints_mode: str
+) -> Tuple[int, str]:
+ env_variables = construct_env_variables_docker_compose_command(shell_params)
+ extra_docker_flags = get_extra_docker_flags(shell_params.mount_sources)
+ cmd_to_run = [
+ "docker",
+ "run",
+ "-t",
+ *extra_docker_flags,
+ "-e",
+ "SKIP_ENVIRONMENT_INITIALIZATION=true",
+ "-e",
+ f"GENERATE_CONSTRAINTS_MODE={generate_constraints_mode}",
+ "--pull",
+ "never",
+ shell_params.airflow_image_name_with_tag,
+ "/opt/airflow/scripts/in_container/run_generate_constraints.sh",
+ ]
+ process = run_command(cmd_to_run, verbose=verbose, dry_run=dry_run, env=env_variables)
+ return (
+ get_return_code(process=process, dry_run=dry_run),
+ f"Generate constraints Python {shell_params.python}:{generate_constraints_mode}",
+ )
+
+
+def run_generate_constraints_in_parallel(
+ shell_params_list: List[ShellParams],
+ python_version_list: List[str],
+ generate_constraints_mode: str,
+ parallelism: int,
+ dry_run: bool,
+ verbose: bool,
+):
+ """Run generate constraints in parallel"""
+ console.print(
+ f"\n[bright_blue]Generating constraints with parallelism = {parallelism} "
+ f"for the constraints: {python_version_list}[/]"
+ )
+ pool = mp.Pool(parallelism)
+ results = [
+ pool.apply_async(
+ run_generate_constraints,
+ args=(shell_param, dry_run, verbose, generate_constraints_mode),
+ )
+ for shell_param in shell_params_list
+ ]
+ check_async_run_results(results)
+ pool.close()
diff --git a/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py b/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
index 781cc07284..05d02c746c 100644
--- a/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
+++ b/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
@@ -18,7 +18,7 @@
import os
import re
import subprocess
-from typing import Dict, List, Union
+from typing import Dict, List, Tuple, Union
from airflow_breeze.build_image.ci.build_ci_params import BuildCiParams
from airflow_breeze.build_image.prod.build_prod_params import BuildProdParams
@@ -35,6 +35,7 @@ except ImportError:
from airflow_breeze.branch_defaults import AIRFLOW_BRANCH, DEFAULT_AIRFLOW_CONSTRAINTS_BRANCH
from airflow_breeze.global_constants import (
+ ALLOWED_PACKAGE_FORMATS,
FLOWER_HOST_PORT,
MIN_DOCKER_COMPOSE_VERSION,
MIN_DOCKER_VERSION,
@@ -52,7 +53,7 @@ from airflow_breeze.global_constants import (
WEBSERVER_HOST_PORT,
)
from airflow_breeze.utils.console import console
-from airflow_breeze.utils.run_utils import commit_sha, prepare_build_command, run_command
+from airflow_breeze.utils.run_utils import commit_sha, get_return_code, prepare_build_command, run_command
NECESSARY_HOST_VOLUMES = [
"/.bash_aliases:/root/.bash_aliases:cached",
@@ -268,15 +269,22 @@ def construct_arguments_for_docker_build_command(
:param optional_args: build arguments that are optional (should not be used if missing or empty)
:return: list of `--build-arg` commands to use for the parameters passed
"""
+
+ def get_env_variable_value(arg_name: str):
+ value = str(getattr(image_params, arg_name))
+ value = "true" if value == "True" else value
+ value = "false" if value == "False" else value
+ return value
+
args_command = []
- for param in required_args:
+ for required_arg in required_args:
args_command.append("--build-arg")
- args_command.append(param.upper() + "=" + str(getattr(image_params, param)))
- for verify_param in optional_args:
- param_value = str(getattr(image_params, verify_param))
+ args_command.append(required_arg.upper() + "=" + get_env_variable_value(arg_name=required_arg))
+ for optional_arg in optional_args:
+ param_value = get_env_variable_value(optional_arg)
if len(param_value) > 0:
args_command.append("--build-arg")
- args_command.append(verify_param.upper() + "=" + param_value)
+ args_command.append(optional_arg.upper() + "=" + param_value)
args_command.extend(image_params.docker_cache_directive)
return args_command
@@ -338,7 +346,9 @@ def construct_docker_push_command(
return ["docker", "push", image_params.airflow_image_name_with_tag]
-def tag_and_push_image(image_params: Union[BuildProdParams, BuildCiParams], dry_run: bool, verbose: bool):
+def tag_and_push_image(
+ image_params: Union[BuildProdParams, BuildCiParams], dry_run: bool, verbose: bool
+) -> Tuple[int, str]:
"""
Tag and push the image according to parameters.
:param image_params: parameters of the image
@@ -351,10 +361,23 @@ def tag_and_push_image(image_params: Union[BuildProdParams, BuildCiParams], dry_
f"{image_params.airflow_image_name_with_tag}.[/]"
)
cmd = construct_docker_tag_command(image_params)
- run_command(cmd, verbose=verbose, dry_run=dry_run, cwd=AIRFLOW_SOURCES_ROOT, text=True, check=True)
- login_to_docker_registry(image_params)
- cmd = construct_docker_push_command(image_params)
- run_command(cmd, verbose=verbose, dry_run=dry_run, cwd=AIRFLOW_SOURCES_ROOT, text=True, check=True)
+ process = run_command(
+ cmd, verbose=verbose, dry_run=dry_run, cwd=AIRFLOW_SOURCES_ROOT, text=True, check=False
+ )
+ if process and process.returncode == 0:
+ return_code, info = login_to_docker_registry(image_params, dry_run=dry_run)
+ if return_code != 0:
+ return return_code, f"Tag and pushing the image {image_params.python}: {info}"
+ cmd = construct_docker_push_command(image_params)
+ process = run_command(
+ cmd, verbose=verbose, dry_run=dry_run, cwd=AIRFLOW_SOURCES_ROOT, text=True, check=False
+ )
+ if process and process.returncode == 0:
+ return 0, f"Tag and pushing the image {image_params.python}"
+ return (
+ get_return_code(process=process, dry_run=dry_run),
+ f"Tag and pushing the image {image_params.python}",
+ )
def construct_empty_docker_build_command(
@@ -386,6 +409,7 @@ def update_expected_environment_variables(env: Dict[str, str]) -> None:
:param env: environment variables to update with missing values if not set.
"""
+ set_value_to_default_if_not_set(env, 'ANSWER', "")
set_value_to_default_if_not_set(env, 'BREEZE', "true")
set_value_to_default_if_not_set(env, 'CI', "false")
set_value_to_default_if_not_set(env, 'CI_BUILD_ID', "0")
@@ -405,11 +429,11 @@ def update_expected_environment_variables(env: Dict[str, str]) -> None:
set_value_to_default_if_not_set(env, 'HOST_OS', get_host_os())
set_value_to_default_if_not_set(env, 'INIT_SCRIPT_FILE', "init.sh")
set_value_to_default_if_not_set(env, 'INSTALL_PROVIDERS_FROM_SOURCES', "true")
- set_value_to_default_if_not_set(env, 'INSTALL_PROVIDERS_FROM_SOURCES', "true")
+ set_value_to_default_if_not_set(env, 'INSTALL_FROM_DOCKER_CONTEXT_FILES', "false")
set_value_to_default_if_not_set(env, 'LIST_OF_INTEGRATION_TESTS_TO_RUN', "")
set_value_to_default_if_not_set(env, 'LOAD_DEFAULT_CONNECTIONS', "false")
set_value_to_default_if_not_set(env, 'LOAD_EXAMPLES', "false")
- set_value_to_default_if_not_set(env, 'PACKAGE_FORMAT', "wheel")
+ set_value_to_default_if_not_set(env, 'PACKAGE_FORMAT', ALLOWED_PACKAGE_FORMATS[0])
set_value_to_default_if_not_set(env, 'PRINT_INFO_FROM_SCRIPTS', "true")
set_value_to_default_if_not_set(env, 'PYTHONDONTWRITEBYTECODE', "true")
set_value_to_default_if_not_set(env, 'RUN_SYSTEM_TESTS', "false")
@@ -425,13 +449,14 @@ def update_expected_environment_variables(env: Dict[str, str]) -> None:
set_value_to_default_if_not_set(env, 'WHEEL_VERSION', "0.36.2")
-VARIABLES_TO_ENTER_DOCKER_COMPOSE = {
+MAP_ENV_VARIABLES_TO_PARAMS_FIELDS = {
"AIRFLOW_CI_IMAGE": "airflow_image_name",
"AIRFLOW_CI_IMAGE_WITH_TAG": "airflow_image_name_with_tag",
"AIRFLOW_IMAGE_KUBERNETES": "airflow_image_kubernetes",
"AIRFLOW_PROD_IMAGE": "airflow_image_name",
"AIRFLOW_SOURCES": "airflow_sources",
"AIRFLOW_VERSION": "airflow_version",
+ "ANSWER": "answer",
"BACKEND": "backend",
"COMPOSE_FILE": "compose_files",
"DB_RESET": 'db_reset',
@@ -445,12 +470,10 @@ VARIABLES_TO_ENTER_DOCKER_COMPOSE = {
"LOAD_DEFAULT_CONNECTIONS": "load_default_connections",
"NUM_RUNS": "num_runs",
"PYTHON_MAJOR_MINOR_VERSION": "python",
- "SKIP_TWINE_CHECK": "skip_twine_check",
+ "SKIP_PACKAGE_VERIFICATION": "skip_package_verification",
"SQLITE_URL": "sqlite_url",
"START_AIRFLOW": "start_airflow",
"USE_AIRFLOW_VERSION": "use_airflow_version",
- "VERSION_SUFFIX_FOR_PYPI": "version_suffix_for_pypi",
- "VERSION_SUFFIX_FOR_SVN": "version_suffix_for_svn",
}
VARIABLES_FOR_DOCKER_COMPOSE_CONSTANTS = {
@@ -498,12 +521,14 @@ def construct_env_variables_docker_compose_command(shell_params: ShellParams) ->
:return: dictionary of env variables to set
"""
env_variables: Dict[str, str] = os.environ.copy()
- for param_name in VARIABLES_TO_ENTER_DOCKER_COMPOSE:
- param_value = VARIABLES_TO_ENTER_DOCKER_COMPOSE[param_name]
- env_variables[param_name] = str(getattr(shell_params, param_value))
- for constant_param_name in VARIABLES_FOR_DOCKER_COMPOSE_CONSTANTS:
- constant_param_value = VARIABLES_FOR_DOCKER_COMPOSE_CONSTANTS[constant_param_name]
+ for variable in MAP_ENV_VARIABLES_TO_PARAMS_FIELDS:
+ param_name = MAP_ENV_VARIABLES_TO_PARAMS_FIELDS[variable]
+ param_value = getattr(shell_params, param_name)
+ env_variables[variable] = str(param_value) if param_value is not None else ""
+ # Set constant defaults if not defined
+ for variable in VARIABLES_FOR_DOCKER_COMPOSE_CONSTANTS:
+ constant_param_value = VARIABLES_FOR_DOCKER_COMPOSE_CONSTANTS[variable]
if not env_variables.get(constant_param_value):
- env_variables[constant_param_name] = str(constant_param_value)
+ env_variables[variable] = str(constant_param_value)
update_expected_environment_variables(env_variables)
return env_variables
diff --git a/dev/breeze/src/airflow_breeze/utils/md5_build_check.py b/dev/breeze/src/airflow_breeze/utils/md5_build_check.py
index 37191a1ef6..5d445a207f 100644
--- a/dev/breeze/src/airflow_breeze/utils/md5_build_check.py
+++ b/dev/breeze/src/airflow_breeze/utils/md5_build_check.py
@@ -86,12 +86,11 @@ def calculate_md5_checksum_for_files(
return modified_files, not_modified_files
-def md5sum_check_if_build_is_needed(md5sum_cache_dir: Path, the_image_type: str) -> bool:
+def md5sum_check_if_build_is_needed(md5sum_cache_dir: Path) -> bool:
"""
Checks if build is needed based on whether important files were modified.
:param md5sum_cache_dir: directory where cached md5 sums are stored
- :param the_image_type: type of the image to check (PROD/CI)
:return: True if build is needed.
"""
build_needed = False
@@ -102,12 +101,10 @@ def md5sum_check_if_build_is_needed(md5sum_cache_dir: Path, the_image_type: str)
)
for file in modified_files:
console.print(f" * [bright_blue]{file}[/]")
- console.print(f'\n[bright_yellow]Likely {the_image_type} image needs rebuild[/]\n')
+ console.print('\n[bright_yellow]Likely CI image needs rebuild[/]\n')
build_needed = True
else:
- console.print(
- f'Docker image build is not needed for {the_image_type} build as no important files are changed!'
- )
+ console.print('Docker image build is not needed for CI build as no important files are changed!')
return build_needed
diff --git a/dev/breeze/src/airflow_breeze/utils/parallel.py b/dev/breeze/src/airflow_breeze/utils/parallel.py
new file mode 100644
index 0000000000..565a1a52c9
--- /dev/null
+++ b/dev/breeze/src/airflow_breeze/utils/parallel.py
@@ -0,0 +1,81 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import sys
+import time
+from multiprocessing.pool import ApplyResult
+from typing import List
+
+from airflow_breeze.utils.console import console
+
+
+def print_async_summary(completed_list: List[ApplyResult]) -> None:
+ """
+ Print summary of completed async results.
+ :param completed_list: list of completed async results.
+ """
+ completed_list.sort(key=lambda x: x.get()[1])
+ console.print()
+ for result in completed_list:
+ return_code, info = result.get()
+ if return_code != 0:
+ console.print(f"[red]NOK[/] for {info}: Return code: {return_code}.")
+ else:
+ console.print(f"[green]OK [/] for {info}.")
+ console.print()
+
+
+def get_completed_result_list(results: List[ApplyResult]) -> List[ApplyResult]:
+ """Return completed results from the list."""
+ return list(filter(lambda result: result.ready(), results))
+
+
+def check_async_run_results(results: List[ApplyResult], poll_time: float = 0.2):
+ """
+ Check if all async results were success. Exits with error if not.
+ :param results: results of parallel runs (expected in the form of Tuple: (return_code, info)
+ :param poll_time: what's the poll time between checks
+ """
+ completed_number = 0
+ total_number_of_results = len(results)
+ completed_list = get_completed_result_list(results)
+ while not len(completed_list) == total_number_of_results:
+ current_completed_number = len(completed_list)
+ if current_completed_number != completed_number:
+ completed_number = current_completed_number
+ console.print(
+ f"\n[bright_blue]Completed {completed_number} out of {total_number_of_results} "
+ f"({int(100*completed_number/total_number_of_results)}%).[/]\n"
+ )
+ print_async_summary(completed_list)
+ time.sleep(poll_time)
+ completed_list = get_completed_result_list(results)
+ completed_number = len(completed_list)
+ console.print(
+ f"\n[bright_blue]Completed {completed_number} out of {total_number_of_results} "
+ f"({int(100*completed_number/total_number_of_results)}%).[/]\n"
+ )
+ print_async_summary(completed_list)
+ errors = False
+ for result in results:
+ if result.get()[0] != 0:
+ errors = True
+ if errors:
+ console.print("\n[red]There were errors when running some tasks. Quitting.[/]\n")
+ sys.exit(1)
+ else:
+ console.print("\n[green]All images are OK.[/]\n")
diff --git a/dev/breeze/src/airflow_breeze/utils/path_utils.py b/dev/breeze/src/airflow_breeze/utils/path_utils.py
index 12eaf5c498..b61fb59e20 100644
--- a/dev/breeze/src/airflow_breeze/utils/path_utils.py
+++ b/dev/breeze/src/airflow_breeze/utils/path_utils.py
@@ -61,7 +61,7 @@ def in_help() -> bool:
def skip_upgrade_check():
- return in_self_upgrade() or in_autocomplete() or in_help()
+ return in_self_upgrade() or in_autocomplete() or in_help() or hasattr(sys, '_called_from_test')
def get_package_setup_metadata_hash() -> str:
diff --git a/dev/breeze/src/airflow_breeze/utils/pulll_image.py b/dev/breeze/src/airflow_breeze/utils/pulll_image.py
new file mode 100644
index 0000000000..4e470e032c
--- /dev/null
+++ b/dev/breeze/src/airflow_breeze/utils/pulll_image.py
@@ -0,0 +1,170 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import multiprocessing as mp
+import time
+from typing import List, Tuple, Union
+
+from airflow_breeze.build_image.ci.build_ci_params import BuildCiParams
+from airflow_breeze.build_image.prod.build_prod_params import BuildProdParams
+from airflow_breeze.utils.console import console
+from airflow_breeze.utils.parallel import check_async_run_results
+from airflow_breeze.utils.run_tests import verify_an_image
+from airflow_breeze.utils.run_utils import get_return_code, run_command
+
+
+def run_pull_in_parallel(
+ dry_run: bool,
+ parallelism: int,
+ image_params_list: Union[List[BuildCiParams], List[BuildProdParams]],
+ python_version_list: List[str],
+ verbose: bool,
+ verify_image: bool,
+ tag_as_latest: bool,
+ wait_for_image: bool,
+ extra_pytest_args: Tuple,
+):
+ """Run image pull in parallel"""
+ console.print(
+ f"\n[bright_blue]Pulling with parallelism = {parallelism} for the images: {python_version_list}:"
+ )
+ pool = mp.Pool(parallelism)
+ poll_time = 10.0
+ if not verify_image:
+ results = [
+ pool.apply_async(
+ run_pull_image, args=(image_param, dry_run, verbose, wait_for_image, tag_as_latest, poll_time)
+ )
+ for image_param in image_params_list
+ ]
+ else:
+ results = [
+ pool.apply_async(
+ run_pull_and_verify_image,
+ args=(
+ image_param,
+ dry_run,
+ verbose,
+ wait_for_image,
+ tag_as_latest,
+ poll_time,
+ extra_pytest_args,
+ ),
+ )
+ for image_param in image_params_list
+ ]
+ check_async_run_results(results)
+ pool.close()
+
+
+def run_pull_image(
+ image_params: Union[BuildCiParams, BuildProdParams],
+ dry_run: bool,
+ verbose: bool,
+ wait_for_image: bool,
+ tag_as_latest: bool,
+ poll_time: float,
+) -> Tuple[int, str]:
+ """
+ Pull image soecified.
+ :param image_params: Image parameters.
+ :param dry_run: whether it's dry run
+ :param verbose: whether it's verbose
+ :param wait_for_image: whether we should wait for the image to be available
+ :param tag_as_latest: tag the image as latest
+ :param poll_time: what's the polling time between checks if images are there
+ :return: Tuple of return code and description of the image pulled
+ """
+ console.print(
+ f"\n[bright_blue]Pulling {image_params.the_image_type} image of airflow python version: "
+ f"{image_params.python} image: {image_params.airflow_image_name_with_tag} "
+ f"with wait for image: {wait_for_image}[/]\n"
+ )
+ while True:
+ command_to_run = ["docker", "pull", image_params.airflow_image_name_with_tag]
+ process = run_command(
+ command_to_run,
+ verbose=verbose,
+ dry_run=dry_run,
+ check=False,
+ )
+ if dry_run or process and process.returncode == 0:
+ process = run_command(
+ ["docker", "inspect", image_params.airflow_image_name_with_tag, "-f", "{{.Size}}"],
+ capture_output=True,
+ verbose=verbose,
+ dry_run=dry_run,
+ text=True,
+ check=False,
+ )
+ if not dry_run:
+ if process and process.returncode == 0:
+ image_size = int(process.stdout.strip())
+ if image_size == 0:
+ console.print("\n[red]The image size was 0 - image creation failed.[/]\n")
+ return 1, f"Image Python {image_params.python}"
+ else:
+ console.print("\n[red]There was an error pulling the size of the image. Failing.[/]\n")
+ return (
+ get_return_code(process=process, dry_run=dry_run),
+ f"Image Python {image_params.python}",
+ )
+ if tag_as_latest:
+ process = run_command(
+ [
+ "docker",
+ "tag",
+ image_params.airflow_image_name_with_tag,
+ image_params.airflow_image_name,
+ ],
+ capture_output=True,
+ verbose=verbose,
+ dry_run=dry_run,
+ check=False,
+ )
+ return get_return_code(process=process, dry_run=dry_run), f"Image Python {image_params.python}"
+ if wait_for_image:
+ if verbose or dry_run:
+ console.print(f"\n[bright_blue]Waiting for {poll_time} seconds.[/]\n")
+ time.sleep(poll_time)
+ continue
+ else:
+ console.print(f"\n[red]There was an error pulling the image {image_params.python}. Failing.[/]\n")
+ return get_return_code(process=process, dry_run=dry_run), f"Image Python {image_params.python}"
+
+
+def run_pull_and_verify_image(
+ image_params: Union[BuildCiParams, BuildProdParams],
+ dry_run: bool,
+ verbose: bool,
+ wait_for_image: bool,
+ tag_as_latest: bool,
+ poll_time: float,
+ extra_pytest_args: Tuple,
+) -> Tuple[int, str]:
+ return_code, info = run_pull_image(
+ image_params, dry_run, verbose, wait_for_image, tag_as_latest, poll_time
+ )
+ if return_code != 0:
+ console.print(f"\n[red]Not running verification for {image_params.python} as pulling failed.[/]\n")
+ return verify_an_image(
+ image_name=image_params.airflow_image_name_with_tag,
+ image_type=image_params.the_image_type,
+ dry_run=dry_run,
+ verbose=verbose,
+ extra_pytest_args=extra_pytest_args,
+ )
diff --git a/dev/breeze/src/airflow_breeze/utils/registry.py b/dev/breeze/src/airflow_breeze/utils/registry.py
index 4f8b3bc66b..8e1a980ed5 100644
--- a/dev/breeze/src/airflow_breeze/utils/registry.py
+++ b/dev/breeze/src/airflow_breeze/utils/registry.py
@@ -16,20 +16,23 @@
# under the License.
import os
-from typing import Union
+from typing import Tuple, Union
from airflow_breeze.build_image.ci.build_ci_params import BuildCiParams
from airflow_breeze.build_image.prod.build_prod_params import BuildProdParams
from airflow_breeze.utils.console import console
-from airflow_breeze.utils.run_utils import run_command
+from airflow_breeze.utils.run_utils import get_return_code, run_command
-def login_to_docker_registry(image_params: Union[BuildProdParams, BuildCiParams]):
+def login_to_docker_registry(
+ image_params: Union[BuildProdParams, BuildCiParams], dry_run: bool
+) -> Tuple[int, str]:
"""
In case of CI environment, we need to login to GitHub Registry if we want to prepare cache.
This method logs in using the params specified.
:param image_params: parameters to use for Building prod image
+ :param dry_run: whether we are in dry_run mode
"""
if os.environ.get("CI"):
if len(image_params.github_token) == 0:
@@ -40,8 +43,8 @@ def login_to_docker_registry(image_params: Union[BuildProdParams, BuildCiParams]
LOGIN_TO_GITHUB_REGISTRY is set as false"
)
elif len(image_params.github_token) > 0:
- run_command(['docker', 'logout', 'ghcr.io'], verbose=True, text=True)
- run_command(
+ run_command(['docker', 'logout', 'ghcr.io'], verbose=True, text=False, check=False)
+ process = run_command(
[
'docker',
'login',
@@ -53,7 +56,9 @@ def login_to_docker_registry(image_params: Union[BuildProdParams, BuildCiParams]
verbose=True,
text=True,
input=image_params.github_token,
- check=True,
+ check=False,
)
+ return get_return_code(process=process, dry_run=dry_run), "Docker login"
else:
console.print('\n[bright_blue]Skip Login to GitHub Container Registry as token is missing')
+ return 0, "Docker login skipped"
diff --git a/dev/breeze/src/airflow_breeze/utils/run_tests.py b/dev/breeze/src/airflow_breeze/utils/run_tests.py
new file mode 100644
index 0000000000..b255d3cb2f
--- /dev/null
+++ b/dev/breeze/src/airflow_breeze/utils/run_tests.py
@@ -0,0 +1,80 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import sys
+from subprocess import DEVNULL
+from typing import Tuple
+
+from airflow_breeze.utils.console import console
+from airflow_breeze.utils.path_utils import AIRFLOW_SOURCES_ROOT
+from airflow_breeze.utils.run_utils import run_command
+
+
+def verify_an_image(
+ image_name: str, image_type: str, dry_run: bool, verbose: bool, extra_pytest_args: Tuple
+) -> Tuple[int, str]:
+ process = run_command(
+ ["docker", "inspect", image_name], dry_run=dry_run, verbose=verbose, check=False, stdout=DEVNULL
+ )
+ if process and process.returncode != 0:
+ console.print(f"[red]Error when inspecting {image_type} image: {process.returncode}[/]")
+ return process.returncode, f"Testing {image_type} python {image_name}"
+ pytest_args = ("-n", "auto", "--color=yes")
+ if image_type == 'PROD':
+ test_path = AIRFLOW_SOURCES_ROOT / "docker_tests" / "test_prod_image.py"
+ else:
+ test_path = AIRFLOW_SOURCES_ROOT / "docker_tests" / "test_ci_image.py"
+ env = os.environ.copy()
+ env['DOCKER_IMAGE'] = image_name
+ process = run_command(
+ [sys.executable, "-m", "pytest", str(test_path), *pytest_args, *extra_pytest_args],
+ dry_run=dry_run,
+ verbose=verbose,
+ env=env,
+ check=False,
+ )
+ if process:
+ return process.returncode, f"Testing {image_type} python {image_name}"
+ else:
+ return 1, f"Testing {image_type} python {image_name}"
+
+
+def run_docker_compose_tests(
+ image_name: str, dry_run: bool, verbose: bool, extra_pytest_args: Tuple
+) -> Tuple[int, str]:
+ process = run_command(
+ ["docker", "inspect", image_name], dry_run=dry_run, verbose=verbose, check=False, stdout=DEVNULL
+ )
+ if process and process.returncode != 0:
+ console.print(f"[red]Error when inspecting PROD image: {process.returncode}[/]")
+ return process.returncode, f"Testing docker-compose python with {image_name}"
+ pytest_args = ("-n", "auto", "--color=yes")
+ test_path = AIRFLOW_SOURCES_ROOT / "docker_tests" / "test_docker_compose_quick_start.py"
+ env = os.environ.copy()
+ env['DOCKER_IMAGE'] = image_name
+ process = run_command(
+ [sys.executable, "-m", "pytest", str(test_path), *pytest_args, *extra_pytest_args],
+ dry_run=dry_run,
+ verbose=verbose,
+ env=env,
+ check=False,
+ )
+ if process:
+ return process.returncode, f"Testing docker-compose python with {image_name}"
+ else:
+ return 1, f"Testing docker-compose python with {image_name}"
diff --git a/dev/breeze/src/airflow_breeze/utils/run_utils.py b/dev/breeze/src/airflow_breeze/utils/run_utils.py
index a56746cced..76d1996308 100644
--- a/dev/breeze/src/airflow_breeze/utils/run_utils.py
+++ b/dev/breeze/src/airflow_breeze/utils/run_utils.py
@@ -163,8 +163,7 @@ def get_filesystem_type(filepath):
def instruct_build_image(python: str):
"""Print instructions to the user that they should build the image"""
console.print(f'[bright_yellow]\nThe CI image for ' f'python version {python} may be outdated[/]\n')
- console.print('Please run this command at earliest convenience:\n')
- console.print(f' `./breeze build-image --python {python}`\n')
+ print(f"\n[yellow]Please run at the earliest convenience:[/]\n\nbreeze build-image --python {python}\n\n")
@contextlib.contextmanager
@@ -312,3 +311,11 @@ def filter_out_none(**kwargs) -> Dict[str, str]:
if kwargs[key] is None:
kwargs.pop(key)
return kwargs
+
+
+def get_return_code(process: Optional[subprocess.CompletedProcess], dry_run: bool):
+ if process:
+ return process.returncode
+ if dry_run:
+ return 0
+ return 1
diff --git a/scripts/in_container/run_migration_reference.sh b/dev/breeze/tests/conftest.py
old mode 100755
new mode 100644
similarity index 77%
rename from scripts/in_container/run_migration_reference.sh
rename to dev/breeze/tests/conftest.py
index 97b188cdf8..875b1744be
--- a/scripts/in_container/run_migration_reference.sh
+++ b/dev/breeze/tests/conftest.py
@@ -1,4 +1,3 @@
-#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -15,6 +14,15 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-# shellcheck source=scripts/in_container/_in_container_script_init.sh
-. "$( dirname "${BASH_SOURCE[0]}" )/_in_container_script_init.sh"
-python scripts/ci/pre_commit/pre_commit_migration_reference.py "$@"
+
+# content of conftest.py
+def pytest_configure(config):
+ import sys
+
+ sys._called_from_test = True
+
+
+def pytest_unconfigure(config):
+ import sys # This was missing from the manual
+
+ del sys._called_from_test
diff --git a/dev/breeze/tests/test_cache.py b/dev/breeze/tests/test_cache.py
index 2e0931d2d5..7bc53ae560 100644
--- a/dev/breeze/tests/test_cache.py
+++ b/dev/breeze/tests/test_cache.py
@@ -27,7 +27,7 @@ from airflow_breeze.utils.cache import (
read_from_cache_file,
)
-AIRFLOW_SOURCES = Path(__file__).parent.parent.parent.parent
+AIRFLOW_SOURCES = Path(__file__).parents[3].absolute()
@pytest.mark.parametrize(
diff --git a/dev/breeze/tests/test_find_airflow_directory.py b/dev/breeze/tests/test_find_airflow_directory.py
index ba3d50500b..f0280142c3 100644
--- a/dev/breeze/tests/test_find_airflow_directory.py
+++ b/dev/breeze/tests/test_find_airflow_directory.py
@@ -21,7 +21,7 @@ from unittest import mock
from airflow_breeze.utils.path_utils import find_airflow_sources_root_to_operate_on
-ACTUAL_AIRFLOW_SOURCES = Path(__file__).parent.parent.parent.parent
+ACTUAL_AIRFLOW_SOURCES = Path(__file__).parents[3].absolute()
ROOT_PATH = Path(Path(__file__).root)
diff --git a/dev/breeze/tests/test_prod_image.py b/dev/breeze/tests/test_prod_image.py
index 1c4b406901..a93e3ecf04 100644
--- a/dev/breeze/tests/test_prod_image.py
+++ b/dev/breeze/tests/test_prod_image.py
@@ -26,7 +26,7 @@ default_params: Dict[str, Union[str, bool]] = {
'disable_mysql_client_installation': False,
'disable_mssql_client_installation': False,
'disable_postgres_client_installation': False,
- 'install_docker_context_files': False,
+ 'install_from_docker_context_files': False,
'disable_airflow_repo_cache': False,
'upgrade_to_newer_dependencies': "false",
'install_providers_from_sources': False,
diff --git a/dev/provider_packages/README.md b/dev/provider_packages/README.md
index 408d6c16cd..cd6f4ce8b1 100644
--- a/dev/provider_packages/README.md
+++ b/dev/provider_packages/README.md
@@ -34,8 +34,6 @@
- [Debugging preparing release-notes](#debugging-preparing-release-notes)
- [Debugging preparing setup files](#debugging-preparing-setup-files)
- [Debugging preparing the packages](#debugging-preparing-the-packages)
-- [Testing provider packages](#testing-provider-packages)
- - [Regular packages](#regular-packages)
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
@@ -97,7 +95,7 @@ The `index.rst` stored in the `docs\apache-airflow-providers-<PROVIDER>` folder
When you want to prepare release notes for a package, you need to run:
```
-./breeze-legacy prepare-provider-documentation <PACKAGE_ID> ...
+breeze prepare-provider-documentation <PACKAGE_ID> ...
```
* <PACKAGE_ID> is usually directory in the `airflow/providers` folder (for example `google` but in several
@@ -118,7 +116,7 @@ You should manually update generated changelog and classify the commits updated
You can repeat this several times, the changes generated will automatically include new commits that
appeared since last run.
-You can also run it in non-interactive mode adding `--non-interactive` flag.
+You can also run it in non-interactive mode adding `--answer yes` flag.
## Preparing provider packages
@@ -133,7 +131,7 @@ the folders (for example Apache Hive's PACKAGE_ID is `apache.hive` ). You can se
providers by running:
```bash
-./breeze prepare-provider-packages -- --help
+breeze prepare-provider-packages --help
```
The examples below show how you can build selected packages, but you can also build all packages by
@@ -142,35 +140,22 @@ omitting the package ids altogether.
By default, you build `both` packages, but you can use `--package-format wheel` to generate
only wheel package, or `--package-format sdist` to only generate sdist package.
-* To build the release candidate packages for SVN Apache upload run the following command:
-
-```bash
-./breeze prepare-provider-packages --version-suffix-for-svn=rc1 [PACKAGE_ID] ...
-```
-
-for example:
-
-```bash
-./breeze prepare-provider-packages --version-suffix-for-svn=rc1 http ...
-```
-
* To build the release candidate packages for PyPI upload run the following command:
```bash
-./breeze prepare-provider-packages --version-suffix-for-pypi=rc1 [PACKAGE_ID] ...
+breeze prepare-provider-packages --version-suffix-for-pypi=rc1 [PACKAGE_ID] ...
```
for example:
```bash
-./breeze prepare-provider-packages --version-suffix-for-pypi=rc1 http ...
+breeze prepare-provider-packages --version-suffix-for-pypi=rc1 http ...
```
-
* To build the final release packages run the following command:
```bash
-./breeze prepare-provider-packages [--package-format PACKAGE_FORMAT] [PACKAGE_ID] ...
+breeze prepare-provider-packages [--package-format PACKAGE_FORMAT] [PACKAGE_ID] ...
```
Where PACKAGE_FORMAT might be one of : `wheel`, `sdist`, `both` (`wheel` is the default format)
@@ -178,7 +163,7 @@ Where PACKAGE_FORMAT might be one of : `wheel`, `sdist`, `both` (`wheel` is the
for example:
```bash
-./breeze prepare-provider-packages http ...
+breeze prepare-provider-packages http ...
```
* For each package, this creates a wheel package and source distribution package in your `dist` folder with
@@ -369,69 +354,3 @@ specifying ``--version-suffix`` (for example ``--version-suffix dev``).
By default, you prepare ``both`` packages, but you can add ``--package-format`` argument and specify
``wheel``, ``sdist`` to build only one of them.
-
-
-# Testing provider packages
-
-The provider packages importing and tests execute within the "CI" environment of Airflow -the
-same image that is used by Breeze. They however require special mounts (no
-sources of Airflow mounted to it) and possibility to install all extras and packages in order to test
-if all classes can be imported. It is rather simple but requires some semi-automated process:
-
-
-## Regular packages
-
-1. Prepare regular packages
-
-```shell script
-./breeze prepare-provider-packages
-```
-
-This prepares all provider packages in the "dist" folder
-
-2. Prepare airflow package from sources
-
-```shell script
-./breeze prepare-airflow-packages
-```
-
-This prepares airflow package in the "dist" folder
-
-2. Enter the container:
-
-```shell script
-export USE_AIRFLOW_VERSION="wheel"
-export USE_PACKAGES_FROM_DIST="true"
-
-./dev/provider_packages/enter_breeze_provider_package_tests.sh
-```
-
-(the rest of it is in the container)
-
-3. \[IN CONTAINER\] Install apache-beam.
-
-```shell script
-pip install apache-beam[gcp]
-```
-
-4. \[IN CONTAINER\] Install the provider packages from /dist
-
-```shell script
-pip install --no-deps /dist/apache_airflow_providers_*.whl
-```
-
-Note! No-deps is because we are installing the version installed from wheel package.
-
-5. \[IN CONTAINER\] Check the installation folder for providers:
-
-```shell script
-python3 <<EOF 2>/dev/null
-import airflow.providers;
-path=airflow.providers.__path__
-for p in path._path:
- print(p)
-EOF
-```
-
-6. \[IN CONTAINER\] Check if all the providers can be imported
-python3 /opt/airflow/dev/import_all_classes.py --path <PATH_REPORTED_IN_THE_PREVIOUS_STEP>
diff --git a/dev/provider_packages/enter_breeze_provider_package_tests.sh b/dev/provider_packages/enter_breeze_provider_package_tests.sh
deleted file mode 100755
index b67b33f469..0000000000
--- a/dev/provider_packages/enter_breeze_provider_package_tests.sh
+++ /dev/null
@@ -1,38 +0,0 @@
-#!/usr/bin/env bash
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-export MOUNT_SELECTED_LOCAL_SOURCES="false"
-
-# shellcheck source=scripts/ci/libraries/_script_init.sh
-. "$(dirname "${BASH_SOURCE[0]}")/../../scripts/ci/libraries/_script_init.sh"
-
-function enter_breeze_with_mapped_sources() {
- docker_v run -it "${EXTRA_DOCKER_FLAGS[@]}" \
- -v "${AIRFLOW_SOURCES}/setup.py:/airflow_sources/setup.py:cached" \
- -v "${AIRFLOW_SOURCES}/setup.cfg:/airflow_sources/setup.cfg:cached" \
- -v "${AIRFLOW_SOURCES}/airflow/__init__.py:/airflow_sources/airflow/__init__.py:cached" \
- -v "${AIRFLOW_SOURCES}/empty:/opt/airflow/airflow:cached" \
- -v "${AIRFLOW_SOURCES}/scripts/in_container:/opt/airflow/scripts/in_container:cached" \
- -v "${AIRFLOW_SOURCES}/dev/import_all_classes.py:/opt/airflow/dev/import_all_classes.py:cached" \
- "${AIRFLOW_CI_IMAGE}"
-}
-
-build_images::prepare_ci_build
-
-build_images::rebuild_ci_image_if_needed
-
-enter_breeze_with_mapped_sources
diff --git a/dev/provider_packages/prepare_provider_packages.py b/dev/provider_packages/prepare_provider_packages.py
index 586920c270..1a46c7292d 100755
--- a/dev/provider_packages/prepare_provider_packages.py
+++ b/dev/provider_packages/prepare_provider_packages.py
@@ -137,13 +137,6 @@ option_git_update = click.option(
help=f"If the git remote {HTTPS_REMOTE} already exists, don't try to update it",
)
-option_interactive = click.option(
- '--interactive/--non-interactive',
- default=True,
- is_flag=True,
- help="If the script should interactively ask the user what to do in case of doubt",
-)
-
option_version_suffix = click.option(
"--version-suffix",
metavar="suffix",
@@ -375,7 +368,9 @@ def get_install_requirements(provider_package_id: str, version_suffix: str) -> s
# or >=2.0.1rc1 if we build rc1 version of the packages.
for dependency in additional_dependencies:
if dependency.startswith("apache-airflow") and ">=" in dependency:
- dependency = dependency + version_suffix
+ dependency = (
+ dependency + ("." if not version_suffix.startswith(".") else "") + version_suffix
+ )
install_requires.append(dependency)
else:
install_requires.extend(additional_dependencies)
@@ -1067,6 +1062,7 @@ def make_sure_remote_apache_exists_and_fetch(git_update: bool, verbose: bool):
)
except subprocess.CalledProcessError as ex:
console.print("[red]Error: when adding remote:[/]", ex)
+ sys.exit(128)
else:
raise
if verbose:
@@ -1516,20 +1512,21 @@ def prepare_readme_file(context):
readme_file.write(readme_content)
-def confirm(message: str):
+def confirm(message: str, answer: Optional[str] = None) -> bool:
"""
Ask user to confirm (case-insensitive).
- :return: True if the answer is Y. Exits with 65 exit code if Q is chosen.
- :rtype: bool
- """
- answer = ""
- while answer not in ["y", "n", "q"]:
- console.print(f"[yellow]{message}[Y/N/Q]?[/] ", end='')
- answer = input("").lower()
- if answer == "q":
+ :param message: message to display
+ :param answer: force answer if set
+ :return: True if the answer is any form of y/yes. Exits with 65 exit code if any form of q/quit is chosen.
+ """
+ given_answer = answer.lower() if answer is not None else ""
+ while given_answer not in ["y", "n", "q", "yes", "no", "quit"]:
+ console.print(f"[yellow]{message}[y/n/q]?[/] ", end='')
+ given_answer = input("").lower()
+ if given_answer.lower() in ["q", "quit"]:
# Returns 65 in case user decided to quit
sys.exit(65)
- return answer == "y"
+ return given_answer in ["y", "yes"]
def mark_latest_changes_as_documentation_only(
@@ -1552,7 +1549,7 @@ def update_release_notes(
version_suffix: str,
force: bool,
verbose: bool,
- interactive: bool,
+ answer: Optional[str],
) -> bool:
"""
Updates generated files (readme, changes and/or setup.cfg/setup.py/manifest.in/provider_info)
@@ -1561,7 +1558,7 @@ def update_release_notes(
:param version_suffix: version suffix corresponding to the version in the code
:param force: regenerate already released documentation
:param verbose: whether to print verbose messages
- :param interactive: whether the script should ask the user in case of doubt
+ :param answer: force answer to questions if set.
:returns False if the package should be skipped, True if everything generated properly
"""
verify_provider_package(provider_package_id)
@@ -1582,7 +1579,7 @@ def update_release_notes(
)
if not force:
if proceed:
- if interactive and not confirm("Provider marked for release. Proceed?"):
+ if not confirm("Provider marked for release. Proceed", answer=answer):
return False
elif not latest_change:
console.print()
@@ -1592,7 +1589,7 @@ def update_release_notes(
console.print()
return False
else:
- if interactive and confirm("Are those changes documentation-only?"):
+ if confirm("Are those changes documentation-only?", answer=answer):
if isinstance(latest_change, Change):
mark_latest_changes_as_documentation_only(provider_details, latest_change)
else:
@@ -1839,14 +1836,20 @@ def list_providers_packages():
@cli.command()
@option_version_suffix
@option_git_update
-@option_interactive
@argument_package_id
@option_force
@option_verbose
+@click.option(
+ "-a",
+ "--answer",
+ type=click.Choice(['y', 'n', 'q', 'yes', 'no', 'quit']),
+ help="Force answer to questions.",
+ envvar='ANSWER',
+)
def update_package_documentation(
version_suffix: str,
git_update: bool,
- interactive: bool,
+ answer: Optional[str],
package_id: str,
force: bool,
verbose: bool,
@@ -1862,7 +1865,7 @@ def update_package_documentation(
console.print("Updating documentation for the latest release version.")
make_sure_remote_apache_exists_and_fetch(git_update, verbose)
if not update_release_notes(
- provider_package_id, version_suffix, force=force, verbose=verbose, interactive=interactive
+ provider_package_id, version_suffix, force=force, verbose=verbose, answer=answer
):
# Returns 64 in case of skipped package
sys.exit(64)
@@ -2632,4 +2635,11 @@ if __name__ == "__main__":
# * 64 in case of skipped package
# * 65 in case user decided to quit
# * 66 in case package has doc-only changes
- cli()
+ try:
+ cli()
+ except KeyboardInterrupt:
+ print('Interrupted')
+ try:
+ sys.exit(65)
+ except SystemExit:
+ os._exit(65)
diff --git a/dev/refresh_images.sh b/dev/refresh_images.sh
index 879c0cbed7..8b3f3ff4b3 100755
--- a/dev/refresh_images.sh
+++ b/dev/refresh_images.sh
@@ -19,17 +19,11 @@
set -euo pipefail
rm -rf docker-context-files/*.whl
rm -rf docker-context-files/*.tgz
-export FORCE_ANSWER_TO_QUESTIONS="yes"
+export ANSWER="yes"
export CI="true"
-if [[ $1 == "" ]]; then
- echo
- echo ERROR! Please specify python version as parameter
- echo
- exit 1
-fi
+breeze build-image --build-multiple-images \
+ --prepare-buildx-cache --platform linux/amd64,linux/arm64 --verbose
-python_version=$1
-
-breeze build-image --prepare-buildx-cache --python "${python_version}" --platform linux/amd64,linux/arm64 --verbose
-breeze build-prod-image --prepare-buildx-cache --python "${python_version}" --platform linux/amd64,linux/arm64 --verbose
+breeze build-prod-image --build-multiple-images \
+ --prepare-buildx-cache --platform linux/amd64,linux/arm64 --verbose
diff --git a/docker-context-files/README.md b/docker-context-files/.README.md
similarity index 100%
rename from docker-context-files/README.md
rename to docker-context-files/.README.md
diff --git a/docs/apache-airflow-providers-databricks/commits.rst b/docs/apache-airflow-providers-databricks/commits.rst
index c0489fa6da..4599f649c2 100644
--- a/docs/apache-airflow-providers-databricks/commits.rst
+++ b/docs/apache-airflow-providers-databricks/commits.rst
@@ -31,11 +31,12 @@ For high-level changelog, see :doc:`package information including changelog <ind
2.6.0
.....
-Latest change: 2022-04-10
+Latest change: 2022-04-13
================================================================================================= =========== ===============================================================================
Commit Committed Subject
================================================================================================= =========== ===============================================================================
+`40831144be <https://github.com/apache/airflow/commit/40831144bedd3e652d8856b918a26d2e0a8e8e02>`_ 2022-04-13 ``Prepare for RC2 release of March Databricks provider (#22979)``
`7be57eb256 <https://github.com/apache/airflow/commit/7be57eb2566651de89048798766f0ad5f267cdc2>`_ 2022-04-10 ``Databricks SQL operators are now Python 3.10 compatible (#22886)``
`aa8c08db38 <https://github.com/apache/airflow/commit/aa8c08db383ebfabf30a7c2b2debb64c0968df48>`_ 2022-04-10 ``Databricks: Correctly handle HTTP exception (#22885)``
`6933022e94 <https://github.com/apache/airflow/commit/6933022e94acf139b2dea9a589bb8b25c62a5d20>`_ 2022-04-10 ``Fix new MyPy errors in main (#22884)``
diff --git a/docs/apache-airflow-providers-hashicorp/commits.rst b/docs/apache-airflow-providers-hashicorp/commits.rst
index 09958d86ca..de0e35981b 100644
--- a/docs/apache-airflow-providers-hashicorp/commits.rst
+++ b/docs/apache-airflow-providers-hashicorp/commits.rst
@@ -31,11 +31,12 @@ For high-level changelog, see :doc:`package information including changelog <ind
2.2.0
.....
-Latest change: 2022-03-24
+Latest change: 2022-04-07
================================================================================================= =========== ==================================================================================
Commit Committed Subject
================================================================================================= =========== ==================================================================================
+`56ab82ed7a <https://github.com/apache/airflow/commit/56ab82ed7a5c179d024722ccc697b740b2b93b6a>`_ 2022-04-07 ``Prepare mid-April provider documentation. (#22819)``
`7ab45d41d6 <https://github.com/apache/airflow/commit/7ab45d41d6c4de322dc8afe8a74b712d0bae4ee7>`_ 2022-03-24 ``Update secrets backends to use get_conn_value instead of get_conn_uri (#22348)``
================================================================================================= =========== ==================================================================================
diff --git a/docs/apache-airflow/installation/supported-versions.rst b/docs/apache-airflow/installation/supported-versions.rst
index 35f41cff02..93617ee8aa 100644
--- a/docs/apache-airflow/installation/supported-versions.rst
+++ b/docs/apache-airflow/installation/supported-versions.rst
@@ -23,7 +23,7 @@ Version Life Cycle
Apache Airflow version life cycle:
- .. This table is automatically updated by pre-commit scripts/ci/pre-commit/supported_versions.py
+ .. This table is automatically updated by pre-commit scripts/ci/pre_commit/pre_commit_supported_versions.py
.. Beginning of auto-generated table
========= ===================== ========= =============== ================= ================
diff --git a/docs/docker-stack/build-arg-ref.rst b/docs/docker-stack/build-arg-ref.rst
index a462179588..ea4c8a7d6c 100644
--- a/docs/docker-stack/build-arg-ref.rst
+++ b/docs/docker-stack/build-arg-ref.rst
@@ -70,7 +70,6 @@ Those are the most common arguments that you use when you want to build a custom
List of default extras in the production Dockerfile:
.. BEGINNING OF EXTRAS LIST UPDATED BY PRE COMMIT
-
* amazon
* async
* celery
@@ -97,7 +96,6 @@ List of default extras in the production Dockerfile:
* ssh
* statsd
* virtualenv
-
.. END OF EXTRAS LIST UPDATED BY PRE COMMIT
Image optimization options
diff --git a/docs/docker-stack/build.rst b/docs/docker-stack/build.rst
index acd043bdb7..9679172a2b 100644
--- a/docs/docker-stack/build.rst
+++ b/docs/docker-stack/build.rst
@@ -501,7 +501,7 @@ You can use ``docker-context-files`` for the following purposes:
* you can place ``.whl`` packages that you downloaded and install them with
- ``INSTALL_DOCKER_CONTEXT_FILES`` set to ``true`` . It's useful if you build the image in
+ ``INSTALL_FROM_DOCKER_CONTEXT_FILES`` set to ``true`` . It's useful if you build the image in
restricted security environments (see: :ref:`image-build-secure-environments` for details):
.. exampleinclude:: docker-examples/restricted/restricted_environments.sh
diff --git a/images/breeze/output-breeze-commands.svg b/images/breeze/output-breeze-commands.svg
deleted file mode 100644
index 5f8cceec28..0000000000
--- a/images/breeze/output-breeze-commands.svg
+++ /dev/null
@@ -1,136 +0,0 @@
-<svg width="1484.0" height="626" viewBox="0 0 1484.0 626"
- xmlns="http://www.w3.org/2000/svg">
- <style>
- @font-face {
- font-family: "Fira Code";
- src: local("FiraCode-Regular"),
- url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff2/FiraCode-Regular.woff2") format("woff2"),
- url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff/FiraCode-Regular.woff") format("woff");
- font-style: normal;
- font-weight: 400;
- }
- @font-face {
- font-family: "Fira Code";
- src: local("FiraCode-Bold"),
- url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff2/FiraCode-Bold.woff2") format("woff2"),
- url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff/FiraCode-Bold.woff") format("woff");
- font-style: bold;
- font-weight: 700;
- }
- span {
- display: inline-block;
- white-space: pre;
- vertical-align: top;
- font-size: 18px;
- font-family:'Fira Code','Cascadia Code',Monaco,Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace;
- }
- a {
- text-decoration: none;
- color: inherit;
- }
- .blink {
- animation: blinker 1s infinite;
- }
- @keyframes blinker {
- from { opacity: 1.0; }
- 50% { opacity: 0.3; }
- to { opacity: 1.0; }
- }
- #wrapper {
- padding: 140px;
- padding-top: 100px;
- }
- #terminal {
- position: relative;
- display: flex;
- flex-direction: column;
- align-items: center;
- background-color: #0c0c0c;
- border-radius: 14px;
- outline: 1px solid #484848;
- }
- #terminal:after {
- position: absolute;
- width: 100%;
- height: 100%;
- content: '';
- border-radius: 14px;
- background: rgb(71,77,102);
- background: linear-gradient(90deg, #804D69 0%, #4E4B89 100%);
- transform: rotate(-4.5deg);
- z-index: -1;
- }
- #terminal-header {
- position: relative;
- width: 100%;
- background-color: #2e2e2e;
- margin-bottom: 12px;
- font-weight: bold;
- border-radius: 14px 14px 0 0;
- color: #f2f2f2;
- font-size: 18px;
- box-shadow: inset 0px -1px 0px 0px #4e4e4e,
- inset 0px -4px 8px 0px #1a1a1a;
- }
- #terminal-title-tab {
- display: inline-block;
- margin-top: 14px;
- margin-left: 124px;
- font-family: sans-serif;
- padding: 14px 28px;
- border-radius: 6px 6px 0 0;
- background-color: #0c0c0c;
- box-shadow: inset 0px 1px 0px 0px #4e4e4e,
- 0px -4px 4px 0px #1e1e1e,
- inset 1px 0px 0px 0px #4e4e4e,
- inset -1px 0px 0px 0px #4e4e4e;
- }
- #terminal-traffic-lights {
- position: absolute;
- top: 24px;
- left: 20px;
- }
- #terminal-body {
- line-height: 22px;
- padding: 14px;
- }
- .r1 {color: #f2f2f2; text-decoration-color: #f2f2f2;background-color: #0c0c0c;}
-.r2 {font-weight: bold;color: #f2f2f2; text-decoration-color: #f2f2f2;;background-color: #0c0c0c;}
-.r3 {color: #e5e510; text-decoration-color: #e5e510; font-weight: bold;background-color: #0c0c0c;}
-.r4 {color: #7f7f7f; text-decoration-color: #7f7f7f;color: #f2f2f2; text-decoration-color: #f2f2f2;;background-color: #0c0c0c;}
-.r5 {color: #11a8cd; text-decoration-color: #11a8cd; font-weight: bold;background-color: #0c0c0c;}
-.r6 {color: #0dbc79; text-decoration-color: #0dbc79; font-weight: bold;background-color: #0c0c0c;}
-.r7 {color: #78780e; text-decoration-color: #78780e;background-color: #0c0c0c;}
- </style>
- <foreignObject x="0" y="0" width="100%" height="100%">
- <body xmlns="http://www.w3.org/1999/xhtml">
- <div id="wrapper">
- <div id="terminal">
- <div id='terminal-header'>
- <svg id="terminal-traffic-lights" width="90" height="21" viewBox="0 0 90 21" xmlns="http://www.w3.org/2000/svg">
- <circle cx="14" cy="8" r="8" fill="#ff6159"/>
- <circle cx="38" cy="8" r="8" fill="#ffbd2e"/>
- <circle cx="62" cy="8" r="8" fill="#28c941"/>
- </svg>
- <div id="terminal-title-tab">Breeze screenshot</div>
- </div>
- <div id='terminal-body'>
- <div><span class="r2"> </span></div>
-<div><span class="r2"> </span><span class="r3">Usage: </span><span class="r2">breeze cleanup [OPTIONS] </span></div>
-<div><span class="r2"> </span></div>
-<div><span class="r1"> </span><span class="r1">Removes the cache of parameters, docker cache and optionally - currently downloaded images. </span><span class="r1"> </span></div>
-<div><span class="r1"> </span></div>
-<div><span class="r4">╭─ Options ────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--also-remove-current-images</span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r1">Also removes currently downloaded Breeze images. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y|n|q)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
-<div><span class="r1"></span><span class="r1"> </span></div>
- </div>
- </div>
- </div>
- </body>
- </foreignObject>
-</svg>
diff --git a/images/breeze/output-build-docs.svg b/images/breeze/output-build-docs.svg
index 085615ef40..572747d685 100644
--- a/images/breeze/output-build-docs.svg
+++ b/images/breeze/output-build-docs.svg
@@ -1,4 +1,4 @@
-<svg width="1720.0" height="1330" viewBox="0 0 1720.0 1330"
+<svg width="1720.0" height="1550" viewBox="0 0 1720.0 1550"
xmlns="http://www.w3.org/2000/svg">
<style>
@font-face {
@@ -124,40 +124,50 @@
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--docs-only</span><span class="r1"> </span><span class="r6">-d</span><span class="r1"> </span><span class="r1">Only build documentation. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--spellcheck-only</span><span class="r1"> </span><span class="r6">-s</span><span class="r1"> </span><span class="r1">Only run spell checking. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--package-filter</span><span class="r1"> </span><span class="r6">-p</span><span class="r1"> </span><span class="r1">List of packages to consider. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">(apache-airflow|apache-airflow-providers|apache-airflow-providers-airbyte|apache-airflow-pr</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">oviders-alibaba|apache-airflow-providers-amazon|apache-airflow-providers-apache-beam|apache</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">-airflow-providers-apache-cassandra|apache-airflow-providers-apache-drill|apache-airflow-pr</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">oviders-apache-druid|apache-airflow-providers-apache-hdfs|apache-airflow-providers-apache-h</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">ive|apache-airflow-providers-apache-kylin|apache-airflow-providers-apache-livy|apache-airfl</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">ow-providers-apache-pig|apache-airflow-providers-apache-pinot|apache-airflow-providers-apac</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">he-spark|apache-airflow-providers-apache-sqoop|apache-airflow-providers-arangodb|apache-air</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">flow-providers-asana|apache-airflow-providers-celery|apache-airflow-providers-cloudant|apac</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">he-airflow-providers-cncf-kubernetes|apache-airflow-providers-databricks|apache-airflow-pro</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">viders-datadog|apache-airflow-providers-dbt-cloud|apache-airflow-providers-dingding|apache-</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">airflow-providers-discord|apache-airflow-providers-docker|apache-airflow-providers-elastics</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">earch|apache-airflow-providers-exasol|apache-airflow-providers-facebook|apache-airflow-prov</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">iders-ftp|apache-airflow-providers-github|apache-airflow-providers-google|apache-airflow-pr</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">oviders-grpc|apache-airflow-providers-hashicorp|apache-airflow-providers-http|apache-airflo</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">w-providers-imap|apache-airflow-providers-influxdb|apache-airflow-providers-jdbc|apache-air</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">flow-providers-jenkins|apache-airflow-providers-jira|apache-airflow-providers-microsoft-azu</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">re|apache-airflow-providers-microsoft-mssql|apache-airflow-providers-microsoft-psrp|apache-</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">airflow-providers-microsoft-winrm|apache-airflow-providers-mongo|apache-airflow-providers-m</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">ysql|apache-airflow-providers-neo4j|apache-airflow-providers-odbc|apache-airflow-providers-</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">openfaas|apache-airflow-providers-opsgenie|apache-airflow-providers-oracle|apache-airflow-p</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">roviders-pagerduty|apache-airflow-providers-papermill|apache-airflow-providers-plexus|apach</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">e-airflow-providers-postgres|apache-airflow-providers-presto|apache-airflow-providers-qubol</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">e|apache-airflow-providers-redis|apache-airflow-providers-salesforce|apache-airflow-provide</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">rs-samba|apache-airflow-providers-segment|apache-airflow-providers-sendgrid|apache-airflow-</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">providers-sftp|apache-airflow-providers-singularity|apache-airflow-providers-slack|apache-a</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">irflow-providers-snowflake|apache-airflow-providers-sqlite|apache-airflow-providers-ssh|apa</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">che-airflow-providers-tableau|apache-airflow-providers-telegram|apache-airflow-providers-tr</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">ino|apache-airflow-providers-vertica|apache-airflow-providers-yandex|apache-airflow-provide</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">rs-zendesk|docker-stack|helm-chart) </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">(apache-airflow | apache-airflow-providers | apache-airflow-providers-airbyte | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-alibaba | apache-airflow-providers-amazon | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-apache-beam | apache-airflow-providers-apache-cassandra | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-apache-drill | apache-airflow-providers-apache-druid | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-apache-hdfs | apache-airflow-providers-apache-hive | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-apache-kylin | apache-airflow-providers-apache-livy | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-apache-pig | apache-airflow-providers-apache-pinot | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-apache-spark | apache-airflow-providers-apache-sqoop | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-arangodb | apache-airflow-providers-asana | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-celery | apache-airflow-providers-cloudant | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-cncf-kubernetes | apache-airflow-providers-databricks | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-datadog | apache-airflow-providers-dbt-cloud | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-dingding | apache-airflow-providers-discord | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-docker | apache-airflow-providers-elasticsearch | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-exasol | apache-airflow-providers-facebook | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-ftp | apache-airflow-providers-github | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-google | apache-airflow-providers-grpc | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-hashicorp | apache-airflow-providers-http | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-imap | apache-airflow-providers-influxdb | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-jdbc | apache-airflow-providers-jenkins | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-jira | apache-airflow-providers-microsoft-azure | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-microsoft-mssql | apache-airflow-providers-microsoft-psrp | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-microsoft-winrm | apache-airflow-providers-mongo | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-mysql | apache-airflow-providers-neo4j | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-odbc | apache-airflow-providers-openfaas | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-opsgenie | apache-airflow-providers-oracle | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-pagerduty | apache-airflow-providers-papermill | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-plexus | apache-airflow-providers-postgres | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-presto | apache-airflow-providers-qubole | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-redis | apache-airflow-providers-salesforce | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-samba | apache-airflow-providers-segment | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-sendgrid | apache-airflow-providers-sftp | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-singularity | apache-airflow-providers-slack | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-snowflake | apache-airflow-providers-sqlite | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-ssh | apache-airflow-providers-tableau | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-telegram | apache-airflow-providers-trino | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-vertica | apache-airflow-providers-yandex | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">apache-airflow-providers-zendesk | docker-stack | helm-chart) </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r4">╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--github-repository</span><span class="r1"> </span><span class="r6">-g</span><span class="r1"> </span><span class="r1">GitHub repository used to pull, push run images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r4">[default: apache/airflow]</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r1"></span><span class="r1"> </span></div>
</div>
diff --git a/images/breeze/output-build-image.svg b/images/breeze/output-build-image.svg
index da0346e317..d2693055b4 100644
--- a/images/breeze/output-build-image.svg
+++ b/images/breeze/output-build-image.svg
@@ -1,4 +1,4 @@
-<svg width="1720.0" height="1308" viewBox="0 0 1720.0 1308"
+<svg width="1720.0" height="1506" viewBox="0 0 1720.0 1506"
xmlns="http://www.w3.org/2000/svg">
<style>
@font-face {
@@ -118,33 +118,41 @@
<div><span class="r2"> </span></div>
<div><span class="r2"> </span><span class="r3">Usage: </span><span class="r2">breeze build-image [OPTIONS] </span></div>
<div><span class="r2"> </span></div>
-<div><span class="r1"> </span><span class="r1">Build CI image. </span><span class="r1"> </span></div>
+<div><span class="r1"> </span><span class="r1">Build CI image. Include building multiple images for all python versions (sequentially). </span><span class="r1"> </span></div>
<div><span class="r1"> </span></div>
<div><span class="r4">╭─ Basic usage ────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--python</span><span class="r1"> </span><span class="r6">-p</span><span class="r1"> </span><span class="r1">Python version to use.</span><span class="r1"> </span><span class="r7">(3.7|3.8|3.9|3.10)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--python</span><span class="r1"> </span><span class="r6">-p</span><span class="r1"> </span><span class="r1">Python major/minor version used in Airflow image for PROD/CI images. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">(3.7 | 3.8 | 3.9 | 3.10) </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--upgrade-to-newer-dependencies</span><span class="r1"> </span><span class="r6">-u</span><span class="r1"> </span><span class="r1">When other than "false", upgrade all PIP packages to latest.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--debian-version</span><span class="r1"> </span><span class="r6">-d</span><span class="r1"> </span><span class="r1">Debian version used for the image.</span><span class="r1"> </span><span class="r7">(buster|bullseye)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--image-tag</span><span class="r1"> </span><span class="r6">-t</span><span class="r1"> </span><span class="r1">Set tag for the image (additionally to default Airflow convention).</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--docker-cache</span><span class="r1"> </span><span class="r6">-c</span><span class="r1"> </span><span class="r1">Cache option for image used during the build.</span><span class="r1"> </span><span class="r7">(pulled|local|disabled)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--github-repository</span><span class="r1"> </span><span class="r6">-g</span><span class="r1"> </span><span class="r1">GitHub repository used to pull, push images. Default: apache/airflow.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r4">[default: false] </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--debian-version</span><span class="r1"> </span><span class="r6">-d</span><span class="r1"> </span><span class="r1">Debian version used for the image.</span><span class="r1"> </span><span class="r7">(bullseye | buster)</span><span class="r1"> </span><span class="r4">[default: bullseye]</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--image-tag</span><span class="r1"> </span><span class="r6">-t</span><span class="r1"> </span><span class="r1">Tag added to the default naming conventions of Airflow CI/PROD images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--docker-cache</span><span class="r1"> </span><span class="r6">-c</span><span class="r1"> </span><span class="r1">Cache option for image used during the build.</span><span class="r1"> </span><span class="r7">(pulled | local | disabled)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r4">[default: pulled] </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--force-build</span><span class="r1"> </span><span class="r6">-f</span><span class="r1"> </span><span class="r1">Force image build no matter if it is determined as needed. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
+<div><span class="r4">╭─ Building multiple images ───────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--build_multiple_images</span><span class="r1"> </span><span class="r1">Run the operation sequentially on all or selected subset of Python versions. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--python-versions</span><span class="r1"> </span><span class="r1">Space separated list of python versions used for build with multiple versions.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r4">[default: 3.7 3.8 3.9 3.10] </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r4">╭─ Advanced options (for power users) ─────────────────────────────────────────────────────────────────────────────────╮</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--install-providers-from-sources</span><span class="r1"> </span><span class="r1">Install providers from sources when installing. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-extras</span><span class="r1"> </span><span class="r1">Additional extra package while installing Airflow in the image.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-dev-apt-deps</span><span class="r1"> </span><span class="r1">Additional apt dev dependencies to use when building the images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-runtime-apt-deps</span><span class="r1"> </span><span class="r1">Additional apt runtime dependencies to use when building the images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-python-deps</span><span class="r1"> </span><span class="r1">Additional python dependencies to use when building the images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-dev-apt-command</span><span class="r1"> </span><span class="r1">Additional command executed before dev apt deps are installed.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--runtime-apt-deps</span><span class="r1"> </span><span class="r1">Apt runtime dependencies to use when building the images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--runtime-apt-command</span><span class="r1"> </span><span class="r1">Command executed before runtime apt deps are installed.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-dev-apt-env</span><span class="r1"> </span><span class="r1">Additional environment variables set when adding dev dependencies.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-extras</span><span class="r1"> </span><span class="r1">Additional extra package while installing Airflow in the image.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-runtime-apt-deps</span><span class="r1"> </span><span class="r1">Additional apt runtime dependencies to use when building the images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-runtime-apt-env</span><span class="r1"> </span><span class="r1">Additional environment variables set when adding runtime dependencies.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-runtime-apt-command</span><span class="r1"> </span><span class="r1">Additional command executed before runtime apt deps are installed.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dev-apt-command</span><span class="r1"> </span><span class="r1">Command executed before dev apt deps are installed.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-dev-apt-deps</span><span class="r1"> </span><span class="r1">Additional apt dev dependencies to use when building the images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-dev-apt-env</span><span class="r1"> </span><span class="r1">Additional environment variables set when adding dev dependencies.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-dev-apt-command</span><span class="r1"> </span><span class="r1">Additional command executed before dev apt deps are installed.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dev-apt-deps</span><span class="r1"> </span><span class="r1">Apt dev dependencies to use when building the images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--runtime-apt-deps</span><span class="r1"> </span><span class="r1">Apt runtime dependencies to use when building the images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dev-apt-command</span><span class="r1"> </span><span class="r1">Command executed before dev apt deps are installed.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r4">╭─ Preparing cache and push (for maintainers and CI) ──────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--platform</span><span class="r1"> </span><span class="r1">Platform for Airflow image.</span><span class="r1"> </span><span class="r7">(linux/amd64|linux/arm64|linux/amd64,linux/arm64)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--platform</span><span class="r1"> </span><span class="r1">Platform for Airflow image.</span><span class="r1"> </span><span class="r7">(linux/amd64 | linux/arm64 | linux/amd64,linux/arm64)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--prepare-buildx-cache</span><span class="r1"> </span><span class="r1">Prepares build cache rather than build images locally. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--push-image</span><span class="r1"> </span><span class="r1">Push image after building it. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--empty-image</span><span class="r1"> </span><span class="r1">Prepare empty image tagged with the same name as the Airflow image. </span><span class="r1"> </span><span class="r4">│</span></div>
@@ -153,10 +161,11 @@
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--login-to-github-registry</span><span class="r1"> </span><span class="r1">Logs in to GitHub registry.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r4">╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y|n|q|yes|no|quit)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--github-repository</span><span class="r1"> </span><span class="r6">-g</span><span class="r1"> </span><span class="r1">GitHub repository used to pull, push run images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r4">[default: apache/airflow]</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y | n | q | yes | no | quit)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r1"></span><span class="r1"> </span></div>
</div>
diff --git a/images/breeze/output-build-prod-image.svg b/images/breeze/output-build-prod-image.svg
index edede13ac1..1a0471b2af 100644
--- a/images/breeze/output-build-prod-image.svg
+++ b/images/breeze/output-build-prod-image.svg
@@ -1,4 +1,4 @@
-<svg width="1720.0" height="1594" viewBox="0 0 1720.0 1594"
+<svg width="1720.0" height="1836" viewBox="0 0 1720.0 1836"
xmlns="http://www.w3.org/2000/svg">
<style>
@font-face {
@@ -118,58 +118,69 @@
<div><span class="r2"> </span></div>
<div><span class="r2"> </span><span class="r3">Usage: </span><span class="r2">breeze build-prod-image [OPTIONS] </span></div>
<div><span class="r2"> </span></div>
-<div><span class="r1"> </span><span class="r1">Build Production image. </span><span class="r1"> </span></div>
+<div><span class="r1"> </span><span class="r1">Build Production image. Include building multiple images for all or selected Python versions sequentially. </span><span class="r1"> </span></div>
<div><span class="r1"> </span></div>
<div><span class="r4">╭─ Basic usage ────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--python</span><span class="r1"> </span><span class="r6">-p</span><span class="r1"> </span><span class="r1">Python version to use.</span><span class="r1"> </span><span class="r7">(3.7|3.8|3.9|3.10)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--python</span><span class="r1"> </span><span class="r6">-p</span><span class="r1"> </span><span class="r1">Python major/minor version used in Airflow image for PROD/CI images. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">(3.7 | 3.8 | 3.9 | 3.10) </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--install-airflow-version</span><span class="r1"> </span><span class="r6">-V</span><span class="r1"> </span><span class="r1">Install version of Airflow from PyPI.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--upgrade-to-newer-dependencies</span><span class="r1"> </span><span class="r6">-u</span><span class="r1"> </span><span class="r1">When other than "false", upgrade all PIP packages to latest.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--debian-version</span><span class="r1"> </span><span class="r6">-d</span><span class="r1"> </span><span class="r1">Debian version used for the image.</span><span class="r1"> </span><span class="r7">(buster|bullseye)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--image-tag</span><span class="r1"> </span><span class="r6">-t</span><span class="r1"> </span><span class="r1">Set tag for the image (additionally to default Airflow convention).</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--docker-cache</span><span class="r1"> </span><span class="r6">-c</span><span class="r1"> </span><span class="r1">Cache option for image used during the build.</span><span class="r1"> </span><span class="r7">(pulled|local|disabled)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--github-repository</span><span class="r1"> </span><span class="r6">-g</span><span class="r1"> </span><span class="r1">GitHub repository used to pull, push images. Default: apache/airflow.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r4">[default: false] </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--debian-version</span><span class="r1"> </span><span class="r6">-d</span><span class="r1"> </span><span class="r1">Debian version used for the image.</span><span class="r1"> </span><span class="r7">(bullseye | buster)</span><span class="r1"> </span><span class="r4">[default: bullseye]</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--image-tag</span><span class="r1"> </span><span class="r6">-t</span><span class="r1"> </span><span class="r1">Tag added to the default naming conventions of Airflow CI/PROD images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--docker-cache</span><span class="r1"> </span><span class="r6">-c</span><span class="r1"> </span><span class="r1">Cache option for image used during the build.</span><span class="r1"> </span><span class="r7">(pulled | local | disabled)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r4">[default: pulled] </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
+<div><span class="r4">╭─ Building multiple images ───────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--build_multiple_images</span><span class="r1"> </span><span class="r1">Run the operation sequentially on all or selected subset of Python versions. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--python-versions</span><span class="r1"> </span><span class="r1">Space separated list of python versions used for build with multiple versions.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r4">[default: 3.7 3.8 3.9 3.10] </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r4">╭─ Options for customizing images ─────────────────────────────────────────────────────────────────────────────────────╮</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--install-providers-from-sources</span><span class="r1"> </span><span class="r1">Install providers from sources when installing. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--extras</span><span class="r1"> </span><span class="r1">Extras to install by default.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-python-deps</span><span class="r1"> </span><span class="r1">Additional python dependencies to use when building the images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-extras</span><span class="r1"> </span><span class="r1">Additional extra package while installing Airflow in the image.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-dev-apt-deps</span><span class="r1"> </span><span class="r1">Additional apt dev dependencies to use when building the images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-runtime-apt-deps</span><span class="r1"> </span><span class="r1">Additional apt runtime dependencies to use when building the images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-python-deps</span><span class="r1"> </span><span class="r1">Additional python dependencies to use when building the images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-dev-apt-command</span><span class="r1"> </span><span class="r1">Additional command executed before dev apt deps are installed.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--runtime-apt-command</span><span class="r1"> </span><span class="r1">Command executed before runtime apt deps are installed.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-dev-apt-env</span><span class="r1"> </span><span class="r1">Additional environment variables set when adding dev dependencies.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-runtime-apt-env</span><span class="r1"> </span><span class="r1">Additional environment variables set when adding runtime dependencies.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-runtime-apt-command</span><span class="r1"> </span><span class="r1">Additional command executed before runtime apt deps are installed.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dev-apt-command</span><span class="r1"> </span><span class="r1">Command executed before dev apt deps are installed.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dev-apt-deps</span><span class="r1"> </span><span class="r1">Apt dev dependencies to use when building the images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-dev-apt-deps</span><span class="r1"> </span><span class="r1">Additional apt dev dependencies to use when building the images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-dev-apt-env</span><span class="r1"> </span><span class="r1">Additional environment variables set when adding dev dependencies.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-dev-apt-command</span><span class="r1"> </span><span class="r1">Additional command executed before dev apt deps are installed.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--extras</span><span class="r1"> </span><span class="r1">Extras to install by default. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">(TEXT) </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r4">[default: </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r4">amazon,async,celery,cncf.kubernetes,dask,docker,elasticsearch,ftp,google,goog…</span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--runtime-apt-deps</span><span class="r1"> </span><span class="r1">Apt runtime dependencies to use when building the images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--runtime-apt-command</span><span class="r1"> </span><span class="r1">Command executed before runtime apt deps are installed.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dev-apt-deps</span><span class="r1"> </span><span class="r1">Apt dev dependencies to use when building the images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dev-apt-command</span><span class="r1"> </span><span class="r1">Command executed before dev apt deps are installed.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r4">╭─ Customization options (for specific customization needs) ───────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--install-from-docker-context-files</span><span class="r1"> </span><span class="r1">Install wheels from local docker-context-files when building image. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--cleanup-docker-context-files</span><span class="r1"> </span><span class="r1">Clean up docker context files before running build. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--install-from-docker-context-files</span><span class="r1"> </span><span class="r1">Install wheels from local docker-context-files when building image. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--disable-mysql-client-installation</span><span class="r1"> </span><span class="r1">Do not install MySQL client. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--disable-mssql-client-installation</span><span class="r1"> </span><span class="r1">Do not install MsSQl client. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--disable-postgres-client-installation</span><span class="r1"> </span><span class="r1">Do not install Postgres client. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--disable-airflow-repo-cache</span><span class="r1"> </span><span class="r1">Disable cache from Airflow repository during building. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--disable-pypi</span><span class="r1"> </span><span class="r1">Disable PyPI during building. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--install-airflow-reference</span><span class="r1"> </span><span class="r1">Install Airflow using GitHub tag or branch.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--installation-method</span><span class="r1"> </span><span class="r1">Install Airflow from: sources or PyPI.</span><span class="r1"> </span><span class="r7">(.|apache-airflow)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--installation-method</span><span class="r1"> </span><span class="r1">Install Airflow from: sources or PyPI.</span><span class="r1"> </span><span class="r7">(. | apache-airflow)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r4">╭─ Preparing cache and push (for maintainers and CI) ──────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--platform</span><span class="r1"> </span><span class="r1">Platform for Airflow image.</span><span class="r1"> </span><span class="r7">(linux/amd64|linux/arm64|linux/amd64,linux/arm64)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--prepare-buildx-cache</span><span class="r1"> </span><span class="r1">Prepares build cache rather than build images locally. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--push-image</span><span class="r1"> </span><span class="r1">Push image after building it. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--empty-image</span><span class="r1"> </span><span class="r1">Prepare empty image tagged with the same name as the Airflow image. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--github-token</span><span class="r1"> </span><span class="r1">The token used to authenticate to GitHub.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--github-username</span><span class="r1"> </span><span class="r1">The user name used to authenticate to GitHub.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--login-to-github-registry</span><span class="r1"> </span><span class="r1">Logs in to GitHub registry.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--push-image</span><span class="r1"> </span><span class="r1">Push image after building it. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--prepare-buildx-cache</span><span class="r1"> </span><span class="r1">Prepares build cache rather than build images locally. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--platform</span><span class="r1"> </span><span class="r1">Platform for Airflow image.</span><span class="r1"> </span><span class="r7">(linux/amd64 | linux/arm64 | linux/amd64,linux/arm64)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--empty-image</span><span class="r1"> </span><span class="r1">Prepare empty image tagged with the same name as the Airflow image. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r4">╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y|n|q|yes|no|quit)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--github-repository</span><span class="r1"> </span><span class="r6">-g</span><span class="r1"> </span><span class="r1">GitHub repository used to pull, push run images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r4">[default: apache/airflow]</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y | n | q | yes | no | quit)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r1"></span><span class="r1"> </span></div>
</div>
diff --git a/images/breeze/output-cleanup.svg b/images/breeze/output-cleanup.svg
index 4da7923766..4d4c22e8d2 100644
--- a/images/breeze/output-cleanup.svg
+++ b/images/breeze/output-cleanup.svg
@@ -1,4 +1,4 @@
-<svg width="1720.0" height="626" viewBox="0 0 1720.0 626"
+<svg width="1720.0" height="670" viewBox="0 0 1720.0 670"
xmlns="http://www.w3.org/2000/svg">
<style>
@font-face {
@@ -118,14 +118,16 @@
<div><span class="r2"> </span></div>
<div><span class="r2"> </span><span class="r3">Usage: </span><span class="r2">breeze cleanup [OPTIONS] </span></div>
<div><span class="r2"> </span></div>
-<div><span class="r1"> </span><span class="r1">the cache of parameters, docker cache and optionally - currently downloaded images. </span><span class="r1"> </span></div>
+<div><span class="r1"> </span><span class="r1">Cleans the cache of parameters, docker cache and optionally - currently downloaded images. </span><span class="r1"> </span></div>
<div><span class="r1"> </span></div>
+<div><span class="r4">╭─ Cleanup flags ──────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--all</span><span class="r1"> </span><span class="r1">Also remove currently downloaded Breeze images. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r4">╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--also-remove-current-images</span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r1">Also remove currently downloaded Breeze images. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y|n|q|yes|no|quit)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y | n | q | yes | no | quit)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r1"></span><span class="r1"> </span></div>
</div>
diff --git a/images/breeze/output-commands.svg b/images/breeze/output-commands.svg
index cd83f79e46..4ef31a5f6f 100644
--- a/images/breeze/output-commands.svg
+++ b/images/breeze/output-commands.svg
@@ -1,4 +1,4 @@
-<svg width="1720.0" height="1264" viewBox="0 0 1720.0 1264"
+<svg width="1720.0" height="1726" viewBox="0 0 1720.0 1726"
xmlns="http://www.w3.org/2000/svg">
<style>
@font-face {
@@ -119,42 +119,63 @@
<div><span class="r2"> </span><span class="r3">Usage: </span><span class="r2">breeze [OPTIONS] COMMAND [ARGS]... </span></div>
<div><span class="r2"> </span></div>
<div><span class="r4">╭─ Basic flags for the default (shell) command ────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--python</span><span class="r1"> </span><span class="r6">-p</span><span class="r1"> </span><span class="r1">Python version to use.</span><span class="r1"> </span><span class="r7">(3.7|3.8|3.9|3.10)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--backend</span><span class="r1"> </span><span class="r6">-b</span><span class="r1"> </span><span class="r1">Database backend to use.</span><span class="r1"> </span><span class="r7">(sqlite|mysql|postgres|mssql)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--python</span><span class="r1"> </span><span class="r6">-p</span><span class="r1"> </span><span class="r1">Python major/minor version used in Airflow image for PROD/CI images. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">(3.7 | 3.8 | 3.9 | 3.10) </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--backend</span><span class="r1"> </span><span class="r6">-b</span><span class="r1"> </span><span class="r1">Database backend to use.</span><span class="r1"> </span><span class="r7">(sqlite | mysql | postgres | mssql)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--use-airflow-version</span><span class="r1"> </span><span class="r6">-V</span><span class="r1"> </span><span class="r1">Use (reinstall at entry) Airflow version from PyPI.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--postgres-version</span><span class="r1"> </span><span class="r6">-P</span><span class="r1"> </span><span class="r1">Version of Postgres.</span><span class="r1"> </span><span class="r7">(10|11|12|13)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--mysql-version</span><span class="r1"> </span><span class="r6">-M</span><span class="r1"> </span><span class="r1">Version of MySQL.</span><span class="r1"> </span><span class="r7">(5.7|8)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--mssql-version</span><span class="r1"> </span><span class="r6">-S</span><span class="r1"> </span><span class="r1">Version of MsSQL.</span><span class="r1"> </span><span class="r7">(2017-latest|2019-latest)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--postgres-version</span><span class="r1"> </span><span class="r6">-P</span><span class="r1"> </span><span class="r1">Version of Postgres.</span><span class="r1"> </span><span class="r7">(10 | 11 | 12 | 13)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--mysql-version</span><span class="r1"> </span><span class="r6">-M</span><span class="r1"> </span><span class="r1">Version of MySQL.</span><span class="r1"> </span><span class="r7">(5.7 | 8)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--mssql-version</span><span class="r1"> </span><span class="r6">-S</span><span class="r1"> </span><span class="r1">Version of MsSQL.</span><span class="r1"> </span><span class="r7">(2017-latest | 2019-latest)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--forward-credentials</span><span class="r1"> </span><span class="r6">-f</span><span class="r1"> </span><span class="r1">Forward local credentials to container when running. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--db-reset</span><span class="r1"> </span><span class="r6">-d</span><span class="r1"> </span><span class="r1">Reset DB when entering the container. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r4">╭─ Advanced flags for the default (shell) command ─────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--force-build</span><span class="r1"> </span><span class="r1">Force image build before running. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--mount-sources</span><span class="r1"> </span><span class="r1">Choose scope of local sources should be mounted (default = selected).</span><span class="r1"> </span><span class="r7">(selected|all|none)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--integration</span><span class="r1"> </span><span class="r1">Integration(s) to enable when running (can be more than one). </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">(cassandra|kerberos|mongo|openldap|pinot|rabbitmq|redis|statsd|trino|all)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--force-build</span><span class="r1"> </span><span class="r6">-f</span><span class="r1"> </span><span class="r1">Force image build no matter if it is determined as needed. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--mount-sources</span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r1">Choose scope of local sources should be mounted (default = selected).</span><span class="r1"> </span><span class="r7">(selected | all | none)</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r4">[default: selected] </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--integration</span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r1">Integration(s) to enable when running (can be more than one). </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">(cassandra | kerberos | mongo | openldap | pinot | rabbitmq | redis | statsd | trino | all)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r4">╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y|n|q|yes|no|quit)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--github-repository</span><span class="r1"> </span><span class="r6">-g</span><span class="r1"> </span><span class="r1">GitHub repository used to pull, push run images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r4">[default: apache/airflow]</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y | n | q | yes | no | quit)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r4">╭─ Developer tools ────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> shell </span><span class="r1"> </span><span class="r1">Enter breeze.py environment. this is the default command use when no other is selected.</span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> start-airflow </span><span class="r1"> </span><span class="r1">Enter breeze.py environment and starts all Airflow components in the tmux session. </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> stop </span><span class="r1"> </span><span class="r1">Stop running breeze environment. </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> build-image </span><span class="r1"> </span><span class="r1">Build CI image. </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> build-prod-image </span><span class="r1"> </span><span class="r1">Build Production image. </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> build-docs </span><span class="r1"> </span><span class="r1">Build documentation in the container. </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> static-checks </span><span class="r1"> </span><span class="r1">Run static checks. </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> shell </span><span class="r1"> </span><span class="r1">Enter breeze.py environment. this is the default command use when no other is selected.</span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> start-airflow </span><span class="r1"> </span><span class="r1">Enter breeze.py environment and starts all Airflow components in the tmux session. </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> stop </span><span class="r1"> </span><span class="r1">Stop running breeze environment. </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> build-docs </span><span class="r1"> </span><span class="r1">Build documentation in the container. </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> static-checks </span><span class="r1"> </span><span class="r1">Run static checks. </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
+<div><span class="r4">╭─ Testing ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> docker-compose-tests </span><span class="r1"> </span><span class="r1">Run docker-compose tests.</span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r4">╭─ Configuration & maintenance ────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> cleanup </span><span class="r1"> </span><span class="r1">the cache of parameters, docker cache and optionally - currently downloaded images.</span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> self-upgrade </span><span class="r1"> </span><span class="r1">Self upgrade Breeze. </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> setup-autocomplete </span><span class="r1"> </span><span class="r1">Enables autocompletion of breeze commands. </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> config </span><span class="r1"> </span><span class="r1">Show/update configuration (Python, Backend, Cheatsheet, ASCIIART). </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> version </span><span class="r1"> </span><span class="r1">Prints version of breeze.py. </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> cleanup </span><span class="r1"> </span><span class="r1">Cleans the cache of parameters, docker cache and optionally - currently downloaded images.</span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> self-upgrade </span><span class="r1"> </span><span class="r1">Self upgrade Breeze. </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> setup-autocomplete </span><span class="r1"> </span><span class="r1">Enables autocompletion of breeze commands. </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> config </span><span class="r1"> </span><span class="r1">Show/update configuration (Python, Backend, Cheatsheet, ASCIIART). </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> version </span><span class="r1"> </span><span class="r1">Print information about version of apache-airflow-breeze. </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
+<div><span class="r4">╭─ CI Image tools ─────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> build-image </span><span class="r1"> </span><span class="r1">Build CI image. Include building multiple images for all python versions (sequentially).</span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> pull-image </span><span class="r1"> </span><span class="r1">Pull and optionally verify CI images - possibly in parallel for all Python versions. </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> verify-image </span><span class="r1"> </span><span class="r1">Verify CI image. </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
+<div><span class="r4">╭─ Production Image tools ─────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> build-prod-image </span><span class="r1"> </span><span class="r1">Build Production image. Include building multiple images for all or selected Python versions </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> </span><span class="r1"> </span><span class="r1">sequentially. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> pull-prod-image </span><span class="r1"> </span><span class="r1">Pull and optionally verify Production images - possibly in parallel for all Python versions. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> verify-prod-image </span><span class="r1"> </span><span class="r1">Verify Production image. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
+<div><span class="r4">╭─ Release management ─────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> prepare-provider-documentation </span><span class="r1"> </span><span class="r1">Prepare CHANGELOG, README and COMMITS information for providers. </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> prepare-provider-packages </span><span class="r1"> </span><span class="r1">Prepare sdist/whl packages of Airflow Providers. </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> prepare-airflow-package </span><span class="r1"> </span><span class="r1">Prepare sdist/whl package of Airflow. </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> generate-constraints </span><span class="r1"> </span><span class="r1">Generates pinned constraint files with all extras from setup.py in parallel.</span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r1"></span><span class="r1"> </span></div>
</div>
diff --git a/images/breeze/output-config.svg b/images/breeze/output-config.svg
index d44a208826..92624fa654 100644
--- a/images/breeze/output-config.svg
+++ b/images/breeze/output-config.svg
@@ -1,4 +1,4 @@
-<svg width="1720.0" height="670" viewBox="0 0 1720.0 670"
+<svg width="1720.0" height="692" viewBox="0 0 1720.0 692"
xmlns="http://www.w3.org/2000/svg">
<style>
@font-face {
@@ -121,8 +121,9 @@
<div><span class="r1"> </span><span class="r1">Show/update configuration (Python, Backend, Cheatsheet, ASCIIART). </span><span class="r1"> </span></div>
<div><span class="r1"> </span></div>
<div><span class="r4">╭─ Config flags ───────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--python</span><span class="r1"> </span><span class="r6">-p</span><span class="r1"> </span><span class="r1">Python version to use.</span><span class="r1"> </span><span class="r7">(3.7|3.8|3.9|3.10)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--backend</span><span class="r1"> </span><span class="r6">-b</span><span class="r1"> </span><span class="r1">Database backend to use.</span><span class="r1"> </span><span class="r7">(sqlite|mysql|postgres|mssql)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--python</span><span class="r1"> </span><span class="r6">-p</span><span class="r1"> </span><span class="r1">Python major/minor version used in Airflow image for PROD/CI images. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">(3.7 | 3.8 | 3.9 | 3.10) </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--backend</span><span class="r1"> </span><span class="r6">-b</span><span class="r1"> </span><span class="r1">Database backend to use.</span><span class="r1"> </span><span class="r7">(sqlite | mysql | postgres | mssql)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--cheatsheet</span><span class="r1">/</span><span class="r5">--no-cheatsheet</span><span class="r1"> </span><span class="r6">-C</span><span class="r1">/</span><span class="r6">-c</span><span class="r1"> </span><span class="r1">Enable/disable cheatsheet. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--asciiart</span><span class="r1">/</span><span class="r5">--no-asciiart</span><span class="r1"> </span><span class="r6">-A</span><span class="r1">/</span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Enable/disable ASCIIart. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
diff --git a/images/breeze/output-setup-autocomplete.svg b/images/breeze/output-docker-compose-tests.svg
similarity index 73%
copy from images/breeze/output-setup-autocomplete.svg
copy to images/breeze/output-docker-compose-tests.svg
index ab424f70c7..7dc3510792 100644
--- a/images/breeze/output-setup-autocomplete.svg
+++ b/images/breeze/output-docker-compose-tests.svg
@@ -1,4 +1,4 @@
-<svg width="1720.0" height="670" viewBox="0 0 1720.0 670"
+<svg width="1720.0" height="714" viewBox="0 0 1720.0 714"
xmlns="http://www.w3.org/2000/svg">
<style>
@font-face {
@@ -112,22 +112,24 @@
<circle cx="38" cy="8" r="8" fill="#ffbd2e"/>
<circle cx="62" cy="8" r="8" fill="#28c941"/>
</svg>
- <div id="terminal-title-tab">Command: setup-autocomplete</div>
+ <div id="terminal-title-tab">Command: docker-compose-tests</div>
</div>
<div id='terminal-body'>
<div><span class="r2"> </span></div>
-<div><span class="r2"> </span><span class="r3">Usage: </span><span class="r2">breeze setup-autocomplete [OPTIONS] </span></div>
+<div><span class="r2"> </span><span class="r3">Usage: </span><span class="r2">breeze docker-compose-tests [OPTIONS] [EXTRA_PYTEST_ARGS]... </span></div>
<div><span class="r2"> </span></div>
-<div><span class="r1"> </span><span class="r1">Enables autocompletion of breeze commands. </span><span class="r1"> </span></div>
+<div><span class="r1"> </span><span class="r1">Run docker-compose tests. </span><span class="r1"> </span></div>
<div><span class="r1"> </span></div>
-<div><span class="r4">╭─ Setup autocomplete flags ───────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--force</span><span class="r1"> </span><span class="r6">-f</span><span class="r1"> </span><span class="r1">Force autocomplete setup even if already setup before (overrides the setup). </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╭─ Docker-compose tests flag ──────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--image-name</span><span class="r1"> </span><span class="r6">-n</span><span class="r1"> Name of the image to verify (overrides </span><span class="r5">--python</span><span class="r1"> and </span><span class="r5">--image-tag</span><span class="r1">). </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--python</span><span class="r1"> </span><span class="r6">-p</span><span class="r1"> </span><span class="r1">Python major/minor version used in Airflow image for PROD/CI images.</span><span class="r1"> </span><span class="r7">(3.7 | 3.8 | 3.9 | 3.10)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--image-tag</span><span class="r1"> </span><span class="r6">-t</span><span class="r1"> </span><span class="r1">Tag added to the default naming conventions of Airflow CI/PROD images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r4">╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y|n|q|yes|no|quit)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--github-repository</span><span class="r1"> </span><span class="r6">-g</span><span class="r1"> </span><span class="r1">GitHub repository used to pull, push run images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r4">[default: apache/airflow]</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r1"></span><span class="r1"> </span></div>
</div>
diff --git a/images/breeze/output-shell.svg b/images/breeze/output-generate-constraints.svg
similarity index 64%
copy from images/breeze/output-shell.svg
copy to images/breeze/output-generate-constraints.svg
index 571abe63cf..7f92500124 100644
--- a/images/breeze/output-shell.svg
+++ b/images/breeze/output-generate-constraints.svg
@@ -1,4 +1,4 @@
-<svg width="1720.0" height="956" viewBox="0 0 1720.0 956"
+<svg width="1720.0" height="934" viewBox="0 0 1720.0 934"
xmlns="http://www.w3.org/2000/svg">
<style>
@font-face {
@@ -112,35 +112,34 @@
<circle cx="38" cy="8" r="8" fill="#ffbd2e"/>
<circle cx="62" cy="8" r="8" fill="#28c941"/>
</svg>
- <div id="terminal-title-tab">Command: shell</div>
+ <div id="terminal-title-tab">Command: generate-constraints</div>
</div>
<div id='terminal-body'>
<div><span class="r2"> </span></div>
-<div><span class="r2"> </span><span class="r3">Usage: </span><span class="r2">breeze shell [OPTIONS] [EXTRA_ARGS]... </span></div>
+<div><span class="r2"> </span><span class="r3">Usage: </span><span class="r2">breeze generate-constraints [OPTIONS] </span></div>
<div><span class="r2"> </span></div>
-<div><span class="r1"> </span><span class="r1">Enter breeze.py environment. this is the default command use when no other is selected. </span><span class="r1"> </span></div>
+<div><span class="r1"> </span><span class="r1">Generates pinned constraint files with all extras from setup.py in parallel. </span><span class="r1"> </span></div>
<div><span class="r1"> </span></div>
-<div><span class="r4">╭─ Basic flags ────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--python</span><span class="r1"> </span><span class="r6">-p</span><span class="r1"> </span><span class="r1">Python version to use.</span><span class="r1"> </span><span class="r7">(3.7|3.8|3.9|3.10)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--backend</span><span class="r1"> </span><span class="r6">-b</span><span class="r1"> </span><span class="r1">Database backend to use.</span><span class="r1"> </span><span class="r7">(sqlite|mysql|postgres|mssql)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--use-airflow-version</span><span class="r1"> </span><span class="r6">-V</span><span class="r1"> </span><span class="r1">Use (reinstall at entry) Airflow version from PyPI.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--postgres-version</span><span class="r1"> </span><span class="r6">-P</span><span class="r1"> </span><span class="r1">Version of Postgres.</span><span class="r1"> </span><span class="r7">(10|11|12|13)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--mysql-version</span><span class="r1"> </span><span class="r6">-M</span><span class="r1"> </span><span class="r1">Version of MySQL.</span><span class="r1"> </span><span class="r7">(5.7|8)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--mssql-version</span><span class="r1"> </span><span class="r6">-S</span><span class="r1"> </span><span class="r1">Version of MsSQL.</span><span class="r1"> </span><span class="r7">(2017-latest|2019-latest)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--forward-credentials</span><span class="r1"> </span><span class="r6">-f</span><span class="r1"> </span><span class="r1">Forward local credentials to container when running. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--db-reset</span><span class="r1"> </span><span class="r6">-d</span><span class="r1"> </span><span class="r1">Reset DB when entering the container. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╭─ Generate constraints flags ─────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--image-tag</span><span class="r1"> </span><span class="r6">-t</span><span class="r1"> </span><span class="r1">Tag added to the default naming conventions of Airflow CI/PROD images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--python</span><span class="r1"> </span><span class="r6">-p</span><span class="r1"> </span><span class="r1">Python major/minor version used in Airflow image for PROD/CI images. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">(3.7 | 3.8 | 3.9 | 3.10) </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--generate-constraints-mode</span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r1">Mode of generating constraints</span><span class="r1"> </span><span class="r7">(source-providers | pypi-providers | no-providers)</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r4">[default: source-providers] </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
-<div><span class="r4">╭─ Advanced flag for running ──────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--force-build</span><span class="r1"> </span><span class="r1">Force image build before running. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--mount-sources</span><span class="r1"> </span><span class="r1">Choose scope of local sources should be mounted (default = selected).</span><span class="r1"> </span><span class="r7">(selected|all|none)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--integration</span><span class="r1"> </span><span class="r1">Integration(s) to enable when running (can be more than one). </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">(cassandra|kerberos|mongo|openldap|pinot|rabbitmq|redis|statsd|trino|all)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╭─ Parallel running ───────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--run-in-parallel</span><span class="r1"> </span><span class="r1">Run the operation in parallel on all or selected subset of Python versions. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--parallelism</span><span class="r1"> </span><span class="r1">Maximum number of processes to use while running the operation in parallel.</span><span class="r1"> </span><span class="r7">(INTEGER RANGE)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r4">[default: 4; 1<=x<=8] </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--python-versions</span><span class="r1"> </span><span class="r1">Space separated list of python versions used for build with multiple versions.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r4">[default: 3.7 3.8 3.9 3.10] </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r4">╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y|n|q|yes|no|quit)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--github-repository</span><span class="r1"> </span><span class="r6">-g</span><span class="r1"> </span><span class="r1">GitHub repository used to pull, push run images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r4">[default: apache/airflow]</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y | n | q | yes | no | quit)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r1"></span><span class="r1"> </span></div>
</div>
diff --git a/images/breeze/output-setup-autocomplete.svg b/images/breeze/output-prepare-airflow-package.svg
similarity index 76%
copy from images/breeze/output-setup-autocomplete.svg
copy to images/breeze/output-prepare-airflow-package.svg
index ab424f70c7..b720df7f61 100644
--- a/images/breeze/output-setup-autocomplete.svg
+++ b/images/breeze/output-prepare-airflow-package.svg
@@ -1,4 +1,4 @@
-<svg width="1720.0" height="670" viewBox="0 0 1720.0 670"
+<svg width="1720.0" height="692" viewBox="0 0 1720.0 692"
xmlns="http://www.w3.org/2000/svg">
<style>
@font-face {
@@ -99,8 +99,8 @@
.r3 {color: #e5e510; text-decoration-color: #e5e510; font-weight: bold;background-color: #0c0c0c;}
.r4 {color: #7f7f7f; text-decoration-color: #7f7f7f;color: #f2f2f2; text-decoration-color: #f2f2f2;;background-color: #0c0c0c;}
.r5 {color: #11a8cd; text-decoration-color: #11a8cd; font-weight: bold;background-color: #0c0c0c;}
-.r6 {color: #0dbc79; text-decoration-color: #0dbc79; font-weight: bold;background-color: #0c0c0c;}
-.r7 {color: #78780e; text-decoration-color: #78780e;background-color: #0c0c0c;}
+.r6 {color: #78780e; text-decoration-color: #78780e;background-color: #0c0c0c;}
+.r7 {color: #0dbc79; text-decoration-color: #0dbc79; font-weight: bold;background-color: #0c0c0c;}
</style>
<foreignObject x="0" y="0" width="100%" height="100%">
<body xmlns="http://www.w3.org/1999/xhtml">
@@ -112,22 +112,23 @@
<circle cx="38" cy="8" r="8" fill="#ffbd2e"/>
<circle cx="62" cy="8" r="8" fill="#28c941"/>
</svg>
- <div id="terminal-title-tab">Command: setup-autocomplete</div>
+ <div id="terminal-title-tab">Command: prepare-airflow-package</div>
</div>
<div id='terminal-body'>
<div><span class="r2"> </span></div>
-<div><span class="r2"> </span><span class="r3">Usage: </span><span class="r2">breeze setup-autocomplete [OPTIONS] </span></div>
+<div><span class="r2"> </span><span class="r3">Usage: </span><span class="r2">breeze prepare-airflow-package [OPTIONS] </span></div>
<div><span class="r2"> </span></div>
-<div><span class="r1"> </span><span class="r1">Enables autocompletion of breeze commands. </span><span class="r1"> </span></div>
+<div><span class="r1"> </span><span class="r1">Prepare sdist/whl package of Airflow. </span><span class="r1"> </span></div>
<div><span class="r1"> </span></div>
-<div><span class="r4">╭─ Setup autocomplete flags ───────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--force</span><span class="r1"> </span><span class="r6">-f</span><span class="r1"> </span><span class="r1">Force autocomplete setup even if already setup before (overrides the setup). </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╭─ Package flags ──────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--package-format</span><span class="r1"> </span><span class="r1">Format of packages.</span><span class="r1"> </span><span class="r6">(wheel | sdist | both)</span><span class="r1"> </span><span class="r4">[default: wheel]</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--version-suffix-for-pypi</span><span class="r1"> </span><span class="r1">Version suffix used for PyPI packages (alpha, beta, rc1, etc.).</span><span class="r1"> </span><span class="r6">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r4">╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y|n|q|yes|no|quit)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r7">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r7">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--github-repository</span><span class="r1"> </span><span class="r7">-g</span><span class="r1"> </span><span class="r1">GitHub repository used to pull, push run images.</span><span class="r1"> </span><span class="r6">(TEXT)</span><span class="r1"> </span><span class="r4">[default: apache/airflow]</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r7">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r1"></span><span class="r1"> </span></div>
</div>
diff --git a/images/breeze/output-setup-autocomplete.svg b/images/breeze/output-prepare-provider-documentation.svg
similarity index 65%
copy from images/breeze/output-setup-autocomplete.svg
copy to images/breeze/output-prepare-provider-documentation.svg
index ab424f70c7..3eb46c8701 100644
--- a/images/breeze/output-setup-autocomplete.svg
+++ b/images/breeze/output-prepare-provider-documentation.svg
@@ -1,4 +1,4 @@
-<svg width="1720.0" height="670" viewBox="0 0 1720.0 670"
+<svg width="1720.0" height="934" viewBox="0 0 1720.0 934"
xmlns="http://www.w3.org/2000/svg">
<style>
@font-face {
@@ -112,22 +112,34 @@
<circle cx="38" cy="8" r="8" fill="#ffbd2e"/>
<circle cx="62" cy="8" r="8" fill="#28c941"/>
</svg>
- <div id="terminal-title-tab">Command: setup-autocomplete</div>
+ <div id="terminal-title-tab">Command: prepare-provider-documentation</div>
</div>
<div id='terminal-body'>
<div><span class="r2"> </span></div>
-<div><span class="r2"> </span><span class="r3">Usage: </span><span class="r2">breeze setup-autocomplete [OPTIONS] </span></div>
+<div><span class="r2"> </span><span class="r3">Usage: </span><span class="r2">breeze prepare-provider-documentation [OPTIONS] [airbyte | alibaba | amazon | apache.beam | apache.cassandra | </span></div>
+<div><span class="r2"> apache.drill | apache.druid | apache.hdfs | apache.hive | apache.kylin | </span></div>
+<div><span class="r2"> apache.livy | apache.pig | apache.pinot | apache.spark | apache.sqoop | </span></div>
+<div><span class="r2"> arangodb | asana | celery | cloudant | cncf.kubernetes | databricks | </span></div>
+<div><span class="r2"> datadog | dbt.cloud | dingding | discord | docker | elasticsearch | </span></div>
+<div><span class="r2"> exasol | facebook | ftp | github | google | grpc | hashicorp | http | </span></div>
+<div><span class="r2"> imap | influxdb | jdbc | jenkins | jira | microsoft.azure | </span></div>
+<div><span class="r2"> microsoft.mssql | microsoft.psrp | microsoft.winrm | mongo | mysql | </span></div>
+<div><span class="r2"> neo4j | odbc | openfaas | opsgenie | oracle | pagerduty | papermill | </span></div>
+<div><span class="r2"> plexus | postgres | presto | qubole | redis | salesforce | samba | </span></div>
+<div><span class="r2"> segment | sendgrid | sftp | singularity | slack | snowflake | sqlite | </span></div>
+<div><span class="r2"> ssh | tableau | telegram | trino | vertica | yandex | zendesk]... </span></div>
<div><span class="r2"> </span></div>
-<div><span class="r1"> </span><span class="r1">Enables autocompletion of breeze commands. </span><span class="r1"> </span></div>
+<div><span class="r1"> </span><span class="r1">Prepare CHANGELOG, README and COMMITS information for providers. </span><span class="r1"> </span></div>
<div><span class="r1"> </span></div>
-<div><span class="r4">╭─ Setup autocomplete flags ───────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--force</span><span class="r1"> </span><span class="r6">-f</span><span class="r1"> </span><span class="r1">Force autocomplete setup even if already setup before (overrides the setup). </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╭─ Provider documentation preparation flags ───────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--skip-package-verification</span><span class="r1"> </span><span class="r1">Skip Provider package verification. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r4">╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y|n|q|yes|no|quit)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--github-repository</span><span class="r1"> </span><span class="r6">-g</span><span class="r1"> </span><span class="r1">GitHub repository used to pull, push run images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r4">[default: apache/airflow]</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y | n | q | yes | no | quit)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r1"></span><span class="r1"> </span></div>
</div>
diff --git a/images/breeze/output-setup-autocomplete.svg b/images/breeze/output-prepare-provider-packages.svg
similarity index 62%
copy from images/breeze/output-setup-autocomplete.svg
copy to images/breeze/output-prepare-provider-packages.svg
index ab424f70c7..cd81f86397 100644
--- a/images/breeze/output-setup-autocomplete.svg
+++ b/images/breeze/output-prepare-provider-packages.svg
@@ -1,4 +1,4 @@
-<svg width="1720.0" height="670" viewBox="0 0 1720.0 670"
+<svg width="1720.0" height="956" viewBox="0 0 1720.0 956"
xmlns="http://www.w3.org/2000/svg">
<style>
@font-face {
@@ -99,8 +99,8 @@
.r3 {color: #e5e510; text-decoration-color: #e5e510; font-weight: bold;background-color: #0c0c0c;}
.r4 {color: #7f7f7f; text-decoration-color: #7f7f7f;color: #f2f2f2; text-decoration-color: #f2f2f2;;background-color: #0c0c0c;}
.r5 {color: #11a8cd; text-decoration-color: #11a8cd; font-weight: bold;background-color: #0c0c0c;}
-.r6 {color: #0dbc79; text-decoration-color: #0dbc79; font-weight: bold;background-color: #0c0c0c;}
-.r7 {color: #78780e; text-decoration-color: #78780e;background-color: #0c0c0c;}
+.r6 {color: #78780e; text-decoration-color: #78780e;background-color: #0c0c0c;}
+.r7 {color: #0dbc79; text-decoration-color: #0dbc79; font-weight: bold;background-color: #0c0c0c;}
</style>
<foreignObject x="0" y="0" width="100%" height="100%">
<body xmlns="http://www.w3.org/1999/xhtml">
@@ -112,22 +112,35 @@
<circle cx="38" cy="8" r="8" fill="#ffbd2e"/>
<circle cx="62" cy="8" r="8" fill="#28c941"/>
</svg>
- <div id="terminal-title-tab">Command: setup-autocomplete</div>
+ <div id="terminal-title-tab">Command: prepare-provider-packages</div>
</div>
<div id='terminal-body'>
<div><span class="r2"> </span></div>
-<div><span class="r2"> </span><span class="r3">Usage: </span><span class="r2">breeze setup-autocomplete [OPTIONS] </span></div>
+<div><span class="r2"> </span><span class="r3">Usage: </span><span class="r2">breeze prepare-provider-packages [OPTIONS] [airbyte | alibaba | amazon | apache.beam | apache.cassandra | </span></div>
+<div><span class="r2"> apache.drill | apache.druid | apache.hdfs | apache.hive | apache.kylin | </span></div>
+<div><span class="r2"> apache.livy | apache.pig | apache.pinot | apache.spark | apache.sqoop | </span></div>
+<div><span class="r2"> arangodb | asana | celery | cloudant | cncf.kubernetes | databricks | datadog </span></div>
+<div><span class="r2"> | dbt.cloud | dingding | discord | docker | elasticsearch | exasol | facebook </span></div>
+<div><span class="r2"> | ftp | github | google | grpc | hashicorp | http | imap | influxdb | jdbc | </span></div>
+<div><span class="r2"> jenkins | jira | microsoft.azure | microsoft.mssql | microsoft.psrp | </span></div>
+<div><span class="r2"> microsoft.winrm | mongo | mysql | neo4j | odbc | openfaas | opsgenie | oracle </span></div>
+<div><span class="r2"> | pagerduty | papermill | plexus | postgres | presto | qubole | redis | </span></div>
+<div><span class="r2"> salesforce | samba | segment | sendgrid | sftp | singularity | slack | </span></div>
+<div><span class="r2"> snowflake | sqlite | ssh | tableau | telegram | trino | vertica | yandex | </span></div>
+<div><span class="r2"> zendesk]... </span></div>
<div><span class="r2"> </span></div>
-<div><span class="r1"> </span><span class="r1">Enables autocompletion of breeze commands. </span><span class="r1"> </span></div>
+<div><span class="r1"> </span><span class="r1">Prepare sdist/whl packages of Airflow Providers. </span><span class="r1"> </span></div>
<div><span class="r1"> </span></div>
-<div><span class="r4">╭─ Setup autocomplete flags ───────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--force</span><span class="r1"> </span><span class="r6">-f</span><span class="r1"> </span><span class="r1">Force autocomplete setup even if already setup before (overrides the setup). </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╭─ Package flags ──────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--package-format</span><span class="r1"> </span><span class="r1">Format of packages.</span><span class="r1"> </span><span class="r6">(wheel | sdist | both)</span><span class="r1"> </span><span class="r4">[default: wheel]</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--version-suffix-for-pypi</span><span class="r1"> </span><span class="r1">Version suffix used for PyPI packages (alpha, beta, rc1, etc.).</span><span class="r1"> </span><span class="r6">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--package-list-file</span><span class="r1"> </span><span class="r1">Read list of packages from text file (one package per line)</span><span class="r1"> </span><span class="r6">(FILENAME)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r4">╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y|n|q|yes|no|quit)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r7">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r7">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--github-repository</span><span class="r1"> </span><span class="r7">-g</span><span class="r1"> </span><span class="r1">GitHub repository used to pull, push run images.</span><span class="r1"> </span><span class="r6">(TEXT)</span><span class="r1"> </span><span class="r4">[default: apache/airflow]</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r7">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r1"></span><span class="r1"> </span></div>
</div>
diff --git a/images/breeze/output-shell.svg b/images/breeze/output-pull-image.svg
similarity index 64%
copy from images/breeze/output-shell.svg
copy to images/breeze/output-pull-image.svg
index 571abe63cf..e77af30370 100644
--- a/images/breeze/output-shell.svg
+++ b/images/breeze/output-pull-image.svg
@@ -1,4 +1,4 @@
-<svg width="1720.0" height="956" viewBox="0 0 1720.0 956"
+<svg width="1720.0" height="934" viewBox="0 0 1720.0 934"
xmlns="http://www.w3.org/2000/svg">
<style>
@font-face {
@@ -112,35 +112,34 @@
<circle cx="38" cy="8" r="8" fill="#ffbd2e"/>
<circle cx="62" cy="8" r="8" fill="#28c941"/>
</svg>
- <div id="terminal-title-tab">Command: shell</div>
+ <div id="terminal-title-tab">Command: pull-image</div>
</div>
<div id='terminal-body'>
<div><span class="r2"> </span></div>
-<div><span class="r2"> </span><span class="r3">Usage: </span><span class="r2">breeze shell [OPTIONS] [EXTRA_ARGS]... </span></div>
+<div><span class="r2"> </span><span class="r3">Usage: </span><span class="r2">breeze pull-image [OPTIONS] [EXTRA_PYTEST_ARGS]... </span></div>
<div><span class="r2"> </span></div>
-<div><span class="r1"> </span><span class="r1">Enter breeze.py environment. this is the default command use when no other is selected. </span><span class="r1"> </span></div>
+<div><span class="r1"> </span><span class="r1">Pull and optionally verify CI images - possibly in parallel for all Python versions. </span><span class="r1"> </span></div>
<div><span class="r1"> </span></div>
-<div><span class="r4">╭─ Basic flags ────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--python</span><span class="r1"> </span><span class="r6">-p</span><span class="r1"> </span><span class="r1">Python version to use.</span><span class="r1"> </span><span class="r7">(3.7|3.8|3.9|3.10)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--backend</span><span class="r1"> </span><span class="r6">-b</span><span class="r1"> </span><span class="r1">Database backend to use.</span><span class="r1"> </span><span class="r7">(sqlite|mysql|postgres|mssql)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--use-airflow-version</span><span class="r1"> </span><span class="r6">-V</span><span class="r1"> </span><span class="r1">Use (reinstall at entry) Airflow version from PyPI.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--postgres-version</span><span class="r1"> </span><span class="r6">-P</span><span class="r1"> </span><span class="r1">Version of Postgres.</span><span class="r1"> </span><span class="r7">(10|11|12|13)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--mysql-version</span><span class="r1"> </span><span class="r6">-M</span><span class="r1"> </span><span class="r1">Version of MySQL.</span><span class="r1"> </span><span class="r7">(5.7|8)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--mssql-version</span><span class="r1"> </span><span class="r6">-S</span><span class="r1"> </span><span class="r1">Version of MsSQL.</span><span class="r1"> </span><span class="r7">(2017-latest|2019-latest)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--forward-credentials</span><span class="r1"> </span><span class="r6">-f</span><span class="r1"> </span><span class="r1">Forward local credentials to container when running. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--db-reset</span><span class="r1"> </span><span class="r6">-d</span><span class="r1"> </span><span class="r1">Reset DB when entering the container. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╭─ Pull image flags ───────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--image-tag</span><span class="r1"> </span><span class="r6">-t</span><span class="r1"> </span><span class="r1">Tag added to the default naming conventions of Airflow CI/PROD images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--python</span><span class="r1"> </span><span class="r6">-p</span><span class="r1"> </span><span class="r1">Python major/minor version used in Airflow image for PROD/CI images. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">(3.7 | 3.8 | 3.9 | 3.10) </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verify-image</span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r1">Verify image. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--wait-for-image</span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r1">Wait until image is available. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--tag-as-latest</span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r1">Tags the image as latest after pulling. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
-<div><span class="r4">╭─ Advanced flag for running ──────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--force-build</span><span class="r1"> </span><span class="r1">Force image build before running. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--mount-sources</span><span class="r1"> </span><span class="r1">Choose scope of local sources should be mounted (default = selected).</span><span class="r1"> </span><span class="r7">(selected|all|none)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--integration</span><span class="r1"> </span><span class="r1">Integration(s) to enable when running (can be more than one). </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">(cassandra|kerberos|mongo|openldap|pinot|rabbitmq|redis|statsd|trino|all)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╭─ Parallel running ───────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--run-in-parallel</span><span class="r1"> </span><span class="r1">Run the operation in parallel on all or selected subset of Python versions. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--parallelism</span><span class="r1"> </span><span class="r1">Maximum number of processes to use while running the operation in parallel.</span><span class="r1"> </span><span class="r7">(INTEGER RANGE)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r4">[default: 4; 1<=x<=8] </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--python-versions</span><span class="r1"> </span><span class="r1">Space separated list of python versions used for build with multiple versions.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r4">[default: 3.7 3.8 3.9 3.10] </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r4">╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y|n|q|yes|no|quit)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--github-repository</span><span class="r1"> </span><span class="r6">-g</span><span class="r1"> </span><span class="r1">GitHub repository used to pull, push run images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r4">[default: apache/airflow]</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r1"></span><span class="r1"> </span></div>
</div>
diff --git a/images/breeze/output-shell.svg b/images/breeze/output-pull-prod-image.svg
similarity index 64%
copy from images/breeze/output-shell.svg
copy to images/breeze/output-pull-prod-image.svg
index 571abe63cf..c432345694 100644
--- a/images/breeze/output-shell.svg
+++ b/images/breeze/output-pull-prod-image.svg
@@ -1,4 +1,4 @@
-<svg width="1720.0" height="956" viewBox="0 0 1720.0 956"
+<svg width="1720.0" height="934" viewBox="0 0 1720.0 934"
xmlns="http://www.w3.org/2000/svg">
<style>
@font-face {
@@ -112,35 +112,34 @@
<circle cx="38" cy="8" r="8" fill="#ffbd2e"/>
<circle cx="62" cy="8" r="8" fill="#28c941"/>
</svg>
- <div id="terminal-title-tab">Command: shell</div>
+ <div id="terminal-title-tab">Command: pull-prod-image</div>
</div>
<div id='terminal-body'>
<div><span class="r2"> </span></div>
-<div><span class="r2"> </span><span class="r3">Usage: </span><span class="r2">breeze shell [OPTIONS] [EXTRA_ARGS]... </span></div>
+<div><span class="r2"> </span><span class="r3">Usage: </span><span class="r2">breeze pull-prod-image [OPTIONS] [EXTRA_PYTEST_ARGS]... </span></div>
<div><span class="r2"> </span></div>
-<div><span class="r1"> </span><span class="r1">Enter breeze.py environment. this is the default command use when no other is selected. </span><span class="r1"> </span></div>
+<div><span class="r1"> </span><span class="r1">Pull and optionally verify Production images - possibly in parallel for all Python versions. </span><span class="r1"> </span></div>
<div><span class="r1"> </span></div>
-<div><span class="r4">╭─ Basic flags ────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--python</span><span class="r1"> </span><span class="r6">-p</span><span class="r1"> </span><span class="r1">Python version to use.</span><span class="r1"> </span><span class="r7">(3.7|3.8|3.9|3.10)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--backend</span><span class="r1"> </span><span class="r6">-b</span><span class="r1"> </span><span class="r1">Database backend to use.</span><span class="r1"> </span><span class="r7">(sqlite|mysql|postgres|mssql)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--use-airflow-version</span><span class="r1"> </span><span class="r6">-V</span><span class="r1"> </span><span class="r1">Use (reinstall at entry) Airflow version from PyPI.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--postgres-version</span><span class="r1"> </span><span class="r6">-P</span><span class="r1"> </span><span class="r1">Version of Postgres.</span><span class="r1"> </span><span class="r7">(10|11|12|13)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--mysql-version</span><span class="r1"> </span><span class="r6">-M</span><span class="r1"> </span><span class="r1">Version of MySQL.</span><span class="r1"> </span><span class="r7">(5.7|8)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--mssql-version</span><span class="r1"> </span><span class="r6">-S</span><span class="r1"> </span><span class="r1">Version of MsSQL.</span><span class="r1"> </span><span class="r7">(2017-latest|2019-latest)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--forward-credentials</span><span class="r1"> </span><span class="r6">-f</span><span class="r1"> </span><span class="r1">Forward local credentials to container when running. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--db-reset</span><span class="r1"> </span><span class="r6">-d</span><span class="r1"> </span><span class="r1">Reset DB when entering the container. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╭─ Pull image flags ───────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--image-tag</span><span class="r1"> </span><span class="r6">-t</span><span class="r1"> </span><span class="r1">Tag added to the default naming conventions of Airflow CI/PROD images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--python</span><span class="r1"> </span><span class="r6">-p</span><span class="r1"> </span><span class="r1">Python major/minor version used in Airflow image for PROD/CI images. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">(3.7 | 3.8 | 3.9 | 3.10) </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verify-image</span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r1">Verify image. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--wait-for-image</span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r1">Wait until image is available. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--tag-as-latest</span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r1">Tags the image as latest after pulling. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
-<div><span class="r4">╭─ Advanced flag for running ──────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--force-build</span><span class="r1"> </span><span class="r1">Force image build before running. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--mount-sources</span><span class="r1"> </span><span class="r1">Choose scope of local sources should be mounted (default = selected).</span><span class="r1"> </span><span class="r7">(selected|all|none)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--integration</span><span class="r1"> </span><span class="r1">Integration(s) to enable when running (can be more than one). </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">(cassandra|kerberos|mongo|openldap|pinot|rabbitmq|redis|statsd|trino|all)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╭─ Parallel running ───────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--run-in-parallel</span><span class="r1"> </span><span class="r1">Run the operation in parallel on all or selected subset of Python versions. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--parallelism</span><span class="r1"> </span><span class="r1">Maximum number of processes to use while running the operation in parallel.</span><span class="r1"> </span><span class="r7">(INTEGER RANGE)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r4">[default: 4; 1<=x<=8] </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--python-versions</span><span class="r1"> </span><span class="r1">Space separated list of python versions used for build with multiple versions.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r4">[default: 3.7 3.8 3.9 3.10] </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r4">╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y|n|q|yes|no|quit)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--github-repository</span><span class="r1"> </span><span class="r6">-g</span><span class="r1"> </span><span class="r1">GitHub repository used to pull, push run images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r4">[default: apache/airflow]</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r1"></span><span class="r1"> </span></div>
</div>
diff --git a/images/breeze/output-self-upgrade.svg b/images/breeze/output-self-upgrade.svg
index 2a49fab6fd..8020900b9e 100644
--- a/images/breeze/output-self-upgrade.svg
+++ b/images/breeze/output-self-upgrade.svg
@@ -1,4 +1,4 @@
-<svg width="1720.0" height="582" viewBox="0 0 1720.0 582"
+<svg width="1720.0" height="626" viewBox="0 0 1720.0 626"
xmlns="http://www.w3.org/2000/svg">
<style>
@font-face {
@@ -119,10 +119,12 @@
<div><span class="r2"> </span></div>
<div><span class="r1"> </span><span class="r1">Self upgrade Breeze. </span><span class="r1"> </span></div>
<div><span class="r1"> </span></div>
-<div><span class="r4">╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--use-current-airflow-sources</span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r1">Use current workdir Airflow sources for upgrade. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╭─ Self-upgrade flags ─────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--use-current-airflow-sources</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Use current workdir Airflow sources for upgrade. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--force</span><span class="r1"> </span><span class="r6">-f</span><span class="r1"> </span><span class="r1">Force upgrade without asking question to the user. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
+<div><span class="r4">╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r1"></span><span class="r1"> </span></div>
</div>
diff --git a/images/breeze/output-setup-autocomplete.svg b/images/breeze/output-setup-autocomplete.svg
index ab424f70c7..86e66e661c 100644
--- a/images/breeze/output-setup-autocomplete.svg
+++ b/images/breeze/output-setup-autocomplete.svg
@@ -124,7 +124,7 @@
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--force</span><span class="r1"> </span><span class="r6">-f</span><span class="r1"> </span><span class="r1">Force autocomplete setup even if already setup before (overrides the setup). </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r4">╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y|n|q|yes|no|quit)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y | n | q | yes | no | quit)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
diff --git a/images/breeze/output-shell.svg b/images/breeze/output-shell.svg
index 571abe63cf..51af37d741 100644
--- a/images/breeze/output-shell.svg
+++ b/images/breeze/output-shell.svg
@@ -1,4 +1,4 @@
-<svg width="1720.0" height="956" viewBox="0 0 1720.0 956"
+<svg width="1720.0" height="1022" viewBox="0 0 1720.0 1022"
xmlns="http://www.w3.org/2000/svg">
<style>
@font-face {
@@ -121,26 +121,29 @@
<div><span class="r1"> </span><span class="r1">Enter breeze.py environment. this is the default command use when no other is selected. </span><span class="r1"> </span></div>
<div><span class="r1"> </span></div>
<div><span class="r4">╭─ Basic flags ────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--python</span><span class="r1"> </span><span class="r6">-p</span><span class="r1"> </span><span class="r1">Python version to use.</span><span class="r1"> </span><span class="r7">(3.7|3.8|3.9|3.10)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--backend</span><span class="r1"> </span><span class="r6">-b</span><span class="r1"> </span><span class="r1">Database backend to use.</span><span class="r1"> </span><span class="r7">(sqlite|mysql|postgres|mssql)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--python</span><span class="r1"> </span><span class="r6">-p</span><span class="r1"> </span><span class="r1">Python major/minor version used in Airflow image for PROD/CI images. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">(3.7 | 3.8 | 3.9 | 3.10) </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--backend</span><span class="r1"> </span><span class="r6">-b</span><span class="r1"> </span><span class="r1">Database backend to use.</span><span class="r1"> </span><span class="r7">(sqlite | mysql | postgres | mssql)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--use-airflow-version</span><span class="r1"> </span><span class="r6">-V</span><span class="r1"> </span><span class="r1">Use (reinstall at entry) Airflow version from PyPI.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--postgres-version</span><span class="r1"> </span><span class="r6">-P</span><span class="r1"> </span><span class="r1">Version of Postgres.</span><span class="r1"> </span><span class="r7">(10|11|12|13)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--mysql-version</span><span class="r1"> </span><span class="r6">-M</span><span class="r1"> </span><span class="r1">Version of MySQL.</span><span class="r1"> </span><span class="r7">(5.7|8)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--mssql-version</span><span class="r1"> </span><span class="r6">-S</span><span class="r1"> </span><span class="r1">Version of MsSQL.</span><span class="r1"> </span><span class="r7">(2017-latest|2019-latest)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--postgres-version</span><span class="r1"> </span><span class="r6">-P</span><span class="r1"> </span><span class="r1">Version of Postgres.</span><span class="r1"> </span><span class="r7">(10 | 11 | 12 | 13)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--mysql-version</span><span class="r1"> </span><span class="r6">-M</span><span class="r1"> </span><span class="r1">Version of MySQL.</span><span class="r1"> </span><span class="r7">(5.7 | 8)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--mssql-version</span><span class="r1"> </span><span class="r6">-S</span><span class="r1"> </span><span class="r1">Version of MsSQL.</span><span class="r1"> </span><span class="r7">(2017-latest | 2019-latest)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--forward-credentials</span><span class="r1"> </span><span class="r6">-f</span><span class="r1"> </span><span class="r1">Forward local credentials to container when running. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--db-reset</span><span class="r1"> </span><span class="r6">-d</span><span class="r1"> </span><span class="r1">Reset DB when entering the container. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r4">╭─ Advanced flag for running ──────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--force-build</span><span class="r1"> </span><span class="r1">Force image build before running. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--mount-sources</span><span class="r1"> </span><span class="r1">Choose scope of local sources should be mounted (default = selected).</span><span class="r1"> </span><span class="r7">(selected|all|none)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--integration</span><span class="r1"> </span><span class="r1">Integration(s) to enable when running (can be more than one). </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">(cassandra|kerberos|mongo|openldap|pinot|rabbitmq|redis|statsd|trino|all)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--force-build</span><span class="r1"> </span><span class="r6">-f</span><span class="r1"> </span><span class="r1">Force image build no matter if it is determined as needed. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--mount-sources</span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r1">Choose scope of local sources should be mounted (default = selected).</span><span class="r1"> </span><span class="r7">(selected | all | none)</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r4">[default: selected] </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--integration</span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r1">Integration(s) to enable when running (can be more than one). </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">(cassandra | kerberos | mongo | openldap | pinot | rabbitmq | redis | statsd | trino | all)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r4">╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y|n|q|yes|no|quit)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--github-repository</span><span class="r1"> </span><span class="r6">-g</span><span class="r1"> </span><span class="r1">GitHub repository used to pull, push run images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r4">[default: apache/airflow]</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y | n | q | yes | no | quit)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r1"></span><span class="r1"> </span></div>
</div>
diff --git a/images/breeze/output-start-airflow.svg b/images/breeze/output-start-airflow.svg
index 353bbaa654..83101d09e4 100644
--- a/images/breeze/output-start-airflow.svg
+++ b/images/breeze/output-start-airflow.svg
@@ -1,4 +1,4 @@
-<svg width="1720.0" height="1000" viewBox="0 0 1720.0 1000"
+<svg width="1720.0" height="1066" viewBox="0 0 1720.0 1066"
xmlns="http://www.w3.org/2000/svg">
<style>
@font-face {
@@ -121,28 +121,31 @@
<div><span class="r1"> </span><span class="r1">Enter breeze.py environment and starts all Airflow components in the tmux session. </span><span class="r1"> </span></div>
<div><span class="r1"> </span></div>
<div><span class="r4">╭─ Basic flags ────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--python</span><span class="r1"> </span><span class="r6">-p</span><span class="r1"> </span><span class="r1">Python version to use.</span><span class="r1"> </span><span class="r7">(3.7|3.8|3.9|3.10)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--backend</span><span class="r1"> </span><span class="r6">-b</span><span class="r1"> </span><span class="r1">Database backend to use.</span><span class="r1"> </span><span class="r7">(sqlite|mysql|postgres|mssql)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--python</span><span class="r1"> </span><span class="r6">-p</span><span class="r1"> </span><span class="r1">Python major/minor version used in Airflow image for PROD/CI images. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">(3.7 | 3.8 | 3.9 | 3.10) </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--backend</span><span class="r1"> </span><span class="r6">-b</span><span class="r1"> </span><span class="r1">Database backend to use.</span><span class="r1"> </span><span class="r7">(sqlite | mysql | postgres | mssql)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--use-airflow-version</span><span class="r1"> </span><span class="r6">-V</span><span class="r1"> </span><span class="r1">Use (reinstall at entry) Airflow version from PyPI.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--postgres-version</span><span class="r1"> </span><span class="r6">-P</span><span class="r1"> </span><span class="r1">Version of Postgres.</span><span class="r1"> </span><span class="r7">(10|11|12|13)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--mysql-version</span><span class="r1"> </span><span class="r6">-M</span><span class="r1"> </span><span class="r1">Version of MySQL.</span><span class="r1"> </span><span class="r7">(5.7|8)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--mssql-version</span><span class="r1"> </span><span class="r6">-S</span><span class="r1"> </span><span class="r1">Version of MsSQL.</span><span class="r1"> </span><span class="r7">(2017-latest|2019-latest)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--postgres-version</span><span class="r1"> </span><span class="r6">-P</span><span class="r1"> </span><span class="r1">Version of Postgres.</span><span class="r1"> </span><span class="r7">(10 | 11 | 12 | 13)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--mysql-version</span><span class="r1"> </span><span class="r6">-M</span><span class="r1"> </span><span class="r1">Version of MySQL.</span><span class="r1"> </span><span class="r7">(5.7 | 8)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--mssql-version</span><span class="r1"> </span><span class="r6">-S</span><span class="r1"> </span><span class="r1">Version of MsSQL.</span><span class="r1"> </span><span class="r7">(2017-latest | 2019-latest)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--load-example-dags</span><span class="r1"> </span><span class="r6">-e</span><span class="r1"> </span><span class="r1">Enable configuration to load example DAGs when starting Airflow. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--load-default-connections</span><span class="r1"> </span><span class="r6">-c</span><span class="r1"> </span><span class="r1">Enable configuration to load default connections when starting Airflow. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--forward-credentials</span><span class="r1"> </span><span class="r6">-f</span><span class="r1"> </span><span class="r1">Forward local credentials to container when running. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--db-reset</span><span class="r1"> </span><span class="r6">-d</span><span class="r1"> </span><span class="r1">Reset DB when entering the container. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r4">╭─ Advanced flag for running ──────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--force-build</span><span class="r1"> </span><span class="r1">Force image build before running. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--mount-sources</span><span class="r1"> </span><span class="r1">Choose scope of local sources should be mounted (default = selected).</span><span class="r1"> </span><span class="r7">(selected|all|none)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--integration</span><span class="r1"> </span><span class="r1">Integration(s) to enable when running (can be more than one). </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">(cassandra|kerberos|mongo|openldap|pinot|rabbitmq|redis|statsd|trino|all)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--force-build</span><span class="r1"> </span><span class="r6">-f</span><span class="r1"> </span><span class="r1">Force image build no matter if it is determined as needed. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--mount-sources</span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r1">Choose scope of local sources should be mounted (default = selected).</span><span class="r1"> </span><span class="r7">(selected | all | none)</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r4">[default: selected] </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--integration</span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r1">Integration(s) to enable when running (can be more than one). </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">(cassandra | kerberos | mongo | openldap | pinot | rabbitmq | redis | statsd | trino | all)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r4">╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y|n|q|yes|no|quit)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--github-repository</span><span class="r1"> </span><span class="r6">-g</span><span class="r1"> </span><span class="r1">GitHub repository used to pull, push run images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r4">[default: apache/airflow]</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y | n | q | yes | no | quit)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r1"></span><span class="r1"> </span></div>
</div>
diff --git a/images/breeze/output-static-checks.svg b/images/breeze/output-static-checks.svg
index 1513cf7b55..e63056e358 100644
--- a/images/breeze/output-static-checks.svg
+++ b/images/breeze/output-static-checks.svg
@@ -1,4 +1,4 @@
-<svg width="1720.0" height="1132" viewBox="0 0 1720.0 1132"
+<svg width="1720.0" height="1242" viewBox="0 0 1720.0 1242"
xmlns="http://www.w3.org/2000/svg">
<style>
@font-face {
@@ -122,33 +122,38 @@
<div><span class="r1"> </span></div>
<div><span class="r4">╭─ Pre-commit flags ───────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--type</span><span class="r1"> </span><span class="r6">-t</span><span class="r1"> </span><span class="r1">Type(s) of the static checks to run (multiple can be added). </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">(airflow-config-yaml|airflow-providers-available|all|autoflake|base-operator|black|bla</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">cken-docs|boring-cyborg|build-providers-dependencies|changelog-duplicates|chart-schema</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">-lint|check-2-1-compatibility|check-apache-license|check-builtin-literals|check-execut</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">ables-have-shebangs|check-extras-order|check-hooks-apply|check-integrations|check-merg</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">e-conflict|check-revision-heads-map|check-system-tests|check-xml|codespell|daysago-imp</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">ort-check|debug-statements|detect-private-key|docstring-params|doctoc|dont-use-safe-fi</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">lter|end-of-file-fixer|fix-encoding-pragma|flake8|flynt|forbid-tabs|helm-lint|identity</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">|incorrect-use-of-LoggingMixin|inline-dockerfile-scripts|insert-license|isort|json-sch</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">ema|language-matters|limit-breeze-dependencies|lint-dockerfile|lint-openapi|markdownli</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">nt|migration-reference|mixed-line-ending|mypy|newsfragments|no-providers-in-core-examp</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">les|no-relative-imports|persist-credentials-disabled|pre-commit-descriptions|pre-commi</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">t-hook-names|pretty-format-json|provide-create-sessions|provider-yamls|providers-init-</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">file|providers-subpackages-init-file|pydevd|pydocstyle|python-no-log-warn|pyupgrade|re</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">strict-start_date|rst-backticks|setup-extra-packages|setup-order|shellcheck|sort-in-th</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">e-wild|sort-spelling-wordlist|stylelint|trailing-whitespace|ui-lint|update-breeze-conf</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">ig-hash|update-breeze-file|update-extras|update-local-yml-file|update-setup-cfg-file|u</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">pdate-supported-versions|update-version|vendor-k8s-json-schema|www-lint|yamllint|yesqa</span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r7">) </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">(airflow-config-yaml | airflow-providers-available | all | autoflake | base-operator |</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">black | blacken-docs | boring-cyborg | build-providers-dependencies | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">changelog-duplicates | chart-schema-lint | check-2-1-compatibility | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">check-apache-license | check-builtin-literals | check-executables-have-shebangs | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">check-extras-order | check-hooks-apply | check-integrations | check-merge-conflict | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">check-revision-heads-map | check-system-tests | check-xml | codespell | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">daysago-import-check | debug-statements | detect-private-key | docstring-params | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">doctoc | dont-use-safe-filter | end-of-file-fixer | fix-encoding-pragma | flake8 | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">flynt | forbid-tabs | helm-lint | identity | incorrect-use-of-LoggingMixin | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">inline-dockerfile-scripts | insert-license | isort | json-schema | language-matters | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">limit-breeze-dependencies | lint-dockerfile | lint-openapi | markdownlint | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">migration-reference | mixed-line-ending | mypy | newsfragments | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">no-providers-in-core-examples | no-relative-imports | persist-credentials-disabled | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">pre-commit-descriptions | pre-commit-hook-names | pretty-format-json | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">provide-create-sessions | provider-yamls | providers-init-file | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">providers-subpackages-init-file | pydevd | pydocstyle | python-no-log-warn | pyupgrade</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">| restrict-start_date | rst-backticks | setup-extra-packages | setup-order | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">shellcheck | sort-in-the-wild | sort-spelling-wordlist | stylelint | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">trailing-whitespace | ui-lint | update-breeze-config-hash | update-breeze-file | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">update-extras | update-local-yml-file | update-setup-cfg-file | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">update-supported-versions | update-version | vendor-k8s-json-schema | www-lint | </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">yamllint | yesqa) </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--files</span><span class="r1"> </span><span class="r6">-f</span><span class="r1"> </span><span class="r1">List of files to run the checks on.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--all-files</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Run checks on all files. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--show-diff-on-failure</span><span class="r1"> </span><span class="r6">-s</span><span class="r1"> </span><span class="r1">Show diff for files modified by the checks. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--last-commit</span><span class="r1"> </span><span class="r6">-c</span><span class="r1"> </span><span class="r1">Run checks for all files in last commit. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r4">╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--github-repository</span><span class="r1"> </span><span class="r6">-g</span><span class="r1"> </span><span class="r1">GitHub repository used to pull, push run images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r4">[default: apache/airflow]</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r1"></span><span class="r1"> </span></div>
</div>
diff --git a/images/breeze/output-setup-autocomplete.svg b/images/breeze/output-verify-image.svg
similarity index 74%
copy from images/breeze/output-setup-autocomplete.svg
copy to images/breeze/output-verify-image.svg
index ab424f70c7..1d8262869a 100644
--- a/images/breeze/output-setup-autocomplete.svg
+++ b/images/breeze/output-verify-image.svg
@@ -1,4 +1,4 @@
-<svg width="1720.0" height="670" viewBox="0 0 1720.0 670"
+<svg width="1720.0" height="714" viewBox="0 0 1720.0 714"
xmlns="http://www.w3.org/2000/svg">
<style>
@font-face {
@@ -112,22 +112,24 @@
<circle cx="38" cy="8" r="8" fill="#ffbd2e"/>
<circle cx="62" cy="8" r="8" fill="#28c941"/>
</svg>
- <div id="terminal-title-tab">Command: setup-autocomplete</div>
+ <div id="terminal-title-tab">Command: verify-image</div>
</div>
<div id='terminal-body'>
<div><span class="r2"> </span></div>
-<div><span class="r2"> </span><span class="r3">Usage: </span><span class="r2">breeze setup-autocomplete [OPTIONS] </span></div>
+<div><span class="r2"> </span><span class="r3">Usage: </span><span class="r2">breeze verify-image [OPTIONS] [EXTRA_PYTEST_ARGS]... </span></div>
<div><span class="r2"> </span></div>
-<div><span class="r1"> </span><span class="r1">Enables autocompletion of breeze commands. </span><span class="r1"> </span></div>
+<div><span class="r1"> </span><span class="r1">Verify CI image. </span><span class="r1"> </span></div>
<div><span class="r1"> </span></div>
-<div><span class="r4">╭─ Setup autocomplete flags ───────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--force</span><span class="r1"> </span><span class="r6">-f</span><span class="r1"> </span><span class="r1">Force autocomplete setup even if already setup before (overrides the setup). </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╭─ Verify image flags ─────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--image-name</span><span class="r1"> </span><span class="r6">-n</span><span class="r1"> Name of the image to verify (overrides </span><span class="r5">--python</span><span class="r1"> and </span><span class="r5">--image-tag</span><span class="r1">). </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--python</span><span class="r1"> </span><span class="r6">-p</span><span class="r1"> </span><span class="r1">Python major/minor version used in Airflow image for PROD/CI images.</span><span class="r1"> </span><span class="r7">(3.7 | 3.8 | 3.9 | 3.10)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--image-tag</span><span class="r1"> </span><span class="r6">-t</span><span class="r1"> </span><span class="r1">Tag added to the default naming conventions of Airflow CI/PROD images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r4">╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y|n|q|yes|no|quit)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--github-repository</span><span class="r1"> </span><span class="r6">-g</span><span class="r1"> </span><span class="r1">GitHub repository used to pull, push run images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r4">[default: apache/airflow]</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r1"></span><span class="r1"> </span></div>
</div>
diff --git a/images/breeze/output-setup-autocomplete.svg b/images/breeze/output-verify-prod-image.svg
similarity index 74%
copy from images/breeze/output-setup-autocomplete.svg
copy to images/breeze/output-verify-prod-image.svg
index ab424f70c7..e84c671b42 100644
--- a/images/breeze/output-setup-autocomplete.svg
+++ b/images/breeze/output-verify-prod-image.svg
@@ -1,4 +1,4 @@
-<svg width="1720.0" height="670" viewBox="0 0 1720.0 670"
+<svg width="1720.0" height="714" viewBox="0 0 1720.0 714"
xmlns="http://www.w3.org/2000/svg">
<style>
@font-face {
@@ -112,22 +112,24 @@
<circle cx="38" cy="8" r="8" fill="#ffbd2e"/>
<circle cx="62" cy="8" r="8" fill="#28c941"/>
</svg>
- <div id="terminal-title-tab">Command: setup-autocomplete</div>
+ <div id="terminal-title-tab">Command: verify-prod-image</div>
</div>
<div id='terminal-body'>
<div><span class="r2"> </span></div>
-<div><span class="r2"> </span><span class="r3">Usage: </span><span class="r2">breeze setup-autocomplete [OPTIONS] </span></div>
+<div><span class="r2"> </span><span class="r3">Usage: </span><span class="r2">breeze verify-prod-image [OPTIONS] [EXTRA_PYTEST_ARGS]... </span></div>
<div><span class="r2"> </span></div>
-<div><span class="r1"> </span><span class="r1">Enables autocompletion of breeze commands. </span><span class="r1"> </span></div>
+<div><span class="r1"> </span><span class="r1">Verify Production image. </span><span class="r1"> </span></div>
<div><span class="r1"> </span></div>
-<div><span class="r4">╭─ Setup autocomplete flags ───────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--force</span><span class="r1"> </span><span class="r6">-f</span><span class="r1"> </span><span class="r1">Force autocomplete setup even if already setup before (overrides the setup). </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╭─ Verify image flags ─────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--image-name</span><span class="r1"> </span><span class="r6">-n</span><span class="r1"> Name of the image to verify (overrides </span><span class="r5">--python</span><span class="r1"> and </span><span class="r5">--image-tag</span><span class="r1">). </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--python</span><span class="r1"> </span><span class="r6">-p</span><span class="r1"> </span><span class="r1">Python major/minor version used in Airflow image for PROD/CI images.</span><span class="r1"> </span><span class="r7">(3.7 | 3.8 | 3.9 | 3.10)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--image-tag</span><span class="r1"> </span><span class="r6">-t</span><span class="r1"> </span><span class="r1">Tag added to the default naming conventions of Airflow CI/PROD images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r4">╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y|n|q|yes|no|quit)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
-<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--github-repository</span><span class="r1"> </span><span class="r6">-g</span><span class="r1"> </span><span class="r1">GitHub repository used to pull, push run images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r4">[default: apache/airflow]</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
<div><span class="r1"></span><span class="r1"> </span></div>
</div>
diff --git a/images/breeze/output-version.svg b/images/breeze/output-version.svg
index 3702f9b8e4..ee6657d3c6 100644
--- a/images/breeze/output-version.svg
+++ b/images/breeze/output-version.svg
@@ -117,7 +117,7 @@
<div><span class="r2"> </span></div>
<div><span class="r2"> </span><span class="r3">Usage: </span><span class="r2">breeze version [OPTIONS] </span></div>
<div><span class="r2"> </span></div>
-<div><span class="r1"> </span><span class="r1">Prints version of breeze.py. </span><span class="r1"> </span></div>
+<div><span class="r1"> </span><span class="r1">Print information about version of apache-airflow-breeze. </span><span class="r1"> </span></div>
<div><span class="r1"> </span></div>
<div><span class="r4">╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
diff --git a/scripts/ci/build_airflow/ci_build_airflow_packages.sh b/scripts/ci/build_airflow/ci_build_airflow_packages.sh
deleted file mode 100755
index e6c405c6c3..0000000000
--- a/scripts/ci/build_airflow/ci_build_airflow_packages.sh
+++ /dev/null
@@ -1,23 +0,0 @@
-#!/usr/bin/env bash
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-# shellcheck source=scripts/ci/libraries/_script_init.sh
-. "$( dirname "${BASH_SOURCE[0]}" )/../libraries/_script_init.sh"
-
-build_images::prepare_ci_build
-build_images::rebuild_ci_image_if_needed_with_group
-runs::run_prepare_airflow_packages
diff --git a/scripts/ci/constraints/ci_generate_all_constraints.sh b/scripts/ci/constraints/ci_generate_all_constraints.sh
deleted file mode 100755
index 9a7a77e014..0000000000
--- a/scripts/ci/constraints/ci_generate_all_constraints.sh
+++ /dev/null
@@ -1,50 +0,0 @@
-#!/usr/bin/env bash
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-set -euo pipefail
-
-
-# We cannot perform full initialization because it will be done later in the "single run" scripts
-# And some readonly variables are set there, therefore we only selectively reuse parallel lib needed
-LIBRARIES_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")/../libraries/" && pwd)
-# shellcheck source=scripts/ci/libraries/_all_libs.sh
-source "${LIBRARIES_DIR}/_all_libs.sh"
-
-initialization::set_output_color_variables
-
-export CHECK_IMAGE_FOR_REBUILD="false"
-echo
-echo "${COLOR_YELLOW}Skip rebuilding CI images. Assume the one we have is good!${COLOR_RESET}"
-echo "${COLOR_YELLOW}You must run './breeze build-image --upgrade-to-newer-dependencies before for all python versions before running this one!${COLOR_RESET}"
-echo
-
-parallel::make_sure_gnu_parallel_is_installed
-
-parallel::make_sure_python_versions_are_specified
-
-echo
-echo "${COLOR_BLUE}Generating all constraint files${COLOR_RESET}"
-echo
-
-parallel::initialize_monitoring
-
-parallel::monitor_progress
-
-# shellcheck disable=SC2086
-parallel --results "${PARALLEL_MONITORED_DIR}" \
- "$( dirname "${BASH_SOURCE[0]}" )/ci_generate_constraints.sh" ::: \
- ${CURRENT_PYTHON_MAJOR_MINOR_VERSIONS_AS_STRING}
diff --git a/scripts/ci/constraints/ci_generate_constraints.sh b/scripts/ci/constraints/ci_generate_constraints.sh
deleted file mode 100755
index fd9b8ddfae..0000000000
--- a/scripts/ci/constraints/ci_generate_constraints.sh
+++ /dev/null
@@ -1,31 +0,0 @@
-#!/usr/bin/env bash
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-if [[ $1 == "" ]]; then
- >&2 echo "Requires python MAJOR/MINOR version as first parameter"
- exit 1
-fi
-
-export PYTHON_MAJOR_MINOR_VERSION=$1
-shift
-
-# shellcheck source=scripts/ci/libraries/_script_init.sh
-. "$( dirname "${BASH_SOURCE[0]}" )/../libraries/_script_init.sh"
-
-build_images::prepare_ci_build
-
-runs::run_generate_constraints
diff --git a/scripts/ci/docker-compose/_docker.env b/scripts/ci/docker-compose/_docker.env
index 6e94776dfd..56e371469a 100644
--- a/scripts/ci/docker-compose/_docker.env
+++ b/scripts/ci/docker-compose/_docker.env
@@ -15,6 +15,8 @@
# specific language governing permissions and limitations
# under the License.
AIRFLOW_CI_IMAGE
+AIRFLOW_EXTRAS
+ANSWER
BACKEND
BREEZE
CI
@@ -58,12 +60,11 @@ LIST_OF_INTEGRATION_TESTS_TO_RUN
RUN_SYSTEM_TESTS
START_AIRFLOW
SKIP_ENVIRONMENT_INITIALIZATION
-SKIP_TWINE_CHECK
+SKIP_PACKAGE_VERIFICATION
SKIP_SSH_SETUP
TEST_TYPE
UPGRADE_TO_NEWER_DEPENDENCIES
VERBOSE
VERBOSE_COMMANDS
VERSION_SUFFIX_FOR_PYPI
-VERSION_SUFFIX_FOR_SVN
WHEEL_VERSION
diff --git a/scripts/ci/docker-compose/base.yml b/scripts/ci/docker-compose/base.yml
index 22e2e51bf0..124ed9f6dc 100644
--- a/scripts/ci/docker-compose/base.yml
+++ b/scripts/ci/docker-compose/base.yml
@@ -29,6 +29,8 @@ services:
# We need all those env variables here because docker-compose-v2 does not really work well
# With env files and there are many problems with it:
- AIRFLOW_CI_IMAGE=${AIRFLOW_CI_IMAGE}
+ - AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}
+ - ANSWER=${ANSWER}
- BACKEND=${BACKEND}
- BREEZE=${BREEZE}
- CI=${CI}
@@ -71,14 +73,13 @@ services:
- RUN_SYSTEM_TESTS=${RUN_SYSTEM_TESTS}
- START_AIRFLOW=${START_AIRFLOW}
- SKIP_ENVIRONMENT_INITIALIZATION=${SKIP_ENVIRONMENT_INITIALIZATION}
- - SKIP_TWINE_CHECK=${SKIP_TWINE_CHECK}
+ - SKIP_PACKAGE_VERIFICATION=${SKIP_PACKAGE_VERIFICATION}
- SKIP_SSH_SETUP=${SKIP_SSH_SETUP}
- TEST_TYPE=${TEST_TYPE}
- UPGRADE_TO_NEWER_DEPENDENCIES=${UPGRADE_TO_NEWER_DEPENDENCIES}
- VERBOSE=${VERBOSE}
- VERBOSE_COMMANDS=${VERBOSE_COMMANDS}
- VERSION_SUFFIX_FOR_PYPI=${VERSION_SUFFIX_FOR_PYPI}
- - VERSION_SUFFIX_FOR_SVN=${VERSION_SUFFIX_FOR_SVN}
- WHEEL_VERSION=${WHEEL_VERSION}
volumes:
# Pass docker to inside of the container so that Kind and Moto tests can use it.
diff --git a/scripts/ci/docker-compose/devcontainer.env b/scripts/ci/docker-compose/devcontainer.env
index c07135d47b..708b1a5496 100644
--- a/scripts/ci/docker-compose/devcontainer.env
+++ b/scripts/ci/docker-compose/devcontainer.env
@@ -1,5 +1,6 @@
HOME=
AIRFLOW_CI_IMAGE="ghcr.io/apache/airflow/main/ci/python3.7:latest"
+ANSWER=
PYTHON_MAJOR_MINOR_VERSION="3.7"
AIRFLOW_EXTRAS=
BREEZE="true"
@@ -40,7 +41,7 @@ RUN_TESTS="false"
LIST_OF_INTEGRATION_TESTS_TO_RUN=""
RUN_SYSTEM_TESTS=""
START_AIRFLOW="false"
-SKIP_TWINE_CHECK="false"
+SKIP_PACKAGE_VERIFICATION="false"
SKIP_SSH_SETUP="true"
SKIP_ENVIRONMENT_INITIALIZATION="false"
TEST_TYPE=
@@ -48,5 +49,4 @@ UPGRADE_TO_NEWER_DEPENDENCIES="false"
VERBOSE="false"
VERBOSE_COMMANDS="false"
VERSION_SUFFIX_FOR_PYPI=
-VERSION_SUFFIX_FOR_SVN=
WHEEL_VERSION=0.36.2
diff --git a/scripts/ci/docs/ci_docs.sh b/scripts/ci/docs/ci_docs.sh
deleted file mode 100755
index be0d2ede88..0000000000
--- a/scripts/ci/docs/ci_docs.sh
+++ /dev/null
@@ -1,25 +0,0 @@
-#!/usr/bin/env bash
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-# shellcheck source=scripts/ci/libraries/_script_init.sh
-. "$( dirname "${BASH_SOURCE[0]}" )/../libraries/_script_init.sh"
-
-build_images::prepare_ci_build
-
-build_images::rebuild_ci_image_if_needed_with_group
-
-runs::run_docs "${@}"
diff --git a/scripts/ci/docs/ci_docs_prepare.sh b/scripts/ci/docs/ci_docs_prepare.sh
deleted file mode 100755
index 6b3958e1d8..0000000000
--- a/scripts/ci/docs/ci_docs_prepare.sh
+++ /dev/null
@@ -1,22 +0,0 @@
-#!/usr/bin/env bash
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-# shellcheck source=scripts/ci/libraries/_script_init.sh
-. "$( dirname "${BASH_SOURCE[0]}" )/../libraries/_script_init.sh"
-
-curl -sSLo "${AIRFLOW_SOURCES}/constraints.txt" \
- "https://raw.githubusercontent.com/apache/airflow/constraints-${DEFAULT_BRANCH}/constraints-${PYTHON_MAJOR_MINOR_VERSION}.txt"
diff --git a/scripts/ci/images/ci_build_ci_image_on_ci.sh b/scripts/ci/images/ci_build_ci_image_on_ci.sh
deleted file mode 100755
index 2647ed0ba8..0000000000
--- a/scripts/ci/images/ci_build_ci_image_on_ci.sh
+++ /dev/null
@@ -1,30 +0,0 @@
-#!/usr/bin/env bash
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-# shellcheck source=scripts/ci/libraries/_script_init.sh
-. "$( dirname "${BASH_SOURCE[0]}" )/../libraries/_script_init.sh"
-
-# Builds CI image in the CI environment
-function build_ci_image_on_ci() {
- build_images::prepare_ci_build
- start_end::group_start "Build CI image ${AIRFLOW_CI_IMAGE}"
- build_images::clean_build_cache
- build_images::rebuild_ci_image_if_needed
- start_end::group_end
-}
-
-build_ci_image_on_ci
diff --git a/scripts/ci/images/ci_build_prod_image_on_ci.sh b/scripts/ci/images/ci_build_prod_image_on_ci.sh
deleted file mode 100755
index 444c3336b8..0000000000
--- a/scripts/ci/images/ci_build_prod_image_on_ci.sh
+++ /dev/null
@@ -1,37 +0,0 @@
-#!/usr/bin/env bash
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-export INSTALL_FROM_PYPI="false"
-export INSTALL_PROVIDERS_FROM_SOURCES="false"
-export INSTALL_FROM_DOCKER_CONTEXT_FILES="true"
-export AIRFLOW_PRE_CACHED_PIP_PACKAGES="false"
-export VERBOSE="true"
-
-# shellcheck source=scripts/ci/libraries/_script_init.sh
-. "$( dirname "${BASH_SOURCE[0]}" )/../libraries/_script_init.sh"
-
-# Builds or waits for the PROD image in the CI environment
-function build_prod_images_on_ci() {
- build_images::prepare_prod_build
- start_end::group_start "Build PROD image ${AIRFLOW_CI_IMAGE}"
- build_images::clean_build_cache
- build_images::build_prod_images_from_locally_built_airflow_packages
- start_end::group_end
-}
-
-build_prod_images_on_ci
diff --git a/scripts/ci/images/ci_pull_ci_image_on_ci.sh b/scripts/ci/images/ci_pull_ci_image_on_ci.sh
deleted file mode 100755
index 02919f59bb..0000000000
--- a/scripts/ci/images/ci_pull_ci_image_on_ci.sh
+++ /dev/null
@@ -1,34 +0,0 @@
-#!/usr/bin/env bash
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-# shellcheck source=scripts/ci/libraries/_script_init.sh
-. "$( dirname "${BASH_SOURCE[0]}" )/../libraries/_script_init.sh"
-
-# Pulls CI image that has already been built
-function pull_ci_image_on_ci() {
- build_images::prepare_ci_build
- start_end::group_start "Pull CI image ${AIRFLOW_CI_IMAGE}"
- build_images::clean_build_cache
- md5sum::calculate_md5sum_for_all_files
- local image_name_with_tag="${AIRFLOW_CI_IMAGE}:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
- push_pull_remove_images::wait_for_image "${image_name_with_tag}"
- docker_v tag "${image_name_with_tag}" "${AIRFLOW_CI_IMAGE}"
- md5sum::update_all_md5_with_group
- start_end::group_end
-}
-
-pull_ci_image_on_ci
diff --git a/scripts/ci/images/ci_push_production_images.sh b/scripts/ci/images/ci_push_production_images.sh
deleted file mode 100755
index f7bc119915..0000000000
--- a/scripts/ci/images/ci_push_production_images.sh
+++ /dev/null
@@ -1,34 +0,0 @@
-#!/usr/bin/env bash
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-# shellcheck source=scripts/ci/libraries/_script_init.sh
-. "$( dirname "${BASH_SOURCE[0]}" )/../libraries/_script_init.sh"
-
-# Pushes PROD images with tags to registry in GitHub
-function push_prod_image_with_tag_to_github () {
- start_end::group_start "Push PROD image"
- local airflow_prod_tagged_image="${AIRFLOW_PROD_IMAGE}:${GITHUB_REGISTRY_PUSH_IMAGE_TAG}"
- docker_v tag "${AIRFLOW_PROD_IMAGE}" "${airflow_prod_tagged_image}"
- push_pull_remove_images::push_image_with_retries "${airflow_prod_tagged_image}"
- start_end::group_end
-}
-
-build_images::prepare_prod_build
-
-build_images::login_to_docker_registry
-
-push_prod_image_with_tag_to_github
diff --git a/scripts/ci/images/ci_run_docker_compose_quick_start_test.sh b/scripts/ci/images/ci_run_docker_compose_quick_start_test.sh
deleted file mode 100755
index 1b6b90d62a..0000000000
--- a/scripts/ci/images/ci_run_docker_compose_quick_start_test.sh
+++ /dev/null
@@ -1,28 +0,0 @@
-#!/usr/bin/env bash
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-# shellcheck source=scripts/ci/libraries/_script_init.sh
-. "$(dirname "${BASH_SOURCE[0]}")/../libraries/_script_init.sh"
-
-
-DOCKER_IMAGE="${AIRFLOW_PROD_IMAGE}:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
-export DOCKER_IMAGE
-
-build_images::prepare_prod_build
-push_pull_remove_images::wait_for_image "${DOCKER_IMAGE}"
-
-python3 "${SCRIPTS_CI_DIR}/images/ci_run_docker_tests.py" "${AIRFLOW_SOURCES}/docker_tests/test_docker_compose_quick_start.py"
diff --git a/scripts/ci/images/ci_run_docker_tests.py b/scripts/ci/images/ci_run_docker_tests.py
deleted file mode 100755
index b99a17bb40..0000000000
--- a/scripts/ci/images/ci_run_docker_tests.py
+++ /dev/null
@@ -1,105 +0,0 @@
-#!/usr/bin/env python3
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-import argparse
-import shlex
-import subprocess
-import sys
-from pathlib import Path
-from typing import List
-
-AIRFLOW_SOURCE = Path(__file__).resolve().parents[3]
-BUILD_CACHE_DIR = AIRFLOW_SOURCE / ".build"
-
-CBLUE = '\033[94m'
-CEND = '\033[0m'
-
-
-def get_parser():
- parser = argparse.ArgumentParser(
- prog="ci_run_docker_tests",
- description="Running Docker tests using pytest",
- epilog="Unknown arguments are passed unchanged to Pytest.",
- )
- parser.add_argument(
- "--interactive",
- "-i",
- action='store_true',
- help="Activates virtual environment ready to run tests and drops you in",
- )
- parser.add_argument("--initialize", action="store_true", help="Initialize virtual environment and exit")
- parser.add_argument("pytestopts", nargs=argparse.REMAINDER, help="Tests to run")
- return parser
-
-
-def run_verbose(cmd: List[str], *, check=True, **kwargs):
- print(f"{CBLUE}$ {' '.join(shlex.quote(c) for c in cmd)}{CEND}")
- subprocess.run(cmd, check=check, **kwargs)
-
-
-def create_virtualenv():
- virtualenv_path = (
- BUILD_CACHE_DIR / ".docker_venv" / f"host_python_{sys.version_info[0]}.{sys.version_info[1]}"
- )
- virtualenv_path.parent.mkdir(parents=True, exist_ok=True)
- if not virtualenv_path.exists():
- print("Creating virtualenv environment")
- run_verbose([sys.executable, "-m", "venv", str(virtualenv_path)])
-
- python_bin = virtualenv_path / "bin" / "python"
- run_verbose(
- [
- str(python_bin),
- "-m",
- "pip",
- "install",
- "-r",
- str(AIRFLOW_SOURCE / "docker_tests" / "requirements.txt"),
- ]
- )
- return python_bin
-
-
-def main():
- parser = get_parser()
- args = parser.parse_args()
-
- python_bin = create_virtualenv()
-
- if args.initialize:
- return
- if args.interactive:
- activate_bin = python_bin.parent / "activate"
- bash_trampoline = f"source {shlex.quote(str(activate_bin))}"
- print("To enter virtual environment, run:")
- print(f" {bash_trampoline}")
- return
-
- extra_pytest_args = (
- args.pytestopts[1:] if args.pytestopts and args.pytestopts[0] == "--" else args.pytestopts
- )
- if not extra_pytest_args:
- raise SystemExit("You must select the tests to run.")
-
- pytest_args = ("-n", "auto", "--color=yes")
-
- run_verbose([str(python_bin), "-m", "pytest", *pytest_args, *extra_pytest_args])
-
-
-if __name__ == "__main__":
- main()
diff --git a/scripts/ci/images/ci_wait_for_and_verify_all_ci_images.sh b/scripts/ci/images/ci_wait_for_and_verify_all_ci_images.sh
deleted file mode 100755
index 0a31ad22f4..0000000000
--- a/scripts/ci/images/ci_wait_for_and_verify_all_ci_images.sh
+++ /dev/null
@@ -1,49 +0,0 @@
-#!/usr/bin/env bash
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-set -euo pipefail
-
-# We cannot perform full initialization because it will be done later in the "single run" scripts
-# And some readonly variables are set there, therefore we only selectively reuse parallel lib needed
-LIBRARIES_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")/../libraries/" && pwd)
-# shellcheck source=scripts/ci/libraries/_all_libs.sh
-source "${LIBRARIES_DIR}/_all_libs.sh"
-
-python3 "$( dirname "${BASH_SOURCE[0]}" )/ci_run_docker_tests.py" "--initialize"
-
-initialization::set_output_color_variables
-
-export PARALLEL_TAIL_LENGTH=5
-
-parallel::make_sure_gnu_parallel_is_installed
-
-parallel::make_sure_python_versions_are_specified
-
-echo
-echo "${COLOR_BLUE}Waiting for all CI images to appear${COLOR_RESET}"
-echo
-
-parallel::initialize_monitoring
-
-parallel::monitor_progress
-
-build_images::login_to_docker_registry
-
-# shellcheck disable=SC2086
-parallel --results "${PARALLEL_MONITORED_DIR}" \
- "$( dirname "${BASH_SOURCE[0]}" )/ci_wait_for_and_verify_ci_image.sh" ::: \
- ${CURRENT_PYTHON_MAJOR_MINOR_VERSIONS_AS_STRING}
diff --git a/scripts/ci/images/ci_wait_for_and_verify_all_prod_images.sh b/scripts/ci/images/ci_wait_for_and_verify_all_prod_images.sh
deleted file mode 100755
index bd6c336b0d..0000000000
--- a/scripts/ci/images/ci_wait_for_and_verify_all_prod_images.sh
+++ /dev/null
@@ -1,49 +0,0 @@
-#!/usr/bin/env bash
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-set -euo pipefail
-
-# We cannot perform full initialization because it will be done later in the "single run" scripts
-# And some readonly variables are set there, therefore we only selectively reuse parallel lib needed
-LIBRARIES_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")/../libraries/" && pwd)
-# shellcheck source=scripts/ci/libraries/_all_libs.sh
-source "${LIBRARIES_DIR}/_all_libs.sh"
-
-initialization::set_output_color_variables
-
-python3 "$( dirname "${BASH_SOURCE[0]}" )/ci_run_docker_tests.py" "--initialize"
-
-export PARALLEL_TAIL_LENGTH=5
-
-parallel::make_sure_gnu_parallel_is_installed
-
-parallel::make_sure_python_versions_are_specified
-
-echo
-echo "${COLOR_BLUE}Waiting for all PROD images to appear${COLOR_RESET}"
-echo
-
-parallel::initialize_monitoring
-
-parallel::monitor_progress
-
-build_images::login_to_docker_registry
-
-# shellcheck disable=SC2086
-parallel --results "${PARALLEL_MONITORED_DIR}" \
- "$( dirname "${BASH_SOURCE[0]}" )/ci_wait_for_and_verify_prod_image.sh" ::: \
- ${CURRENT_PYTHON_MAJOR_MINOR_VERSIONS_AS_STRING}
diff --git a/scripts/ci/images/ci_wait_for_and_verify_ci_image.sh b/scripts/ci/images/ci_wait_for_and_verify_ci_image.sh
deleted file mode 100755
index 7bbd9c9af0..0000000000
--- a/scripts/ci/images/ci_wait_for_and_verify_ci_image.sh
+++ /dev/null
@@ -1,43 +0,0 @@
-#!/usr/bin/env bash
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-if [[ $1 == "" ]]; then
- >&2 echo "Requires python MAJOR/MINOR version as first parameter"
- exit 1
-fi
-
-export PYTHON_MAJOR_MINOR_VERSION=$1
-shift
-
-
-# shellcheck source=scripts/ci/libraries/_script_init.sh
-. "$( dirname "${BASH_SOURCE[0]}" )/../libraries/_script_init.sh"
-
-image_name_with_tag="${AIRFLOW_CI_IMAGE}:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
-
-build_images::prepare_ci_build
-
-push_pull_remove_images::wait_for_image "${image_name_with_tag}"
-
-if [[ ${VERIFY_IMAGE=} != "false" ]]; then
- verify_image::verify_ci_image "${image_name_with_tag}"
-fi
-
-md5sum::update_all_md5_with_group
-
-docker_v tag "${image_name_with_tag}" "${AIRFLOW_CI_IMAGE}"
diff --git a/scripts/ci/images/ci_wait_for_and_verify_prod_image.sh b/scripts/ci/images/ci_wait_for_and_verify_prod_image.sh
deleted file mode 100755
index 4a79abdc6c..0000000000
--- a/scripts/ci/images/ci_wait_for_and_verify_prod_image.sh
+++ /dev/null
@@ -1,40 +0,0 @@
-#!/usr/bin/env bash
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-if [[ $1 == "" ]]; then
- >&2 echo "Requires python MAJOR/MINOR version as first parameter"
- exit 1
-fi
-
-export PYTHON_MAJOR_MINOR_VERSION=$1
-shift
-
-
-# shellcheck source=scripts/ci/libraries/_script_init.sh
-. "$( dirname "${BASH_SOURCE[0]}" )/../libraries/_script_init.sh"
-
-image_name_with_tag="${AIRFLOW_PROD_IMAGE}:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
-
-build_images::prepare_prod_build
-push_pull_remove_images::wait_for_image "${image_name_with_tag}"
-
-if [[ ${VERIFY_IMAGE=} != "false" ]]; then
- verify_image::verify_prod_image "${image_name_with_tag}"
-fi
-
-docker_v tag "${image_name_with_tag}" "${AIRFLOW_PROD_IMAGE}"
diff --git a/scripts/ci/images/self_terminate.sh b/scripts/ci/images/self_terminate.sh
old mode 100644
new mode 100755
diff --git a/scripts/ci/kubernetes/ci_setup_cluster_and_deploy_airflow_to_kubernetes.sh b/scripts/ci/kubernetes/ci_setup_cluster_and_deploy_airflow_to_kubernetes.sh
index fce0a4ce08..d7c576ebec 100755
--- a/scripts/ci/kubernetes/ci_setup_cluster_and_deploy_airflow_to_kubernetes.sh
+++ b/scripts/ci/kubernetes/ci_setup_cluster_and_deploy_airflow_to_kubernetes.sh
@@ -27,8 +27,6 @@ traps::add_trap "kind::dump_kind_logs" EXIT HUP INT TERM
kind::make_sure_kubernetes_tools_are_installed
kind::get_kind_cluster_name
kind::perform_kind_cluster_operation "start"
-build_images::prepare_prod_build
-build_images::build_prod_images
kind::build_image_for_kubernetes_tests
kind::load_image_to_kind_cluster
kind::deploy_airflow_with_helm
diff --git a/scripts/ci/libraries/_all_libs.sh b/scripts/ci/libraries/_all_libs.sh
index 95807c409b..8945e5d88b 100755
--- a/scripts/ci/libraries/_all_libs.sh
+++ b/scripts/ci/libraries/_all_libs.sh
@@ -36,8 +36,6 @@ readonly SCRIPTS_CI_DIR
. "${LIBRARIES_DIR}"/_repeats.sh
# shellcheck source=scripts/ci/libraries/_sanity_checks.sh
. "${LIBRARIES_DIR}"/_sanity_checks.sh
-# shellcheck source=scripts/ci/libraries/_build_images.sh
-. "${LIBRARIES_DIR}"/_build_images.sh
# shellcheck source=scripts/ci/libraries/_kind.sh
. "${LIBRARIES_DIR}"/_kind.sh
# shellcheck source=scripts/ci/libraries/_local_mounts.sh
@@ -48,13 +46,9 @@ readonly SCRIPTS_CI_DIR
. "${LIBRARIES_DIR}"/_parameters.sh
# shellcheck source=scripts/ci/libraries/_push_pull_remove_images.sh
. "${LIBRARIES_DIR}"/_push_pull_remove_images.sh
-# shellcheck source=scripts/ci/libraries/_runs.sh
-. "${LIBRARIES_DIR}"/_runs.sh
# shellcheck source=scripts/ci/libraries/_start_end.sh
. "${LIBRARIES_DIR}"/_start_end.sh
# shellcheck source=scripts/ci/libraries/_testing.sh
. "${LIBRARIES_DIR}"/_testing.sh
# shellcheck source=scripts/ci/libraries/_verbosity.sh
. "${LIBRARIES_DIR}"/_verbosity.sh
-# shellcheck source=scripts/ci/libraries/_verify_image.sh
-. "${LIBRARIES_DIR}"/_verify_image.sh
diff --git a/scripts/ci/libraries/_build_images.sh b/scripts/ci/libraries/_build_images.sh
deleted file mode 100644
index 531fb3fc3e..0000000000
--- a/scripts/ci/libraries/_build_images.sh
+++ /dev/null
@@ -1,862 +0,0 @@
-#!/usr/bin/env bash
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-# Needs to be declared outside of function for MacOS
-
-BUILD_COMMAND=()
-
-# Fixing permissions for all important files that are going to be added to Docker context
-# This is necessary, because there are different default umask settings on different *NIX
-# In case of some systems (especially in the CI environments) there is default +w group permission
-# set automatically via UMASK when git checkout is performed.
-# https://unix.stackexchange.com/questions/315121/why-is-the-default-umask-002-or-022-in-many-unix-systems-seems-insecure-by-defa
-# Unfortunately default setting in git is to use UMASK by default:
-# https://git-scm.com/docs/git-config/1.6.3.1#git-config-coresharedRepository
-# This messes around with Docker context invalidation because the same files have different permissions
-# and effectively different hash used for context validation calculation.
-#
-# We fix it by removing write permissions for other/group for all files that are in the Docker context.
-#
-# Since we can't (easily) tell what dockerignore would restrict, we'll just to
-# it to "all" files in the git repo, making sure to exclude the www/static/docs
-# symlink which is broken until the docs are built.
-function build_images::filterout_deleted_files {
- # Take NUL-separated stdin, return only files that exist on stdout NUL-separated
- # This is to cope with users deleting files or folders locally but not doing `git rm`
- xargs -0 "$STAT_BIN" --printf '%n\0' 2>/dev/null || true;
-}
-
-# Fixes permissions for groups for all the files that are quickly filtered using
-# the build_images::filterout_deleted_files
-function build_images::fix_group_permissions() {
- if [[ ${PERMISSIONS_FIXED:=} == "true" ]]; then
- return
- fi
- pushd "${AIRFLOW_SOURCES}" >/dev/null 2>&1 || exit 1
- # This deals with files
- git ls-files -z -- ./ | build_images::filterout_deleted_files | xargs -0 chmod og-w
- # and this deals with directories
- git ls-tree -z -r -d --name-only HEAD | build_images::filterout_deleted_files | xargs -0 chmod og-w,og+x
- popd >/dev/null 2>&1 || exit 1
- export PERMISSIONS_FIXED="true"
-}
-
-
-# For remote installation of airflow (from GitHub or PyPI) when building the image, you need to
-# pass build flags depending on the version and method of the installation (for example to
-# get proper requirement constraint files)
-function build_images::add_build_args_for_remote_install() {
- # entrypoint is used as AIRFLOW_SOURCES_(WWW)_FROM/TO in order to avoid costly copying of all sources of
- # Airflow - those are not needed for remote install at all. Entrypoint is later overwritten by
- EXTRA_DOCKER_PROD_BUILD_FLAGS+=(
- "--build-arg" "AIRFLOW_SOURCES_WWW_FROM=Dockerfile"
- "--build-arg" "AIRFLOW_SOURCES_WWW_TO=/Dockerfile"
- "--build-arg" "AIRFLOW_SOURCES_FROM=Dockerfile"
- "--build-arg" "AIRFLOW_SOURCES_TO=/Dockerfile"
- )
- if [[ -n "${AIRFLOW_CONSTRAINTS_REFERENCE}" ]]; then
- EXTRA_DOCKER_PROD_BUILD_FLAGS+=(
- "--build-arg" "AIRFLOW_CONSTRAINTS_REFERENCE=${AIRFLOW_CONSTRAINTS_REFERENCE}"
- )
- else
- if [[ ${AIRFLOW_VERSION} =~ v?2.* ]]; then
- EXTRA_DOCKER_PROD_BUILD_FLAGS+=(
- # For specified minor version of 2.0 or v2 branch use specific reference constraints
- "--build-arg" "AIRFLOW_CONSTRAINTS_REFERENCE=constraints-${AIRFLOW_VERSION}"
- )
- else
- # For all other we just get the default constraint branch coming from the _initialization.sh
- EXTRA_DOCKER_PROD_BUILD_FLAGS+=(
- "--build-arg" "AIRFLOW_CONSTRAINTS_REFERENCE=${DEFAULT_CONSTRAINTS_BRANCH}"
- )
- fi
- fi
- if [[ -n "${AIRFLOW_CONSTRAINTS_LOCATION}" ]]; then
- EXTRA_DOCKER_PROD_BUILD_FLAGS+=(
- "--build-arg" "AIRFLOW_CONSTRAINTS_LOCATION=${AIRFLOW_CONSTRAINTS_LOCATION}"
- )
- fi
- # Depending on the version built, we choose the right branch for preloading the packages from
- # For v2-*-test we choose v2-*-test
- # all other builds when you choose a specific version (1.0, 2.0, 2.1. series) should choose stable branch
- # to preload. For all other builds we use the default branch defined in _initialization.sh
- # TODO: Generalize me
- if [[ ${AIRFLOW_VERSION} == 'v2-0-test' ]]; then
- AIRFLOW_BRANCH_FOR_PYPI_PRELOADING="v2-0-test"
- elif [[ ${AIRFLOW_VERSION} == 'v2-1-test' ]]; then
- AIRFLOW_BRANCH_FOR_PYPI_PRELOADING="v2-1-test"
- elif [[ ${AIRFLOW_VERSION} == 'v2-2-test' ]]; then
- AIRFLOW_BRANCH_FOR_PYPI_PRELOADING="v2-2-test"
- elif [[ ${AIRFLOW_VERSION} =~ v?2\.0* ]]; then
- AIRFLOW_BRANCH_FOR_PYPI_PRELOADING="v2-0-stable"
- elif [[ ${AIRFLOW_VERSION} =~ v?2\.1* ]]; then
- AIRFLOW_BRANCH_FOR_PYPI_PRELOADING="v2-1-stable"
- elif [[ ${AIRFLOW_VERSION} =~ v?2\.2* ]]; then
- AIRFLOW_BRANCH_FOR_PYPI_PRELOADING="v2-2-stable"
- else
- AIRFLOW_BRANCH_FOR_PYPI_PRELOADING=${DEFAULT_BRANCH}
- fi
-}
-
-# Retrieves version of airflow stored in the production image (used to display the actual
-# Version we use if it was build from PyPI or GitHub
-function build_images::get_airflow_version_from_production_image() {
- docker run --entrypoint /bin/bash "${AIRFLOW_PROD_IMAGE}" -c 'echo "${AIRFLOW_VERSION}"'
-}
-
-function build_images::reconfirm_rebuilding_if_not_rebased() {
- local latest_main_commit_sha
- latest_main_commit_sha=$(curl -s -H "Accept: application/vnd.github.VERSION.sha" \
- "https://api.github.com/repos/${GITHUB_REPOSITORY}/commits/${DEFAULT_BRANCH}")
- if [[ "$(git log --format=format:%H | grep -c "${latest_main_commit_sha}")" == "0" ]]; then
- echo
- echo "${COLOR_YELLOW}WARNING!!!!:You are not rebased on top of the latest ${DEFAULT_BRANCH} branch of the airflow repo.${COLOR_RESET}"
- echo "${COLOR_YELLOW}The rebuild might take a lot of time and you might need to do it again${COLOR_RESET}"
- echo
- echo "${COLOR_YELLOW}It is STRONGLY RECOMMENDED that you rebase your code first!${COLOR_RESET}"
- echo
- "${AIRFLOW_SOURCES}/scripts/tools/confirm" "You are really sure you want to rebuild ${THE_IMAGE_TYPE}-python${PYTHON_MAJOR_MINOR_VERSION}"
- RES=$?
- fi
-}
-
-function build_images::print_modified_files() {
- echo "${MODIFIED_FILES[@]}" | xargs -n 1 echo " * "
-}
-
-function build_images::encourage_rebuilding_on_modified_files() {
- echo
- set +u
- if [[ ${#MODIFIED_FILES[@]} != "" ]]; then
- echo
- echo "${COLOR_YELLOW}The CI image for Python ${PYTHON_MAJOR_MINOR_VERSION} might be outdated${COLOR_RESET}"
- echo
- echo "${COLOR_BLUE}Please run this command at earliest convenience: ${COLOR_RESET}"
- echo
- echo "${COLOR_YELLOW}./breeze build-image --python ${PYTHON_MAJOR_MINOR_VERSION}${COLOR_RESET}"
- echo
- fi
-}
-
-function build_images::confirm_rebuilding_on_modified_files() {
- echo
- set +u
- if [[ ${#MODIFIED_FILES[@]} != "" ]]; then
- echo "${COLOR_BLUE}The CI image for Python ${PYTHON_MAJOR_MINOR_VERSION} image likely needs to be rebuild${COLOR_RESET}"
- echo "${COLOR_BLUE}The files were modified since last build:${COLOR_RESET}"
- echo
- echo "${COLOR_BLUE}$(build_images::print_modified_files)${COLOR_RESET}"
- echo
- fi
- set -u
- # Make sure to use output of tty rather than stdin/stdout when available - this way confirm
- # will works also in case of pre-commits (git does not pass stdin/stdout to pre-commit hooks)
- # shellcheck disable=SC2094
- "${AIRFLOW_SOURCES}/scripts/tools/confirm" "PULL & BUILD the image ${THE_IMAGE_TYPE}-python${PYTHON_MAJOR_MINOR_VERSION}"
- RES=$?
- if [[ ${RES} == "0" ]]; then
- build_images::reconfirm_rebuilding_if_not_rebased
- fi
-}
-
-# Confirms if the image should be rebuilt and interactively checks it with the user.
-# In case iit needs to be rebuilt. It only ask the user if it determines that the rebuild
-# is needed and that the rebuild is not already forced. It asks the user using available terminals
-# So that the script works also from within pre-commit run via git hooks - where stdin is not
-# available - it tries to find usable terminal and ask the user via this terminal.
-function build_images::confirm_image_rebuild() {
- set +e
- local RES
- if [[ ${CI:="false"} == "true" ]]; then
- verbosity::print_info
- verbosity::print_info "CI environment - forcing pull and rebuild for image ${THE_IMAGE_TYPE}."
- verbosity::print_info
- RES="0"
- elif [[ -n "${FORCE_ANSWER_TO_QUESTIONS=}" ]]; then
- verbosity::print_info
- verbosity::print_info "Forcing answer '${FORCE_ANSWER_TO_QUESTIONS}'"
- verbosity::print_info
- case "${FORCE_ANSWER_TO_QUESTIONS}" in
- [yY][eE][sS] | [yY])
- RES="0"
- ;;
- [qQ][uU][iI][tT] | [qQ])
- RES="2"
- ;;
- *)
- RES="1"
- ;;
- esac
- elif [[ -t 0 ]]; then
- # Check if this script is run interactively with stdin open and terminal attached
- build_images::confirm_rebuilding_on_modified_files
- elif [[ ${DETECTED_TERMINAL:=$(tty)} != "not a tty" ]]; then
- export DETECTED_TERMINAL
- # shellcheck disable=SC2094
- build_images::encourage_rebuilding_on_modified_files >"${DETECTED_TERMINAL}" <"${DETECTED_TERMINAL}"
- RES=1
- elif [[ -c /dev/tty ]]; then
- export DETECTED_TERMINAL=/dev/tty
- # shellcheck disable=SC2094
- build_images::encourage_rebuilding_on_modified_files >"${DETECTED_TERMINAL}" <"${DETECTED_TERMINAL}"
- RES=1
- else
- verbosity::print_info
- verbosity::print_info "No terminal, no stdin - quitting"
- verbosity::print_info
- # No terminal, no stdin, no force answer - quitting!
- RES="2"
- fi
- set -e
- if [[ ${RES} == "1" ]]; then
- verbosity::print_info
- verbosity::print_info "Skipping rebuilding the image ${THE_IMAGE_TYPE}-python${PYTHON_MAJOR_MINOR_VERSION}"
- verbosity::print_info
- export SKIP_REBUILD="true"
- # Force "no" also to subsequent questions so that if you answer it once, you are not asked
- # For all other pre-commits and you will continue using the images you already have
- export FORCE_ANSWER_TO_QUESTIONS="no"
- elif [[ ${RES} == "2" ]]; then
- echo
- echo "${COLOR_RED}ERROR: The ${THE_IMAGE_TYPE} needs to be rebuilt - it is outdated. ${COLOR_RESET}"
- echo """
-
- ${COLOR_YELLOW}Make sure you build the image by running:${COLOR_RESET}
-
- ./breeze --python ${PYTHON_MAJOR_MINOR_VERSION} build-image
-
-"""
- exit 1
- else
- # Force "yes" also to subsequent questions
- export FORCE_ANSWER_TO_QUESTIONS="yes"
- fi
-}
-
-function build_images::check_for_docker_context_files() {
- local num_docker_context_files
- local docker_context_files_dir="${AIRFLOW_SOURCES}/docker-context-files/"
- num_docker_context_files=$(find "${docker_context_files_dir}" -type f | grep -c -v "README.md" || true)
- if [[ ${num_docker_context_files} == "0" ]]; then
- if [[ ${INSTALL_FROM_DOCKER_CONTEXT_FILES} != "false" ]]; then
- echo
- echo "${COLOR_YELLOW}ERROR! You want to install packages from docker-context-files${COLOR_RESET}"
- echo "${COLOR_YELLOW} but there are no packages to install in this folder.${COLOR_RESET}"
- echo
- exit 1
- fi
- else
- if [[ ${INSTALL_FROM_DOCKER_CONTEXT_FILES} == "false" ]]; then
- echo
- echo "${COLOR_YELLOW}ERROR! There are some extra files in docker-context-files except README.md${COLOR_RESET}"
... 2333 lines suppressed ...