You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2022/04/10 18:47:39 UTC
[airflow] branch main updated: Replace old Breeze with Python based implementation (#22880)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 10078287e2 Replace old Breeze with Python based implementation (#22880)
10078287e2 is described below
commit 10078287e2bc5df0cf4f1889fdd6714c6a766b9e
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Sun Apr 10 20:47:29 2022 +0200
Replace old Breeze with Python based implementation (#22880)
Over the last few months together with Outreachy interns
we rewrote the most important functionality of the old Bash-based
Breeze with Python Based implementation.
We approached it in systematic way with capturing all our decisions
in the ADR format (dev/breeze/docs) and implementing the parts
that are used on a daily basis by the users. Breeze2 as it was
called is ready for Prime-Time with the users so we are swapping
out the old breeze wiht the new one.
The old `breeze` has been moved to `breeze-legacy` and we will
gradually parts of it that are already migrated to Python and proven
while continue rewriting the parts that are missing (mostly the
maintainer tools) and replacing the remaining CI shell scripts with
the new `breeze` commands.
We also need to make sure that there is no accidental top-level
import for extra packages added in the future, because people
who installed breeze before will not have it - so we have
a pre-commit that checks if breeze.py can be parsed and
--help executed with just rich and click installed.
This PR:
* moves `breeze` to `breeze-legacy`
* moves `Breeze2` to `breeze`
* updates documentation and screenshots where applicable
* explains old vs. new breeze in documentation
* adds protection so that no accidental top-level import is
added to breeze.py and files imported from there
Fixes: #22827
---
.github/workflows/build-images.yml | 2 +-
.github/workflows/ci.yml | 4 +-
.gitignore | 4 +-
.gitpod.yml | 2 +-
.pre-commit-config.yaml | 23 +-
BREEZE.rst | 2111 ++---------
Breeze2 | 42 -
CI.rst | 10 +-
CONTRIBUTING.rst | 22 +-
CONTRIBUTORS_QUICK_START.rst | 38 +-
IMAGES.rst | 28 +-
LOCAL_VIRTUALENV.rst | 4 +-
STATIC_CODE_CHECKS.rst | 24 +-
TESTING.rst | 90 +-
breeze | 3698 +-------------------
breeze-complete | 7 +-
breeze => breeze-legacy | 0
dev/PROVIDER_PACKAGE_DETAILS.md | 18 +-
dev/README_RELEASE_AIRFLOW.md | 8 +-
dev/README_RELEASE_HELM_CHART.md | 2 +-
dev/README_RELEASE_PROVIDER_PACKAGES.md | 23 +-
dev/REFRESHING_CI_CACHE.md | 2 +-
dev/TRACKING_BACKTRACKING_ISSUES.md | 7 +-
dev/breeze/README.md | 2 +-
...e2-complete-bash.sh => breeze-complete-bash.sh} | 10 +-
...e2-complete-fish.sh => breeze-complete-fish.sh} | 6 +-
...eze2-complete-zsh.sh => breeze-complete-zsh.sh} | 10 +-
dev/breeze/doc/BREEZE2.md | 141 -
.../0002-implement-standalone-python-command.md | 6 +-
.../adr/0003-bootstrapping-virtual-environment.md | 28 +-
dev/breeze/setup.cfg | 2 +-
dev/breeze/src/airflow_breeze/__init__.py | 4 +-
dev/breeze/src/airflow_breeze/breeze.py | 317 +-
dev/breeze/src/airflow_breeze/global_constants.py | 4 +-
dev/breeze/src/airflow_breeze/pre_commit_ids.py | 1 +
dev/breeze/src/airflow_breeze/shell/enter_shell.py | 7 +-
dev/breeze/src/airflow_breeze/utils/cache.py | 1 -
dev/breeze/src/airflow_breeze/utils/confirm.py | 29 +-
dev/breeze/src/airflow_breeze/utils/console.py | 13 +-
dev/breeze/src/airflow_breeze/utils/path_utils.py | 1 -
dev/breeze/src/airflow_breeze/utils/run_utils.py | 2 +-
dev/breeze/src/airflow_breeze/utils/visuals.py | 14 +-
dev/provider_packages/README.md | 2 +-
dev/refresh_images.sh | 4 +-
docs/build_docs.py | 6 +-
images/breeze/output-breeze-commands.svg | 136 +
images/breeze/output-build-docs.svg | 168 +
images/breeze/output-build-image.svg | 162 +
images/breeze/output-build-prod-image.svg | 175 +
images/breeze/output-cleanup.svg | 136 +
images/breeze/output-commands.svg | 165 +
images/breeze/output-config.svg | 138 +
images/breeze/output-self-upgrade.svg | 133 +
images/breeze/output-setup-autocomplete.svg | 138 +
images/breeze/output-shell.svg | 151 +
images/breeze/output-start-airflow.svg | 153 +
images/breeze/output-static-checks.svg | 158 +
images/breeze/output-stop.svg | 136 +
images/breeze/output-version.svg | 132 +
.../ci/pre_commit/pre_commit_breeze_cmd_line.py | 82 +
.../ci/pre_commit/pre_commit_breeze_cmd_line.sh | 82 -
scripts/tools/check_if_limited_dependencies.py | 52 +
scripts/tools/confirm | 14 +-
63 files changed, 3031 insertions(+), 6059 deletions(-)
diff --git a/.github/workflows/build-images.yml b/.github/workflows/build-images.yml
index a6091c5c86..a6ff039ebc 100644
--- a/.github/workflows/build-images.yml
+++ b/.github/workflows/build-images.yml
@@ -225,7 +225,7 @@ jobs:
- name: "Free space"
run: airflow-freespace
- name: "Build CI image ${{ matrix.python-version }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- run: Breeze2 build-image
+ run: breeze build-image
env:
GITHUB_REGISTRY_PULL_IMAGE_TAG: "latest"
GITHUB_REGISTRY_PUSH_IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 38824665a2..308ca20946 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -329,7 +329,7 @@ jobs:
run: airflow-freespace
if: needs.build-info.outputs.inWorkflowBuild == 'true'
- name: "Build CI image ${{ matrix.python-version }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
- run: Breeze2 build-image
+ run: breeze build-image
if: needs.build-info.outputs.inWorkflowBuild == 'true'
env:
GITHUB_REGISTRY_PULL_IMAGE_TAG: "latest"
@@ -417,7 +417,7 @@ jobs:
run-new-breeze-tests:
timeout-minutes: 10
- name: Breeze2 tests
+ name: breeze tests
runs-on: ${{ fromJson(needs.build-info.outputs.runsOn) }}
needs: [build-info]
steps:
diff --git a/.gitignore b/.gitignore
index f22c530e2c..a47b9ba269 100644
--- a/.gitignore
+++ b/.gitignore
@@ -222,5 +222,5 @@ pip-wheel-metadata
# Generated UI licenses
licenses/LICENSES-ui.txt
-# Packaged Breeze2 on Windows
-/Breeze2.exe
+# Packaged breeze on Windows
+/breeze.exe
diff --git a/.gitpod.yml b/.gitpod.yml
index e1afcf55b0..b4115c3801 100644
--- a/.gitpod.yml
+++ b/.gitpod.yml
@@ -20,7 +20,7 @@
# Docs: https://www.gitpod.io/docs/config-gitpod-file/
tasks:
- - init: ./breeze -y
+ - init: ./breeze-legacy -y
- name: Install pre-commit
openMode: split-right
command: |
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 7fe3358058..9ba107dcd6 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -95,7 +95,7 @@ repos:
- id: insert-license
name: Add license for all shell files
exclude: ^\.github/.*$|^airflow/_vendor/|^dev/breeze/autocomplete/.*$
- files: ^breeze$|^breeze-complete$|\.bash$|\.sh$
+ files: ^breeze-legacy$|^breeze-complete$|\.bash$|\.sh$
args:
- --comment-style
- "|#|"
@@ -266,7 +266,7 @@ repos:
The word(s) should be in lowercase." && exec codespell "$@"' --
language: python
types: [text]
- exclude: ^airflow/_vendor/|^CHANGELOG\.txt$|^airflow/www/static/css/material-icons\.css$
+ exclude: ^airflow/_vendor/|^CHANGELOG\.txt$|^airflow/www/static/css/material-icons\.css$|^images/.*$
args:
- --ignore-words=docs/spelling_wordlist.txt
- --skip=docs/*/commits.rst,airflow/providers/*/*.rst,*.lock,INTHEWILD.md,*.min.js,docs/apache-airflow/pipeline_example.csv
@@ -320,11 +320,12 @@ repos:
files: ^airflow/providers/.*\.py$
additional_dependencies: ['rich']
- id: update-breeze-file
- name: Update output of breeze command in BREEZE.rst
- entry: ./scripts/ci/pre_commit/pre_commit_breeze_cmd_line.sh
- language: system
- files: ^BREEZE\.rst$|^breeze$|^breeze-complete$|^Dockerfile$
+ name: Update output of breeze commands in BREEZE.rst
+ entry: ./scripts/ci/pre_commit/pre_commit_breeze_cmd_line.py
+ language: python
+ files: ^BREEZE\.rst$|^dev/breeze/.*$
pass_filenames: false
+ additional_dependencies: ['rich']
- id: update-local-yml-file
name: Update mounts in the local yml file
entry: ./scripts/ci/pre_commit/pre_commit_local_yml_mounts.sh
@@ -525,7 +526,7 @@ repos:
name: Check Shell scripts syntax correctness
language: docker_image
entry: koalaman/shellcheck:v0.7.2 -x -a
- files: ^breeze$|^breeze-complete$|\.sh$|^hooks/build$|^hooks/push$|\.bash$
+ files: ^breeze-legacy$|^breeze-complete$|\.sh$|^hooks/build$|^hooks/push$|\.bash$
exclude: ^dev/breeze/autocomplete/.*$
- id: stylelint
name: stylelint
@@ -591,6 +592,14 @@ repos:
files: ^dev/breeze/setup.*$|^dev/breeze/pyproject.toml$|^dev/breeze/README.md$
pass_filenames: false
require_serial: true
+ - id: limit-breeze-dependencies
+ name: Breeze should have small number of top-level dependencies
+ language: python
+ entry: ./scripts/tools/check_if_limited_dependencies.py
+ files: ^dev/breeze/.*$
+ pass_filenames: false
+ require_serial: true
+ additional_dependencies: ['click', 'rich']
- id: check-system-tests
name: Check if system tests have required segments of code
entry: ./scripts/ci/pre_commit/pre_commit_check_system_tests.py
diff --git a/BREEZE.rst b/BREEZE.rst
index 03608ee6f6..4f6c69d3b4 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -31,15 +31,24 @@ Airflow Breeze is an easy-to-use development and test environment using
`Docker Compose <https://docs.docker.com/compose/>`_.
The environment is available for local use and is also used in Airflow's CI tests.
-We called it *Airflow Breeze* as **It's a Breeze to contribute to Airflow**.
+We call it *Airflow Breeze* as **It's a Breeze to contribute to Airflow**.
The advantages and disadvantages of using the Breeze environment vs. other ways of testing Airflow
are described in `CONTRIBUTING.rst <CONTRIBUTING.rst#integration-test-development-environment>`_.
-All the output from the last ./breeze command is automatically logged to the ``logs/breeze.out`` file.
+.. note::
+ We are currently migrating old Bash-based ./breeze-legacy to the Python-based breeze. Some of the
+ commands are already converted to breeze, but some old commands should use breeze-legacy. The
+ documentation mentions when ``./breeze-legacy`` is involved.
+
+ The new ``breeze`` after installing is available on your PATH and you should launch it simply as
+ ``breeze <COMMAND> <FLAGS>``. Previously you had to prepend breeze with ``./`` but this is not needed
+ any more. For convenience, we will keep ``./breeze`` script for a while to run the new breeze and you
+ can still use the legacy Breeze with ``./breeze-legacy``.
Watch the video below about Airflow Breeze. It explains the motivation for Breeze
-and screencasts all its uses.
+and screencast all its uses. The video describes old ``./breeze-legacy`` (in video it still
+called ``./breeze`` ).
.. raw:: html
@@ -68,7 +77,7 @@ Docker Desktop
See also `Docker for Mac - Space <https://docs.docker.com/docker-for-mac/space>`_ for details
on increasing disk space available for Docker on Mac.
- **Docker problems**: Sometimes it is not obvious that space is an issue when you run into
- a problem with Docker. If you see a weird behaviour, try ``breeze cleanup-image`` command.
+ a problem with Docker. If you see a weird behaviour, try ``breeze cleanup`` command.
Also see `pruning <https://docs.docker.com/config/pruning/>`_ instructions from Docker.
Here is an example configuration with more than 200GB disk space for Docker:
@@ -126,11 +135,11 @@ Docker in WSL 2
WSL 2 can consume a lot of memory under the process name "Vmmem". To reclaim the memory after
development you can:
- * On the Linux distro clear cached memory: ``sudo sysctl -w vm.drop_caches=3``
- * If no longer using Docker you can quit Docker Desktop
- (right click system try icon and select "Quit Docker Desktop")
- * If no longer using WSL you can shut it down on the Windows Host
- with the following command: ``wsl --shutdown``
+ * On the Linux distro clear cached memory: ``sudo sysctl -w vm.drop_caches=3``
+ * If no longer using Docker you can quit Docker Desktop
+ (right click system try icon and select "Quit Docker Desktop")
+ * If no longer using WSL you can shut it down on the Windows Host
+ with the following command: ``wsl --shutdown``
- **Developing in WSL 2**:
You can use all the standard Linux command line utilities to develop on WSL 2.
@@ -138,50 +147,25 @@ Docker in WSL 2
If VS Code is installed on the Windows host system then in the WSL Linux Distro
you can run ``code .`` in the root directory of you Airflow repo to launch VS Code.
-Getopt and gstat
-----------------
-
-* For Linux, run ``apt install util-linux coreutils`` or an equivalent if your system is not Debian-based.
-* For macOS, install GNU ``getopt`` and ``gstat`` utilities to get Airflow Breeze running.
-
- Run ``brew install gnu-getopt coreutils``.
-
-.. warning::
- Pay attention to the ``brew install`` command and follow instructions to link the gnu-getopt version
- to become the first one on the PATH. Make sure to re-login after you make the suggested changes.
-
-**Examples:**
-
-If you use bash, run this command and re-login:
-
-.. code-block:: bash
-
- echo 'export PATH="$(brew --prefix)/opt/gnu-getopt/bin:$PATH"' >> ~/.bash_profile
- . ~/.bash_profile
+The pipx tool
+--------------
+We are using ``pipx`` tool to install and manage Breeze. The ``pipx`` tool is created by the creators
+of ``pip`` from `Python Packaging Authority <https://www.pypa.io/en/latest/>`_
-If you use zsh, run this command and re-login:
+Install pipx
.. code-block:: bash
- echo 'export PATH="$(brew --prefix)/opt/gnu-getopt/bin:$PATH"' >> ~/.zprofile
- . ~/.zprofile
+ pip install --user pipx
-
-Confirm that ``getopt`` and ``gstat`` utilities are successfully installed
+Breeze, is not globally accessible until your PATH is updated. Add <USER FOLDER>\.local\bin as a variable
+environments. This can be done automatically by the following command (follow instructions printed).
.. code-block:: bash
- $ getopt --version
- getopt from util-linux *
- $ gstat --version
- stat (GNU coreutils) *
- Copyright (C) 2020 Free Software Foundation, Inc.
- License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
- This is free software: you are free to change and redistribute it.
- There is NO WARRANTY, to the extent permitted by law.
+ pipx ensurepath
- Written by Michael Meskes.
Resources required
==================
@@ -202,7 +186,7 @@ Disk
Minimum 40GB free disk space is required for your Docker Containers.
-On Mac OS This might deteriorate over time so you might need to increase it or run ``docker system --prune``
+On Mac OS This might deteriorate over time so you might need to increase it or run ``breeze cleanup``
periodically. For details see
`Docker for Mac - Advanced tab <https://docs.docker.com/v17.12/docker-for-mac/#advanced-tab>`_.
@@ -218,9 +202,9 @@ them, you may end up with some unused image data.
To clean up the Docker environment:
-1. Stop Breeze with ``./breeze stop``. (If Breeze is already running)
+1. Stop Breeze with ``breeze stop``. (If Breeze is already running)
-2. Run the ``docker system prune`` command.
+2. Run the ``breeze cleanup`` command.
3. Run ``docker images --all`` and ``docker ps --all`` to verify that your Docker is clean.
@@ -236,14 +220,56 @@ In case of disk space errors on macOS, increase the disk space available for Doc
Installation
============
-Installation is as easy as checking out Airflow repository and running Breeze command.
-You enter the Breeze test environment by running the ``./breeze`` script. You can run it with
-the ``help`` command to see the list of available options. See `Breeze Command-Line Interface Reference`_
-for details.
+Run this command to install Breeze (make sure to use ``-e`` flag):
+
+.. code-block:: bash
+
+ pipx install -e ./dev/breeze
+
+Once this is complete, you should have ``breeze`` binary on your PATH and available to run by ``breeze``
+command.
+
+Those are all available commands for Breeze and details about the commands are described below:
+
+.. image:: ./images/breeze/output-commands.svg
+ :width: 100%
+ :alt: Breeze commands
+
+Breeze installed this way is linked to your checked out sources of Airflow so Breeze will
+automatically use latest version of sources from ``./dev/breeze``. Sometimes, when dependencies are
+updated ``breeze`` commands with offer you to ``self-upgrade`` (you just need to answer ``y`` when asked).
+
+You can always run such self-upgrade at any time:
.. code-block:: bash
- ./breeze
+ breeze self-upgrade
+
+Those are all available flags of ``self-upgrade`` command:
+
+.. image:: ./images/breeze/output-self-upgrade.svg
+ :width: 100%
+ :alt: Breeze self-upgrade
+
+If you have several checked out Airflow sources, Breeze will warn you if you are using it from a different
+source tree and will offer you to re-install from those sources - to make sure that you are using the right
+version.
+
+By default Breeze works on the version of Airflow that you run it in - in case you are outside of the
+sources of Airflow and you installed Breeze from a directory - Breeze will be run on Airflow sources from
+where it was installed.
+
+You can run ``breeze version`` command to see where breeze installed from and what are the current sources
+that Breeze works on
+
+Those are all available flags of ``version`` command:
+
+.. image:: ./images/breeze/output-version.svg
+ :width: 100%
+ :alt: Breeze version
+
+Running Breeze for the first time
+=================================
The First time you run Breeze, it pulls and builds a local version of Docker images.
It pulls the latest Airflow CI images from the
@@ -254,20 +280,31 @@ minutes on a fast connection to start. Subsequent runs should be much faster.
Once you enter the environment, you are dropped into bash shell of the Airflow container and you can
run tests immediately.
-To use the full potential of breeze you should set up autocomplete and you can
-add the checked-out Airflow repository to your PATH to run Breeze without the ``./`` and from any directory.
-
-The ``breeze`` command comes with a built-in bash/zsh autocomplete setup command. After installing, when you
-start typing the command, you can use <TAB> to show all the available switches and get
+To use the full potential of breeze you should set up autocomplete. The ``breeze`` command comes
+with a built-in bash/zsh/fish autocomplete setup command. After installing,
+when you start typing the command, you can use <TAB> to show all the available switches and get
auto-completion on typical values of parameters that you can use.
You should set up the autocomplete option automatically by running:
.. code-block:: bash
- ./breeze setup-autocomplete
+ breeze setup-autocomplete
+
+You get the auto-completion working when you re-enter the shell (follow the instructions printed).
+The command will warn you and not reinstall autocomplete if you already did, but you can
+also force reinstalling the autocomplete via:
+
+.. code-block:: bash
+
+ breeze setup-autocomplete --force
+
+Those are all available flags of ``setu-autocomplete`` command:
+
+.. image:: ./images/breeze/output-setup-autocomplete.svg
+ :width: 100%
+ :alt: Breeze setup autocomplete
-You get the auto-completion working when you re-enter the shell.
Customize your environment
--------------------------
@@ -288,7 +325,6 @@ so you can change it at any place, and run
inside container, to enable modified tmux configurations.
-
.. raw:: html
<div align="center">
@@ -305,6 +341,9 @@ Breeze helps with running tests in the same environment/way as CI tests are run.
types of tests while you enter Breeze CI interactive environment - this is described in detail
in `<TESTING.rst>`_
+Here is the part of Breeze video which is relevant (note that it refers to the old ``./breeze-legacy``
+command and it is not yet available in the new ``breeze`` command):
+
.. raw:: html
<div align="center">
@@ -326,7 +365,7 @@ follows:
.. code-block:: bash
- ./breeze --python 3.7 --backend mysql
+ breeze --python 3.7 --backend mysql
The choices you make are persisted in the ``./.build/`` cache directory so that next time when you use the
``breeze`` script, it could use the values that were used previously. This way you do not have to specify
@@ -335,6 +374,9 @@ default settings.
The defaults when you run the Breeze environment are Python 3.7 version and SQLite database.
+Here is the part of Breeze video which is relevant (note that it refers to the old ``./breeze-legacy``
+command but it is very similar to current ``breeze`` command):
+
.. raw:: html
<div align="center">
@@ -344,6 +386,32 @@ The defaults when you run the Breeze environment are Python 3.7 version and SQLi
</a>
</div>
+Those are all available flags of ``config`` command:
+
+.. image:: ./images/breeze/output-config.svg
+ :width: 100%
+ :alt: Breeze config
+
+Starting complete Airflow installation
+======================================
+
+For testing Airflow oyou often want to start multiple components (in multiple terminals). Breeze has
+built-in ``start-airflow`` command that start breeze container, launches multiple terminals using tmux
+and launches all Airflow necessary components in those terminals.
+
+You can also use it to start any released version of Airflow from ``PyPI`` with the
+``--use-airflow-version`` flag.
+
+.. code-block:: bash
+
+ breeze --python 3.7 --backend mysql --use-airflow-version 2.2.5 start-airflow
+
+Those are all available flags of ``start-airflow`` command:
+
+.. image:: ./images/breeze/output-start-airflow.svg
+ :width: 100%
+ :alt: Breeze start-airflow
+
Troubleshooting
===============
@@ -351,13 +419,13 @@ Troubleshooting
If you are having problems with the Breeze environment, try the steps below. After each step you
can check whether your problem is fixed.
-1. If you are on macOS, check if you have enough disk space for Docker.
-2. Restart Breeze with ``./breeze restart``.
-3. Delete the ``.build`` directory and run ``./breeze build-image``.
-4. Clean up Docker images via ``breeze cleanup-image`` command.
+1. If you are on macOS, check if you have enough disk space for Docker (Breeze will warn you if not).
+2. Stop Breeze with ``breeze stop``.
+3. Delete the ``.build`` directory and run ``breeze build-image``.
+4. Clean up Docker images via ``breeze cleanup`` command.
5. Restart your Docker Engine and try again.
6. Restart your machine and try again.
-7. Re-install Docker CE and try again.
+7. Re-install Docker Desktop and try again.
In case the problems are not solved, you can set the VERBOSE_COMMANDS variable to "true":
@@ -386,30 +454,28 @@ Regular development tasks:
* Enter interactive shell in CI container when ``shell`` (or no command) is specified
* Start containerised, development-friendly airflow installation with ``breeze start-airflow`` command
* Build documentation with ``breeze build-docs`` command
-* Initialize local virtualenv with ``breeze initialize-local-virtualenv`` command
+* Initialize local virtualenv with ``./breeze-legacy initialize-local-virtualenv`` command
* Build CI docker image with ``breeze build-image`` command
-* Cleanup CI docker image with ``breeze cleanup-image`` command
+* Cleanup breeze with ``breeze cleanup`` command
* Run static checks with autocomplete support ``breeze static-check`` command
-* Run test specified with ``breeze tests`` command
+* Run test specified with ``./breeze-legacy tests`` command
Additional management tasks:
-* Join running interactive shell with ``breeze exec`` command
+* Join running interactive shell with ``./breeze-legacy exec`` command
* Stop running interactive environment with ``breeze stop`` command
-* Restart running interactive environment with ``breeze restart`` command
* Execute arbitrary command in the test environment with ``breeze shell`` command
-* Execute arbitrary docker-compose command with ``breeze docker-compose`` command
+* Execute arbitrary docker-compose command with ``./breeze-legacy docker-compose`` command
Kubernetes tests related:
-* Manage KinD Kubernetes cluster and deploy Airflow to KinD cluster ``breeze kind-cluster`` commands
-* Run Kubernetes tests specified with ``breeze kind-cluster tests`` command
-* Enter the interactive kubernetes test environment with ``breeze kind-cluster shell`` command
+* Manage KinD Kubernetes cluster and deploy Airflow to KinD cluster ``./breeze-legacy kind-cluster`` commands
+* Run Kubernetes tests specified with ``./breeze-legacy kind-cluster tests`` command
+* Enter the interactive kubernetes test environment with ``./breeze-legacy kind-cluster shell`` command
-Airflow can also be used for managing Production images (with ``--production-image`` flag added for image
-related command) - this is a development-only feature, regular users of Airflow should use ``docker build``
-commands to manage the images as described in the user documentation about
-`building the image <https://airflow.apache.org/docs/docker-stack/build.html>`_
+Airflow can also be used for managing Production images - this is a development-only feature,
+regular users of Airflow should use ``docker build`` commands to manage the images as described
+in the user documentation about `building the image <https://airflow.apache.org/docs/docker-stack/build.html>`_
Maintainer tasks
----------------
@@ -417,11 +483,12 @@ Maintainer tasks
Maintainers also can use Breeze for other purposes (those are commands that regular contributors likely
do not need):
-* Prepare cache for CI: ``breeze prepare-build-cache`` (needs buildx plugin and write access to cache ghcr.io)
-* Generate constraints with ``breeze generate-constraints`` (needed when conflicting changes are merged)
-* Prepare airflow packages: ``breeze prepare-airflow-packages`` (when releasing Airflow)
-* Prepare provider documentation ``breeze prepare-provider-documentation`` and prepare provider packages
- ``breeze prepare-provider-packages`` (when releasing provider packages)
+* Prepare cache for CI: ``breeze build-image --prepare-build-cache`` and
+ ``breeze build-prod image --prepare-build-cache``(needs buildx plugin and write access to registry ghcr.io)
+* Generate constraints with ``./breeze-legacy generate-constraints`` (needed when conflicting changes are merged)
+* Prepare airflow packages: ``./breeze-legacy prepare-airflow-packages`` (when releasing Airflow)
+* Prepare provider documentation ``./breeze-legacy prepare-provider-documentation`` and prepare provider packages
+ ``./breeze-legacy prepare-provider-packages`` (when releasing provider packages)
Details of Breeze usage
=======================
@@ -430,11 +497,30 @@ Database volumes in Breeze
--------------------------
Breeze keeps data for all it's integration in named docker volumes. Each backend and integration
-keeps data in their own volume. Those volumes are persisted until ``./breeze stop`` command or
-``./breeze restart`` command is run. You can also preserve the volumes by adding flag
-``--preserve-volumes`` when you run either of those commands. Then, next time when you start
-``Breeze``, it will have the data pre-populated. You can always delete the volumes by
-running ``./breeze stop`` without the ``--preserve-volumes`` flag.
+keeps data in their own volume. Those volumes are persisted until ``breeze stop`` command.
+You can also preserve the volumes by adding flag ``--preserve-volumes`` when you run the command.
+Then, next time when you start Breeze, it will have the data pre-populated.
+
+Those are all available flags of ``stop`` command:
+
+.. image:: ./images/breeze/output-stop.svg
+ :width: 100%
+ :alt: Breeze stop
+
+Image cleanup
+--------------
+
+Breeze uses docker images heavily and those images are rebuild periodically. This might cause extra
+disk usage by the images. If you need to clean-up the images periodically you can run
+``breeze cleanup`` command (by default it will skip removing your images before cleaning up but you
+can also remove the images to clean-up everything by adding ``--include-current-images``).
+
+Those are all available flags of ``cleanup`` command:
+
+
+.. image:: ./images/breeze/output-cleanup.svg
+ :width: 100%
+ :alt: Breeze cleanup
Launching multiple terminals
----------------------------
@@ -448,6 +534,9 @@ capability of creating multiple virtual terminals and multiplex between them. Mo
found at `tmux GitHub wiki page <https://github.com/tmux/tmux/wiki>`_ . Tmux has several useful shortcuts
that allow you to split the terminals, open new tabs etc - it's pretty useful to learn it.
+Here is the part of Breeze video which is relevant (note that it refers to the old ``./breeze-legacy``
+command but it is very similar to current ``breeze`` command):
+
.. raw:: html
<div align="center">
@@ -460,10 +549,13 @@ that allow you to split the terminals, open new tabs etc - it's pretty useful to
Another way is to exec into Breeze terminal from the host's terminal. Often you can
have multiple terminals in the host (Linux/MacOS/WSL2 on Windows) and you can simply use those terminals
-to enter the running container. It's as easy as launching ``breeze exec`` while you already started the
+to enter the running container. It's as easy as launching ``./breeze-legacy exec`` while you already started the
Breeze environment. You will be dropped into bash and environment variables will be read in the same
way as when you enter the environment. You can do it multiple times and open as many terminals as you need.
+Here is the part of Breeze video which is relevant (note that it refers to the old ``./breeze-legacy``
+command and it is not yet available in the current ``breeze`` command):
+
.. raw:: html
<div align="center">
@@ -521,6 +613,9 @@ Once integration is started, it will continue to run until the environment is st
Note that running integrations uses significant resources - CPU and memory.
+Here is the part of Breeze video which is relevant (note that it refers to the old ``./breeze-legacy``
+command but it is very similar to current ``breeze`` command):
+
.. raw:: html
<div align="center">
@@ -543,6 +638,9 @@ The CI image is built automatically as needed, however it can be rebuilt manuall
image should be built manually - but also a variant of this image is built automatically when
kubernetes tests are executed see `Running Kubernetes tests <#running-kubernetes-tests>`_
+Here is the part of Breeze video which is relevant (note that it refers to the old ``./breeze-legacy``
+command but it is very similar to current ``breeze`` command):
+
.. raw:: html
<div align="center">
@@ -562,10 +660,11 @@ latest image build on CI. This might happen when for example latest patches have
Python images or when significant changes are made in the Dockerfile. In such cases, Breeze will
download the latest images before rebuilding because this is usually faster than rebuilding the image.
-In most cases, rebuilding an image requires network connectivity (for example, to download new
-dependencies). If you work offline and do not want to rebuild the images when needed, you can set the
-``FORCE_ANSWER_TO_QUESTIONS`` variable to ``no`` as described in the
-`Setting default behaviour for user interaction <#setting-default-behaviour-for-user-interaction>`_ section.
+Those are all available flags of ``build-image`` command:
+
+.. image:: ./images/breeze/output-build-image.svg
+ :width: 100%
+ :alt: Breeze build-image
Preparing packages
------------------
@@ -592,7 +691,7 @@ The below example builds provider packages in the wheel format.
.. code-block:: bash
- ./breeze prepare-provider-packages
+ ./breeze-legacy prepare-provider-packages
If you run this command without packages, you will prepare all packages, you can however specify
providers that you would like to build. By default ``both`` types of packages are prepared (
@@ -600,20 +699,20 @@ providers that you would like to build. By default ``both`` types of packages ar
.. code-block:: bash
- ./breeze prepare-provider-packages google amazon
+ ./breeze-legacy prepare-provider-packages google amazon
You can see all providers available by running this command:
.. code-block:: bash
- ./breeze prepare-provider-packages -- --help
+ ./breeze-legacy prepare-provider-packages -- --help
You can also prepare airflow packages using breeze:
.. code-block:: bash
- ./breeze prepare-airflow-packages
+ ./breeze-legacy prepare-airflow-packages
This prepares airflow .whl package in the dist folder.
@@ -622,7 +721,7 @@ default is to build ``both`` type of packages ``sdist`` and ``wheel``.
.. code-block:: bash
- ./breeze prepare-airflow-packages --package-format=wheel
+ ./breeze-legacy prepare-airflow-packages --package-format=wheel
Building Production images
@@ -636,30 +735,30 @@ However in many cases you want to add your own custom version of the image - wit
python dependencies, additional Airflow extras. Breeze's ``build-image`` command helps to build your own,
customized variant of the image that contains everything you need.
-You can switch to building the production image by adding ``--production-image`` flag to the ``build_image``
-command. Note, that the images can also be built using ``docker build`` command by passing appropriate
+You can switch to building the production image by using ``build-prod-image`` command.
+Note, that the images can also be built using ``docker build`` command by passing appropriate
build-args as described in `IMAGES.rst <IMAGES.rst>`_ , but Breeze provides several flags that
-makes it easier to do it. You can see all the flags by running ``./breeze build-image --help``,
+makes it easier to do it. You can see all the flags by running ``breeze build-prod-image --help``,
but here typical examples are presented:
.. code-block:: bash
- ./breeze build-image --production-image --additional-extras "jira"
+ breeze build-prod-image --additional-extras "jira"
This installs additional ``jira`` extra while installing airflow in the image.
.. code-block:: bash
- ./breeze build-image --production-image --additional-python-deps "torchio==0.17.10"
+ breeze build-prod-image --additional-python-deps "torchio==0.17.10"
This install additional pypi dependency - torchio in specified version.
.. code-block:: bash
- ./breeze build-image --production-image --additional-dev-apt-deps "libasound2-dev" \
- --additional-runtime-apt-deps "libasound2"
+ breeze build-prod-image --additional-dev-apt-deps "libasound2-dev" \
+ --additional-runtime-apt-deps "libasound2"
This installs additional apt dependencies - ``libasound2-dev`` in the build image and ``libasound`` in the
final image. Those are development dependencies that might be needed to build and use python packages added
@@ -671,11 +770,20 @@ suffix and they need to also be paired with corresponding runtime dependency add
.. code-block:: bash
- ./breeze build-image --production-image --python 3.7 --additional-dev-deps "libasound2-dev" \
+ breeze build-prod-image --python 3.7 --additional-dev-deps "libasound2-dev" \
--additional-runtime-apt-deps "libasound2"
Same as above but uses python 3.7.
+Those are all available flags of ``build-prod-image`` command:
+
+.. image:: ./images/breeze/output-build-prod-image.svg
+ :width: 100%
+ :alt: Breeze commands
+
+Here is the part of Breeze video which is relevant (note that it refers to the old ``./breeze-legacy``
+command but it is very similar to current ``breeze`` command):
+
.. raw:: html
<div align="center">
@@ -694,18 +802,43 @@ you have auto-complete setup you should see auto-completable list of all checks
.. code-block:: bash
- ./breeze static-check mypy
+ breeze static-checks -t mypy
The above will run mypy check for currently staged files.
-You can also add arbitrary pre-commit flag after ``--``
+You can also pass specific pre-commit flags for example ``--all-files`` :
+
+.. code-block:: bash
+
+ breeze static-checks -t mypy --all-files
+
+The above will run mypy check for all files.
+
+There is also a convenience ``--last-commit`` flag that you can use to run static check on last commit only:
.. code-block:: bash
- ./breeze static-check mypy -- --all-files
+ breeze static-checks -t mypy --last-commit
The above will run mypy check for all files.
+
+If you ever need to get a list of the files that will be checked (for troubleshooting) use these commands:
+
+.. code-block:: bash
+
+ breeze static-checks -t identity --verbose # currently staged files
+ breeze static-checks -t identity --verbose --from-ref $(git merge-base main HEAD) --to-ref HEAD # branch updates
+
+Those are all available flags of ``static-checks`` command:
+
+.. image:: ./images/breeze/output-static-checks.svg
+ :width: 100%
+ :alt: Breeze static checks
+
+Here is the part of Breeze video which is relevant (note that it refers to the old ``./breeze-legacy``
+command but it is very similar to current ``breeze`` command):
+
.. raw:: html
<div align="center">
@@ -715,13 +848,6 @@ The above will run mypy check for all files.
</a>
</div>
-If you ever need to get a list of the files that will be checked (for troubleshooting when playing with
-``--from-ref`` and ``--to-ref``), use these commands:
-
-.. code-block:: bash
-
- ./breeze static-check identity --verbose # currently staged files
- ./breeze static-check identity --verbose -- --from-ref $(git merge-base main HEAD) --to-ref HEAD # branch updates
Building the Documentation
--------------------------
@@ -730,7 +856,7 @@ To build documentation in Breeze, use the ``build-docs`` command:
.. code-block:: bash
- ./breeze build-docs
+ breeze build-docs
Results of the build can be found in the ``docs/_build`` folder.
@@ -745,7 +871,7 @@ extra ``--`` flag.
.. code-block:: bash
- ./breeze build-docs -- --spellcheck-only
+ breeze build-docs --spellcheck-only
This process can take some time, so in order to make it shorter you can filter by package, using the flag
``--package-filter <PACKAGE-NAME>``. The package name has to be one of the providers or ``apache-airflow``. For
@@ -753,12 +879,21 @@ instance, for using it with Amazon, the command would be:
.. code-block:: bash
- ./breeze build-docs -- --package-filter apache-airflow-providers-amazon
+ breeze build-docs --package-filter apache-airflow-providers-amazon
Often errors during documentation generation come from the docstrings of auto-api generated classes.
During the docs building auto-api generated files are stored in the ``docs/_api`` folder. This helps you
easily identify the location the problems with documentation originated from.
+Those are all available flags of ``build-docs`` command:
+
+.. image:: ./images/breeze/output-build-docs.svg
+ :width: 100%
+ :alt: Breeze build documentation
+
+Here is the part of Breeze video which is relevant (note that it refers to the old ``./breeze-legacy``
+command but it is very similar to current ``breeze`` command):
+
.. raw:: html
<div align="center">
@@ -777,12 +912,12 @@ files are stored in separated orphan branches: ``constraints-main``, ``constrain
Those are constraint files as described in detail in the
`<CONTRIBUTING.rst#pinned-constraint-files>`_ contributing documentation.
-You can use ``./breeze generate-constraints`` command to manually generate constraints for a single python
-version and single constraint mode like this:
+You can use ``./breeze-legacy generate-constraints`` command to manually generate constraints for
+a single python version and single constraint mode like this:
.. code-block:: bash
- ./breeze generate-constraints --generate-constraints-mode pypi-providers
+ ./breeze-legacy generate-constraints --generate-constraints-mode pypi-providers
Constraints are generated separately for each python version and there are separate constraints modes:
@@ -833,7 +968,7 @@ To use your host IDE with Breeze:
.. code-block:: bash
- ./breeze initialize-local-virtualenv --python 3.8
+ ./breeze-legacy initialize-local-virtualenv --python 3.8
.. warning::
Make sure that you use the right Python version in this command - matching the Python version you have
@@ -846,6 +981,9 @@ This is a lightweight solution that has its own limitations.
More details on using the local virtualenv are available in the `LOCAL_VIRTUALENV.rst <LOCAL_VIRTUALENV.rst>`_.
+Here is the part of Breeze video which is relevant (note that it refers to the old ``./breeze-legacy``
+but it is not available in the ``breeze`` command):
+
.. raw:: html
<div align="center">
@@ -864,6 +1002,9 @@ automatically to run the tests.
This is described in detail in `Testing Kubernetes <TESTING.rst#running-tests-with-kubernetes>`_.
+Here is the part of Breeze video which is relevant (note that it refers to the old ``./breeze-legacy``
+command and it is not yet available in the current ``breeze`` command):
+
.. raw:: html
<div align="center">
@@ -881,8 +1022,16 @@ You can always stop it via:
.. code-block:: bash
- ./breeze stop
+ breeze stop
+
+Those are all available flags of ``stop`` command:
+.. image:: ./images/breeze/output-stop.svg
+ :width: 100%
+ :alt: Breeze stop
+
+Here is the part of Breeze video which is relevant (note that it refers to the old ``./breeze-legacy``
+command but it is very similar to current ``breeze`` command):
.. raw:: html
@@ -938,11 +1087,17 @@ Running Arbitrary commands in the Breeze environment
----------------------------------------------------
To run other commands/executables inside the Breeze Docker-based environment, use the
-``./breeze shell`` command. You should add your command as -c "command" after ``--`` as extra arguments.
+``breeze shell`` command.
.. code-block:: bash
- ./breeze shell -- -c "ls -la"
+ breeze shell "ls -la"
+
+Those are all available flags of ``shell`` command:
+
+.. image:: ./images/breeze/output-shell.svg
+ :width: 100%
+ :alt: Breeze shell
Running "Docker Compose" commands
---------------------------------
@@ -953,49 +1108,23 @@ after ``--`` as extra arguments.
.. code-block:: bash
- ./breeze docker-compose pull -- --ignore-pull-failures
-
-Restarting Breeze environment
------------------------------
-
-You can also restart the environment and enter it via:
-
-.. code-block:: bash
-
- ./breeze restart
-
+ ./breeze-legacy docker-compose pull -- --ignore-pull-failures
Setting default answers for user interaction
--------------------------------------------
Sometimes during the build, you are asked whether to perform an action, skip it, or quit. This happens
-when rebuilding or removing an image - actions that take a lot of time and could be potentially destructive.
+when rebuilding or removing an image and in few other cases - actions that take a lot of time
+or could be potentially destructive. You can force answer to the questions by providing an
+``--answer`` flag in the commands that support it.
-For automation scripts, you can export one of the three variables to control the default
-interaction behaviour:
+For automation scripts, you can export the ``FORCE_ANSWER_TO_QUESTIONS`` variable (and set it to
+``y``, ``n``, ``q``, ``yes``, ``no``, ``quit`` - in all case combinations).
.. code-block::
export FORCE_ANSWER_TO_QUESTIONS="yes"
-If ``FORCE_ANSWER_TO_QUESTIONS`` is set to ``yes``, the images are automatically rebuilt when needed.
-Images are deleted without asking.
-
-.. code-block::
-
- export FORCE_ANSWER_TO_QUESTIONS="no"
-
-If ``FORCE_ANSWER_TO_QUESTIONS`` is set to ``no``, the old images are used even if rebuilding is needed.
-This is useful when you work offline. Deleting images is aborted.
-
-.. code-block::
-
- export FORCE_ANSWER_TO_QUESTIONS="quit"
-
-If ``FORCE_ANSWER_TO_QUESTIONS`` is set to ``quit``, the whole script is aborted. Deleting images is aborted.
-
-If more than one variable is set, ``yes`` takes precedence over ``no``, which takes precedence over ``quit``.
-
Fixing File/Directory Ownership
-------------------------------
@@ -1120,1664 +1249,10 @@ of the ``Dockerfile.ci``. This way dependencies will be added incrementally.
Before merge, these dependencies should be moved to the appropriate ``apt-get install`` command,
which is already in the ``Dockerfile.ci``.
+Recording command output
+========================
-Breeze Command-Line Interface Reference
-=======================================
-
-Airflow Breeze Syntax
----------------------
-
-This is the current syntax for `./breeze <./breeze>`_:
-
- .. START BREEZE HELP MARKER
-
-.. code-block:: text
-
-
- ####################################################################################################
-
- usage: breeze [FLAGS] [COMMAND] -- <EXTRA_ARGS>
-
- By default the script enters the CI container and drops you to bash shell, but you can choose
- one of the commands to run specific actions instead.
-
- Add --help after each command to see details:
-
- Commands without arguments:
-
- shell [Default] Enters interactive shell in the container
- build-docs Builds documentation in the container
- build-image Builds CI or Production docker image
- prepare-build-cache Prepares CI or Production build cache
- cleanup-image Cleans up the container image created
- exec Execs into running breeze container in new terminal
- generate-constraints Generates pinned constraint files
- initialize-local-virtualenv Initializes local virtualenv
- prepare-airflow-packages Prepares airflow packages
- setup-autocomplete Sets up autocomplete for breeze
- start-airflow Starts Scheduler and Webserver and enters the shell
- stop Stops the docker-compose environment
- restart Stops the docker-compose environment including DB cleanup
- toggle-suppress-cheatsheet Toggles on/off cheatsheet
- toggle-suppress-asciiart Toggles on/off asciiart
-
- Commands with arguments:
-
- docker-compose <ARG> Executes specified docker-compose command
- kind-cluster <ARG> Manages KinD cluster on the host
- prepare-provider-documentation <ARG> Prepares provider packages documentation
- prepare-provider-packages <ARG> Prepares provider packages
- static-check <ARG> Performs selected static check for changed files
- tests <ARG> Runs selected tests in the container
-
- Help commands:
-
- flags Shows all breeze's flags
- help Shows this help message
- help-all Shows detailed help for all commands and flags
-
- ####################################################################################################
-
- Detailed usage
-
- ####################################################################################################
-
-
- Detailed usage for command: shell
-
-
- breeze shell [FLAGS] [-- <EXTRA_ARGS>]
-
- This is default subcommand if no subcommand is used.
-
- Enters interactive shell where you can run all tests, start Airflow webserver, scheduler,
- workers, interact with the database, run DAGs etc. It is the default command if no command
- is selected. The shell is executed in the container and in case integrations are chosen,
- the integrations will be started as separated docker containers - under the docker-compose
- supervision. Local sources are by default mounted to within the container so you can edit
- them locally and run tests immediately in the container. Several folders ('files', 'dist')
- are also mounted so that you can exchange files between the host and container.
-
- The 'files/airflow-breeze-config/variables.env' file can contain additional variables
- and setup. This file is automatically sourced when you enter the container. Database
- and webserver ports are forwarded to appropriate database/webserver so that you can
- connect to it from your host environment.
-
- You can also pass <EXTRA_ARGS> after -- they will be passed as bash parameters, this is
- especially useful to pass bash options, for example -c to execute command:
-
- 'breeze shell -- -c "ls -la"'
- 'breeze -- -c "ls -la"'
-
- For GitHub repository, the --github-repository flag can be used to specify the repository
- to pull and push images. You can also use --github-image-id <COMMIT_SHA> in case
- you want to pull the image with specific COMMIT_SHA tag.
-
- 'breeze shell \
- --github-image-id 9a621eaa394c0a0a336f8e1b31b35eff4e4ee86e' - pull/use image with SHA
- 'breeze \
- --github-image-id 9a621eaa394c0a0a336f8e1b31b35eff4e4ee86e' - pull/use image with SHA
-
- Most flags are applicable to the shell command as it will run build when needed.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: build-docs
-
-
- breeze build-docs [-- <EXTRA_ARGS>]
-
- Builds Airflow documentation. The documentation is build inside docker container - to
- maintain the same build environment for everyone. Appropriate sources are mapped from
- the host to the container so that latest sources are used. The folders where documentation
- is generated ('docs/_build') are also mounted to the container - this way results of
- the documentation build is available in the host.
-
- The possible extra args are: --docs-only, --spellcheck-only, --package-filter, --help
-
-
- ####################################################################################################
-
-
- Detailed usage for command: build-image
-
-
- breeze build-image [FLAGS]
-
- Builds docker image (CI or production) without entering the container. You can pass
- additional options to this command, such as:
-
- Choosing python version:
- '--python'
-
- Choosing cache option:
- '--build-cache-local' or '-build-cache-pulled', or '--build-cache-none'
-
- Choosing whether to force pull images or force build the image:
- '--force-build-image'
-
- You can also pass '--production-image' flag to build production image rather than CI image.
-
- For GitHub repository, the '--github-repository' can be used to choose repository
- to pull/push images.
-
- Flags:
-
- -p, --python PYTHON_MAJOR_MINOR_VERSION
- Python version used for the image. This is always major/minor version.
-
- One of:
-
- 3.7 3.8 3.9 3.10
-
- --platform PLATFORM
- Builds image for the platform specified.
-
- One of:
-
- linux/amd64 linux/arm64 linux/amd64,linux/arm64
-
-
- -d, --debian DEBIAN_VERSION
- Debian version used for the image. This is always name of the debian distribution version.
-
- One of:
-
- bullseye buster
-
- -a, --install-airflow-version INSTALL_AIRFLOW_VERSION
- Uses different version of Airflow when building PROD image.
-
- 2.0.2 2.0.1 2.0.0 wheel sdist
-
- -t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
- Installs Airflow directly from reference in GitHub when building PROD image.
- This can be a GitHub branch like main or v2-2-test, or a tag like 2.2.0rc1.
-
- --installation-method INSTALLATION_METHOD
- Method of installing Airflow in PROD image - either from the sources ('.')
- or from package 'apache-airflow' to install from PyPI.
- Default in Breeze is to install from sources. One of:
-
- . apache-airflow
-
- --upgrade-to-newer-dependencies
- Upgrades PIP packages to latest versions available without looking at the constraints.
-
- -I, --production-image
- Use production image for entering the environment and builds (not for tests).
-
- -F, --force-build-images
- Forces building of the local docker images. The images are rebuilt
- automatically for the first time or when changes are detected in
- package-related files, but you can force it using this flag.
-
- --cleanup-docker-context-files
- Removes whl and tar.gz files created in docker-context-files before running the command.
- In case there are some files there it unnecessarily increases the context size and
- makes the COPY . always invalidated - if you happen to have those files when you build your
- image.
-
- Customization options:
-
- -E, --extras EXTRAS
- Extras to pass to build images The default are different for CI and production images:
-
- CI image:
- devel_ci
-
- Production image:
- amazon,async,celery,cncf.kubernetes,dask,docker,elasticsearch,ftp,google,google_auth,
- grpc,hashicorp,http,ldap,microsoft.azure,mysql,odbc,pandas,postgres,redis,sendgrid,
- sftp,slack,ssh,statsd,virtualenv
-
- --image-tag TAG
- Additional tag in the image.
-
- --disable-pypi-when-building
- Disable installing Airflow from pypi when building. If you use this flag and want
- to install Airflow, you have to install it from packages placed in
- 'docker-context-files' and use --install-from-docker-context-files flag.
-
- --additional-extras ADDITIONAL_EXTRAS
- Additional extras to pass to build images The default is no additional extras.
-
- --additional-python-deps ADDITIONAL_PYTHON_DEPS
- Additional python dependencies to use when building the images.
-
- --dev-apt-command DEV_APT_COMMAND
- The basic command executed before dev apt deps are installed.
-
- --additional-dev-apt-command ADDITIONAL_DEV_APT_COMMAND
- Additional command executed before dev apt deps are installed.
-
- --additional-dev-apt-deps ADDITIONAL_DEV_APT_DEPS
- Additional apt dev dependencies to use when building the images.
-
- --dev-apt-deps DEV_APT_DEPS
- The basic apt dev dependencies to use when building the images.
-
- --additional-dev-apt-deps ADDITIONAL_DEV_DEPS
- Additional apt dev dependencies to use when building the images.
-
- --additional-dev-apt-envs ADDITIONAL_DEV_APT_ENVS
- Additional environment variables set when adding dev dependencies.
-
- --runtime-apt-command RUNTIME_APT_COMMAND
- The basic command executed before runtime apt deps are installed.
-
- --additional-runtime-apt-command ADDITIONAL_RUNTIME_APT_COMMAND
- Additional command executed before runtime apt deps are installed.
-
- --runtime-apt-deps ADDITIONAL_RUNTIME_APT_DEPS
- The basic apt runtime dependencies to use when building the images.
-
- --additional-runtime-apt-deps ADDITIONAL_RUNTIME_DEPS
- Additional apt runtime dependencies to use when building the images.
-
- --additional-runtime-apt-envs ADDITIONAL_RUNTIME_APT_DEPS
- Additional environment variables set when adding runtime dependencies.
-
- Build options:
-
- --disable-mysql-client-installation
- Disables installation of the mysql client which might be problematic if you are building
- image in controlled environment. Only valid for production image.
-
- --disable-mssql-client-installation
- Disables installation of the mssql client which might be problematic if you are building
- image in controlled environment. Only valid for production image.
-
- --constraints-location
- Url to the constraints file. In case of the production image it can also be a path to the
- constraint file placed in 'docker-context-files' folder, in which case it has to be
- in the form of '/docker-context-files/<NAME_OF_THE_FILE>'
-
- --disable-pip-cache
- Disables GitHub PIP cache during the build. Useful if GitHub is not reachable during build.
-
- --install-from-docker-context-files
- This flag is used during image building. If it is used additionally to installing
- Airflow from PyPI, the packages are installed from the .whl and .tar.gz packages placed
- in the 'docker-context-files' folder. The same flag can be used during entering the image in
- the CI image - in this case also the .whl and .tar.gz files will be installed automatically
-
- -C, --force-clean-images
- Force build images with cache disabled. This will remove the pulled or build images
- and start building images from scratch. This might take a long time.
-
- -r, --skip-rebuild-check
- Skips checking image for rebuilds. It will use whatever image is available locally/pulled.
-
- -L, --build-cache-local
- Uses local cache to build images. No pulled images will be used, but results of local
- builds in the Docker cache are used instead. This will take longer than when the pulled
- cache is used for the first time, but subsequent '--build-cache-local' builds will be
- faster as they will use mostly the locally build cache.
-
- This is default strategy used by the Production image builds.
-
- -U, --build-cache-pulled
- Uses images pulled from GitHub Container Registry to build images.
- Those builds are usually faster than when ''--build-cache-local'' with the exception if
- the registry images are not yet updated. The images are updated after successful merges
- to main.
-
- This is default strategy used by the CI image builds.
-
- -X, --build-cache-disabled
- Disables cache during docker builds. This is useful if you want to make sure you want to
- rebuild everything from scratch.
-
- This strategy is used by default for both Production and CI images for the scheduled
- (nightly) builds in CI.
-
- -g, --github-repository GITHUB_REPOSITORY
- GitHub repository used to pull, push images.
- Default: apache/airflow.
-
- -v, --verbose
- Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
- debugging - when you run breeze with --verbose flags you will be able to see the commands
- executed under the hood and copy&paste them to your terminal to debug them more easily.
-
- Note that you can further increase verbosity and see all the commands executed by breeze
- by running 'export VERBOSE_COMMANDS="true"' before running breeze.
-
- --dry-run-docker
- Only show docker commands to execute instead of actually executing them. The docker
- commands are printed in yellow color.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: prepare-build-cache
-
-
- breeze prepare-build-cache [FLAGS]
-
- Prepares build cache (CI or production) without entering the container. You can pass
- additional options to this command, such as:
-
- Choosing python version:
- '--python'
-
- You can also pass '--production-image' flag to build production image rather than CI image.
-
- For GitHub repository, the '--github-repository' can be used to choose repository
- to pull/push images. Cleanup docker context files and pull cache are forced. This command
- requires buildx to be installed.
-
- Flags:
-
- -p, --python PYTHON_MAJOR_MINOR_VERSION
- Python version used for the image. This is always major/minor version.
-
- One of:
-
- 3.7 3.8 3.9 3.10
-
- --platform PLATFORM
- Builds image for the platform specified.
-
- One of:
-
- linux/amd64 linux/arm64 linux/amd64,linux/arm64
-
-
- -d, --debian DEBIAN_VERSION
- Debian version used for the image. This is always name of the debian distribution version.
-
- One of:
-
- bullseye buster
-
- -a, --install-airflow-version INSTALL_AIRFLOW_VERSION
- Uses different version of Airflow when building PROD image.
-
- 2.0.2 2.0.1 2.0.0 wheel sdist
-
- -t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
- Installs Airflow directly from reference in GitHub when building PROD image.
- This can be a GitHub branch like main or v2-2-test, or a tag like 2.2.0rc1.
-
- --installation-method INSTALLATION_METHOD
- Method of installing Airflow in PROD image - either from the sources ('.')
- or from package 'apache-airflow' to install from PyPI.
- Default in Breeze is to install from sources. One of:
-
- . apache-airflow
-
- --upgrade-to-newer-dependencies
- Upgrades PIP packages to latest versions available without looking at the constraints.
-
- -I, --production-image
- Use production image for entering the environment and builds (not for tests).
-
- -g, --github-repository GITHUB_REPOSITORY
- GitHub repository used to pull, push images.
- Default: apache/airflow.
-
- -v, --verbose
- Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
- debugging - when you run breeze with --verbose flags you will be able to see the commands
- executed under the hood and copy&paste them to your terminal to debug them more easily.
-
- Note that you can further increase verbosity and see all the commands executed by breeze
- by running 'export VERBOSE_COMMANDS="true"' before running breeze.
-
- --dry-run-docker
- Only show docker commands to execute instead of actually executing them. The docker
- commands are printed in yellow color.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: cleanup-image
-
-
- breeze cleanup-image [FLAGS]
-
- Removes the breeze-related images created in your local docker image cache. This will
- not reclaim space in docker cache. You need to 'docker system prune' (optionally
- with --all) to reclaim that space.
-
- Flags:
-
- -p, --python PYTHON_MAJOR_MINOR_VERSION
- Python version used for the image. This is always major/minor version.
-
- One of:
-
- 3.7 3.8 3.9 3.10
-
- --platform PLATFORM
- Builds image for the platform specified.
-
- One of:
-
- linux/amd64 linux/arm64 linux/amd64,linux/arm64
-
-
- -d, --debian DEBIAN_VERSION
- Debian version used for the image. This is always name of the debian distribution version.
-
- One of:
-
- bullseye buster
-
- -I, --production-image
- Use production image for entering the environment and builds (not for tests).
-
- -v, --verbose
- Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
- debugging - when you run breeze with --verbose flags you will be able to see the commands
- executed under the hood and copy&paste them to your terminal to debug them more easily.
-
- Note that you can further increase verbosity and see all the commands executed by breeze
- by running 'export VERBOSE_COMMANDS="true"' before running breeze.
-
- --dry-run-docker
- Only show docker commands to execute instead of actually executing them. The docker
- commands are printed in yellow color.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: exec
-
-
- breeze exec [-- <EXTRA_ARGS>]
-
- Execs into interactive shell to an already running container. The container mus be started
- already by breeze shell command. If you are not familiar with tmux, this is the best
- way to run multiple processes in the same container at the same time for example scheduler,
- webserver, workers, database console and interactive terminal.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: generate-constraints
-
-
- breeze generate-constraints [FLAGS]
-
- Generates pinned constraint files with all extras from setup.py. Those files are generated in
- files folder - separate files for different python version. Those constraint files when
- pushed to orphan constraints-main, constraints-2-0 branches are used
- to generate repeatable CI test runs as well as run repeatable production image builds and
- upgrades when you want to include installing or updating some of the released providers
- released at the time particular airflow version was released. You can use those
- constraints to predictably install released Airflow versions. This is mainly used to test
- the constraint generation or manually fix them - constraints are pushed to the orphan
- branches by a successful scheduled CRON job in CI automatically, but sometimes manual fix
- might be needed.
-
- Flags:
-
- --generate-constraints-mode GENERATE_CONSTRAINTS_MODE
- Mode of generating constraints - determines whether providers are installed when generating
- constraints and which version of them (either the ones from sources are used or the ones
- from pypi.
-
- One of:
-
- source-providers pypi-providers no-providers
-
- -p, --python PYTHON_MAJOR_MINOR_VERSION
- Python version used for the image. This is always major/minor version.
-
- One of:
-
- 3.7 3.8 3.9 3.10
-
- --platform PLATFORM
- Builds image for the platform specified.
-
- One of:
-
- linux/amd64 linux/arm64 linux/amd64,linux/arm64
-
-
- -d, --debian DEBIAN_VERSION
- Debian version used for the image. This is always name of the debian distribution version.
-
- One of:
-
- bullseye buster
-
- -v, --verbose
- Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
- debugging - when you run breeze with --verbose flags you will be able to see the commands
- executed under the hood and copy&paste them to your terminal to debug them more easily.
-
- Note that you can further increase verbosity and see all the commands executed by breeze
- by running 'export VERBOSE_COMMANDS="true"' before running breeze.
-
- --dry-run-docker
- Only show docker commands to execute instead of actually executing them. The docker
- commands are printed in yellow color.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: initialize-local-virtualenv
-
-
- breeze initialize-local-virtualenv [FLAGS]
-
- Initializes locally created virtualenv installing all dependencies of Airflow
- taking into account the constraints for the version specified.
- This local virtualenv can be used to aid auto-completion and IDE support as
- well as run unit tests directly from the IDE. You need to have virtualenv
- activated before running this command.
-
- Flags:
-
- -p, --python PYTHON_MAJOR_MINOR_VERSION
- Python version used for the image. This is always major/minor version.
-
- One of:
-
- 3.7 3.8 3.9 3.10
-
- --platform PLATFORM
- Builds image for the platform specified.
-
- One of:
-
- linux/amd64 linux/arm64 linux/amd64,linux/arm64
-
-
- -d, --debian DEBIAN_VERSION
- Debian version used for the image. This is always name of the debian distribution version.
-
- One of:
-
- bullseye buster
-
-
- ####################################################################################################
-
-
- Detailed usage for command: prepare-airflow-packages
-
-
- breeze prepare-airflow-packages [FLAGS]
-
- Prepares airflow packages (sdist and wheel) in dist folder. Note that
- prepare-provider-packages command cleans up the dist folder, so if you want also
- to generate provider packages, make sure you run prepare-provider-packages first,
- and prepare-airflow-packages second. You can specify optional
- --version-suffix-for-pypi flag to generate rc candidates for PyPI packages.
- The packages are prepared in dist folder
-
- Examples:
-
- 'breeze prepare-airflow-packages --package-format wheel' or
- 'breeze prepare-airflow-packages --version-suffix-for-pypi rc1'
-
- Flags:
-
- --package-format PACKAGE_FORMAT
-
- Chooses format of packages to prepare.
-
- One of:
-
- both,sdist,wheel
-
- Default: both
-
- -S, --version-suffix-for-pypi SUFFIX
- Adds optional suffix to the version in the generated provider package. It can be used
- to generate rc1/rc2 ... versions of the packages to be uploaded to PyPI.
-
- -N, --version-suffix-for-svn SUFFIX
- Adds optional suffix to the generated names of package. It can be used to generate
- rc1/rc2 ... versions of the packages to be uploaded to SVN.
-
- -v, --verbose
- Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
- debugging - when you run breeze with --verbose flags you will be able to see the commands
- executed under the hood and copy&paste them to your terminal to debug them more easily.
-
- Note that you can further increase verbosity and see all the commands executed by breeze
- by running 'export VERBOSE_COMMANDS="true"' before running breeze.
-
- --dry-run-docker
- Only show docker commands to execute instead of actually executing them. The docker
- commands are printed in yellow color.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: setup-autocomplete
-
-
- breeze setup-autocomplete
-
- Sets up autocomplete for breeze commands. Once you do it you need to re-enter the bash
- shell and when typing breeze command <TAB> will provide autocomplete for
- parameters and values.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: start-airflow
-
-
- breeze start-airflow
-
- Like the Shell command this will enter the interactive shell, but it will also start
- automatically the Scheduler and the Webserver. It will leave you in a tmux session where you
- can also observe what is happening in your Airflow.
-
- This is a convenient way to setup a development environment. Your dags will be loaded from the
- folder 'files/dags' on your host machine (it could take some times).
-
- If you want to load default connections and example dags you can use the dedicated flags.
-
- Flags:
-
- --use-airflow-version AIRFLOW_SPECIFICATION
- In CI image, installs Airflow at runtime from PIP released version or using
- the installation method specified (sdist, wheel, none). When 'none' is used,
- airflow is just removed. In this case airflow package should be added to dist folder
- and --use-packages-from-dist flag should be used.
-
- 2.0.2 2.0.1 2.0.0 wheel sdist none
-
- --use-packages-from-dist
- In CI image, if specified it will look for packages placed in dist folder and
- it will install the packages after entering the image.
- This is useful for testing provider packages.
-
- --load-example-dags
- Include Airflow example dags.
-
- --load-default-connections
- Include Airflow Default Connections.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: stop
-
-
- breeze stop
-
- Brings down running docker compose environment. When you start the environment, the docker
- containers will continue running so that startup time is shorter. But they take quite a lot of
- memory and CPU. This command stops all running containers from the environment.
-
- Flags:
-
- --preserve-volumes
- Use this flag if you would like to preserve data volumes from the databases used
- by the integrations. By default, those volumes are deleted, so when you run 'stop'
- or 'restart' commands you start from scratch, but by using this flag you can
- preserve them. If you want to delete those volumes after stopping Breeze, just
- run the 'breeze stop' again without this flag.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: restart
-
-
- breeze restart [FLAGS]
-
- Restarts running docker compose environment. When you restart the environment, the docker
- containers will be restarted. That includes cleaning up the databases. This is
- especially useful if you switch between different versions of Airflow.
-
- Flags:
-
- --preserve-volumes
- Use this flag if you would like to preserve data volumes from the databases used
- by the integrations. By default, those volumes are deleted, so when you run 'stop'
- or 'restart' commands you start from scratch, but by using this flag you can
- preserve them. If you want to delete those volumes after stopping Breeze, just
- run the 'breeze stop' again without this flag.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: toggle-suppress-cheatsheet
-
-
- breeze toggle-suppress-cheatsheet
-
- Toggles on/off cheatsheet displayed before starting bash shell.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: toggle-suppress-asciiart
-
-
- breeze toggle-suppress-asciiart
-
- Toggles on/off asciiart displayed before starting bash shell.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: docker-compose
-
-
- breeze docker-compose [FLAGS] COMMAND [-- <EXTRA_ARGS>]
-
- Run docker-compose command instead of entering the environment. Use 'help' as command
- to see available commands. The <EXTRA_ARGS> passed after -- are treated
- as additional options passed to docker-compose. For example
-
- 'breeze docker-compose pull -- --ignore-pull-failures'
-
- Flags:
-
- -p, --python PYTHON_MAJOR_MINOR_VERSION
- Python version used for the image. This is always major/minor version.
-
- One of:
-
- 3.7 3.8 3.9 3.10
-
- --platform PLATFORM
- Builds image for the platform specified.
-
- One of:
-
- linux/amd64 linux/arm64 linux/amd64,linux/arm64
-
-
- -d, --debian DEBIAN_VERSION
- Debian version used for the image. This is always name of the debian distribution version.
-
- One of:
-
- bullseye buster
-
- -b, --backend BACKEND
- Backend to use for tests - it determines which database is used.
- One of:
-
- sqlite mysql postgres mssql
-
- Default: sqlite
-
- --postgres-version POSTGRES_VERSION
- Postgres version used. One of:
-
- 10 11 12 13
-
- --mysql-version MYSQL_VERSION
- MySql version used. One of:
-
- 5.7 8
-
- --mssql-version MSSQL_VERSION
- MSSql version used. One of:
-
- 2017-latest 2019-latest
-
- -v, --verbose
- Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
- debugging - when you run breeze with --verbose flags you will be able to see the commands
- executed under the hood and copy&paste them to your terminal to debug them more easily.
-
- Note that you can further increase verbosity and see all the commands executed by breeze
- by running 'export VERBOSE_COMMANDS="true"' before running breeze.
-
- --dry-run-docker
- Only show docker commands to execute instead of actually executing them. The docker
- commands are printed in yellow color.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: kind-cluster
-
-
- breeze kind-cluster [FLAGS] OPERATION
-
- Manages host-side Kind Kubernetes cluster that is used to run Kubernetes integration tests.
- It allows to start/stop/restart/status the Kind Kubernetes cluster and deploy Airflow to it.
- This enables you to run tests inside the breeze environment with latest airflow images.
- Note that in case of deploying airflow, the first step is to rebuild the image and loading it
- to the cluster so you can also pass appropriate build image flags that will influence
- rebuilding the production image. Operation is one of:
-
- start stop restart status deploy test shell k9s
-
- The last two operations - shell and k9s allow you to perform interactive testing with
- kubernetes tests. You can enter the shell from which you can run kubernetes tests and in
- another terminal you can start the k9s CLI to debug kubernetes instance. It is an easy
- way to debug the kubernetes deployments.
-
- You can read more about k9s at https://k9scli.io/
-
- Flags:
-
- -p, --python PYTHON_MAJOR_MINOR_VERSION
- Python version used for the image. This is always major/minor version.
-
- One of:
-
- 3.7 3.8 3.9 3.10
-
- --platform PLATFORM
- Builds image for the platform specified.
-
- One of:
-
- linux/amd64 linux/arm64 linux/amd64,linux/arm64
-
-
- -d, --debian DEBIAN_VERSION
- Debian version used for the image. This is always name of the debian distribution version.
-
- One of:
-
- bullseye buster
-
- -F, --force-build-images
- Forces building of the local docker images. The images are rebuilt
- automatically for the first time or when changes are detected in
- package-related files, but you can force it using this flag.
-
- --cleanup-docker-context-files
- Removes whl and tar.gz files created in docker-context-files before running the command.
- In case there are some files there it unnecessarily increases the context size and
- makes the COPY . always invalidated - if you happen to have those files when you build your
- image.
-
- Customization options:
-
- -E, --extras EXTRAS
- Extras to pass to build images The default are different for CI and production images:
-
- CI image:
- devel_ci
-
- Production image:
- amazon,async,celery,cncf.kubernetes,dask,docker,elasticsearch,ftp,google,google_auth,
- grpc,hashicorp,http,ldap,microsoft.azure,mysql,odbc,pandas,postgres,redis,sendgrid,
- sftp,slack,ssh,statsd,virtualenv
-
- --image-tag TAG
- Additional tag in the image.
-
- --disable-pypi-when-building
- Disable installing Airflow from pypi when building. If you use this flag and want
- to install Airflow, you have to install it from packages placed in
- 'docker-context-files' and use --install-from-docker-context-files flag.
-
- --additional-extras ADDITIONAL_EXTRAS
- Additional extras to pass to build images The default is no additional extras.
-
- --additional-python-deps ADDITIONAL_PYTHON_DEPS
- Additional python dependencies to use when building the images.
-
- --dev-apt-command DEV_APT_COMMAND
- The basic command executed before dev apt deps are installed.
-
- --additional-dev-apt-command ADDITIONAL_DEV_APT_COMMAND
- Additional command executed before dev apt deps are installed.
-
- --additional-dev-apt-deps ADDITIONAL_DEV_APT_DEPS
- Additional apt dev dependencies to use when building the images.
-
- --dev-apt-deps DEV_APT_DEPS
- The basic apt dev dependencies to use when building the images.
-
- --additional-dev-apt-deps ADDITIONAL_DEV_DEPS
- Additional apt dev dependencies to use when building the images.
-
- --additional-dev-apt-envs ADDITIONAL_DEV_APT_ENVS
- Additional environment variables set when adding dev dependencies.
-
- --runtime-apt-command RUNTIME_APT_COMMAND
- The basic command executed before runtime apt deps are installed.
-
- --additional-runtime-apt-command ADDITIONAL_RUNTIME_APT_COMMAND
- Additional command executed before runtime apt deps are installed.
-
- --runtime-apt-deps ADDITIONAL_RUNTIME_APT_DEPS
- The basic apt runtime dependencies to use when building the images.
-
- --additional-runtime-apt-deps ADDITIONAL_RUNTIME_DEPS
- Additional apt runtime dependencies to use when building the images.
-
- --additional-runtime-apt-envs ADDITIONAL_RUNTIME_APT_DEPS
- Additional environment variables set when adding runtime dependencies.
-
- Build options:
-
- --disable-mysql-client-installation
- Disables installation of the mysql client which might be problematic if you are building
- image in controlled environment. Only valid for production image.
-
- --disable-mssql-client-installation
- Disables installation of the mssql client which might be problematic if you are building
- image in controlled environment. Only valid for production image.
-
- --constraints-location
- Url to the constraints file. In case of the production image it can also be a path to the
- constraint file placed in 'docker-context-files' folder, in which case it has to be
- in the form of '/docker-context-files/<NAME_OF_THE_FILE>'
-
- --disable-pip-cache
- Disables GitHub PIP cache during the build. Useful if GitHub is not reachable during build.
-
- --install-from-docker-context-files
- This flag is used during image building. If it is used additionally to installing
- Airflow from PyPI, the packages are installed from the .whl and .tar.gz packages placed
- in the 'docker-context-files' folder. The same flag can be used during entering the image in
- the CI image - in this case also the .whl and .tar.gz files will be installed automatically
-
- -C, --force-clean-images
- Force build images with cache disabled. This will remove the pulled or build images
- and start building images from scratch. This might take a long time.
-
- -r, --skip-rebuild-check
- Skips checking image for rebuilds. It will use whatever image is available locally/pulled.
-
- -L, --build-cache-local
- Uses local cache to build images. No pulled images will be used, but results of local
- builds in the Docker cache are used instead. This will take longer than when the pulled
- cache is used for the first time, but subsequent '--build-cache-local' builds will be
- faster as they will use mostly the locally build cache.
-
- This is default strategy used by the Production image builds.
-
- -U, --build-cache-pulled
- Uses images pulled from GitHub Container Registry to build images.
- Those builds are usually faster than when ''--build-cache-local'' with the exception if
- the registry images are not yet updated. The images are updated after successful merges
- to main.
-
- This is default strategy used by the CI image builds.
-
- -X, --build-cache-disabled
- Disables cache during docker builds. This is useful if you want to make sure you want to
- rebuild everything from scratch.
-
- This strategy is used by default for both Production and CI images for the scheduled
- (nightly) builds in CI.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: prepare-provider-documentation
-
-
- breeze prepare-provider-documentation [FLAGS] [PACKAGE_ID ...]
-
- Prepares documentation files for provider packages.
-
- The command is optionally followed by the list of packages to generate readme for.
- If the first parameter is not formatted as a date, then today is regenerated.
- If no packages are specified, readme for all packages are generated.
- If no date is specified, current date + 3 days is used (allowing for PMC votes to pass).
-
- Examples:
-
- 'breeze prepare-provider-documentation' or
- 'breeze prepare-provider-documentation --version-suffix-for-pypi rc1'
-
- General form:
-
- 'breeze prepare-provider-documentation <PACKAGE_ID> ...'
-
- * <PACKAGE_ID> is usually directory in the airflow/providers folder (for example
- 'google' but in several cases, it might be one level deeper separated with
- '.' for example 'apache.hive'
-
- Flags:
-
- -S, --version-suffix-for-pypi SUFFIX
- Adds optional suffix to the version in the generated provider package. It can be used
- to generate rc1/rc2 ... versions of the packages to be uploaded to PyPI.
-
- -N, --version-suffix-for-svn SUFFIX
- Adds optional suffix to the generated names of package. It can be used to generate
- rc1/rc2 ... versions of the packages to be uploaded to SVN.
-
- --package-format PACKAGE_FORMAT
-
- Chooses format of packages to prepare.
-
- One of:
-
- both,sdist,wheel
-
- Default: both
-
- --non-interactive
-
- Runs the command in non-interactive mode.
-
- --generate-providers-issue
-
- Generate providers issue that should be created.
-
- -v, --verbose
- Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
- debugging - when you run breeze with --verbose flags you will be able to see the commands
- executed under the hood and copy&paste them to your terminal to debug them more easily.
-
- Note that you can further increase verbosity and see all the commands executed by breeze
- by running 'export VERBOSE_COMMANDS="true"' before running breeze.
-
- --dry-run-docker
- Only show docker commands to execute instead of actually executing them. The docker
- commands are printed in yellow color.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: prepare-provider-packages
-
-
- breeze prepare-provider-packages [FLAGS] [PACKAGE_ID ...]
-
- Prepares provider packages. You can provide (after --) optional list of packages to prepare.
- If no packages are specified, readme for all packages are generated. You can specify optional
- --version-suffix-for-svn flag to generate rc candidate packages to upload to SVN or
- --version-suffix-for-pypi flag to generate rc candidates for PyPI packages. You can also
- provide both suffixes in case you prepare alpha/beta versions. The packages are prepared in
- dist folder. Note that this command also cleans up dist folder before generating the packages
- so that you do not have accidental files there. This will delete airflow package if it is
- prepared there so make sure you run prepare-provider-packages first,
- and prepare-airflow-packages second.
-
- Examples:
-
- 'breeze prepare-provider-packages' or
- 'breeze prepare-provider-packages google' or
- 'breeze prepare-provider-packages --package-format wheel google' or
- 'breeze prepare-provider-packages --version-suffix-for-svn rc1 http google amazon' or
- 'breeze prepare-provider-packages --version-suffix-for-pypi rc1 http google amazon'
- 'breeze prepare-provider-packages --version-suffix-for-pypi a1
- --version-suffix-for-svn a1 http google amazon'
-
- General form:
-
- 'breeze prepare-provider-packages [--package-format PACKAGE_FORMAT] \
- [--version-suffix-for-svn|--version-suffix-for-pypi] <PACKAGE_ID> ...'
-
- * <PACKAGE_ID> is usually directory in the airflow/providers folder (for example
- 'google'), but in several cases, it might be one level deeper separated with '.'
- for example 'apache.hive'
-
- Flags:
-
- --package-format PACKAGE_FORMAT
-
- Chooses format of packages to prepare.
-
- One of:
-
- both,sdist,wheel
-
- Default: both
-
- -S, --version-suffix-for-pypi SUFFIX
- Adds optional suffix to the version in the generated provider package. It can be used
- to generate rc1/rc2 ... versions of the packages to be uploaded to PyPI.
-
- -N, --version-suffix-for-svn SUFFIX
- Adds optional suffix to the generated names of package. It can be used to generate
- rc1/rc2 ... versions of the packages to be uploaded to SVN.
-
- -v, --verbose
- Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
- debugging - when you run breeze with --verbose flags you will be able to see the commands
- executed under the hood and copy&paste them to your terminal to debug them more easily.
-
- Note that you can further increase verbosity and see all the commands executed by breeze
- by running 'export VERBOSE_COMMANDS="true"' before running breeze.
-
- --dry-run-docker
- Only show docker commands to execute instead of actually executing them. The docker
- commands are printed in yellow color.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: static-check
-
-
- breeze static-check [FLAGS] static_check [-- <EXTRA_ARGS>]
-
- Run selected static checks for currently changed files. You should specify static check that
- you would like to run or 'all' to run all checks. One of:
-
- all airflow-config-yaml airflow-providers-available airflow-provider-yaml-files-ok
- autoflake base-operator black blacken-docs boring-cyborg build
- build-providers-dependencies chart-schema-lint capitalized-breeze
- changelog-duplicates check-2-1-compatibility check-apache-license
- check-builtin-literals check-executables-have-shebangs check-extras-order
- check-hooks-apply check-integrations check-merge-conflict check-revision-heads-map
- check-xml check-system-tests daysago-import-check debug-statements
- detect-private-key docstring-params doctoc dont-use-safe-filter end-of-file-fixer
- fix-encoding-pragma flake8 flynt codespell forbid-tabs helm-lint identity
- incorrect-use-of-LoggingMixin insert-license inline-dockerfile-scripts isort
- json-schema language-matters lint-dockerfile lint-openapi markdownlint mermaid
- migration-reference mixed-line-ending mypy mypy-helm no-providers-in-core-examples
- no-relative-imports persist-credentials-disabled pre-commit-descriptions
- pre-commit-hook-names pretty-format-json provide-create-sessions
- providers-changelogs providers-init-file providers-subpackages-init-file
- provider-yamls pydevd pydocstyle python-no-log-warn pyupgrade restrict-start_date
- rst-backticks setup-order setup-extra-packages shellcheck sort-in-the-wild
- sort-spelling-wordlist stylelint trailing-whitespace ui-lint
- update-breeze-config-hash update-breeze-file update-extras update-local-yml-file
- update-setup-cfg-file update-supported-versions update-versions
- vendor-k8s-json-schema verify-db-migrations-documented version-sync www-lint
- yamllint yesqa
-
- You can pass extra arguments including options to the pre-commit framework as
- <EXTRA_ARGS> passed after --. For example:
-
- 'breeze static-check mypy' or
- 'breeze static-check mypy -- --files tests/core.py'
- 'breeze static-check mypy -- --all-files'
-
- To check all files that differ between you current branch and main run:
-
- 'breeze static-check all -- --from-ref $(git merge-base main HEAD) --to-ref HEAD'
-
- To check all files that are in the HEAD commit run:
-
- 'breeze static-check mypy -- --from-ref HEAD^ --to-ref HEAD'
-
-
- You can see all the options by adding --help EXTRA_ARG:
-
- 'breeze static-check mypy -- --help'
-
-
- ####################################################################################################
-
-
- Detailed usage for command: tests
-
-
- breeze tests [FLAGS] [TEST_TARGET ..] [-- <EXTRA_ARGS>]
-
- Run the specified unit test target. There might be multiple
- targets specified separated with comas. The <EXTRA_ARGS> passed after -- are treated
- as additional options passed to pytest. You can pass 'tests' as target to
- run all tests. For example:
-
- 'breeze tests tests/core/test_core.py -- --logging-level=DEBUG'
- 'breeze tests tests
-
- Flags:
-
- --test-type TEST_TYPE
- Type of the test to run. One of:
-
- All,Always,Core,Providers,API,CLI,Integration,Other,WWW,Postgres,MySQL,Helm,
- Quarantined
-
- Default: All
-
-
- ####################################################################################################
-
-
- Detailed usage for command: flags
-
-
- Explains in detail all the flags that can be used with breeze.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: help
-
-
- breeze help
-
- Shows general help message for all commands.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: help-all
-
-
- breeze help-all
-
- Shows detailed help for all commands and flags.
-
-
- ####################################################################################################
-
-
- ####################################################################################################
-
- Summary of all flags supported by Breeze:
-
- ****************************************************************************************************
- Choose Airflow variant
-
- -p, --python PYTHON_MAJOR_MINOR_VERSION
- Python version used for the image. This is always major/minor version.
-
- One of:
-
- 3.7 3.8 3.9 3.10
-
- --platform PLATFORM
- Builds image for the platform specified.
-
- One of:
-
- linux/amd64 linux/arm64 linux/amd64,linux/arm64
-
-
- -d, --debian DEBIAN_VERSION
- Debian version used for the image. This is always name of the debian distribution version.
-
- One of:
-
- bullseye buster
-
- ****************************************************************************************************
- Choose backend to run for Airflow
-
- -b, --backend BACKEND
- Backend to use for tests - it determines which database is used.
- One of:
-
- sqlite mysql postgres mssql
-
- Default: sqlite
-
- --postgres-version POSTGRES_VERSION
- Postgres version used. One of:
-
- 10 11 12 13
-
- --mysql-version MYSQL_VERSION
- MySql version used. One of:
-
- 5.7 8
-
- --mssql-version MSSQL_VERSION
- MSSql version used. One of:
-
- 2017-latest 2019-latest
-
- ****************************************************************************************************
- Enable production image
-
- -I, --production-image
- Use production image for entering the environment and builds (not for tests).
-
- ****************************************************************************************************
- Additional actions executed while entering breeze
-
- -d, --db-reset
- Resets the database at entry to the environment. It will drop all the tables
- and data and recreate the DB from scratch even if 'restart' command was not used.
- Combined with 'restart' command it enters the environment in the state that is
- ready to start Airflow webserver/scheduler/worker. Without the switch, the database
- does not have any tables and you need to run reset db manually.
-
- -i, --integration INTEGRATION
- Integration to start during tests - it determines which integrations are started
- for integration tests. There can be more than one integration started, or all to
- start all integrations. Selected integrations are not saved for future execution.
- One of:
-
- cassandra kerberos mongo openldap pinot rabbitmq redis statsd trino all
-
- --init-script INIT_SCRIPT_FILE
- Initialization script name - Sourced from files/airflow-breeze-config. Default value
- init.sh. It will be executed after the environment is configured and started.
-
- ****************************************************************************************************
- Additional actions executed while starting Airflow
-
- --load-example-dags
- Include Airflow example dags.
-
- --load-default-connections
- Include Airflow Default Connections.
-
- ****************************************************************************************************
- Cleanup options when stopping Airflow
-
- --preserve-volumes
- Use this flag if you would like to preserve data volumes from the databases used
- by the integrations. By default, those volumes are deleted, so when you run 'stop'
- or 'restart' commands you start from scratch, but by using this flag you can
- preserve them. If you want to delete those volumes after stopping Breeze, just
- run the 'breeze stop' again without this flag.
-
- ****************************************************************************************************
- Kind kubernetes and Kubernetes tests configuration(optional)
-
- Configuration for the KinD Kubernetes cluster and tests:
-
- -K, --kubernetes-mode KUBERNETES_MODE
- Kubernetes mode - only used in case one of kind-cluster commands is used.
- One of:
-
- image
-
- Default: image
-
- -V, --kubernetes-version KUBERNETES_VERSION
- Kubernetes version - only used in case one of kind-cluster commands is used.
- One of:
-
- v1.23.4 v1.22.7 v1.21.10 v1.20.15
-
- Default: v1.23.4
-
- --kind-version KIND_VERSION
- Kind version - only used in case one of kind-cluster commands is used.
- One of:
-
- v0.12.0
-
- Default: v0.12.0
-
- --helm-version HELM_VERSION
- Helm version - only used in case one of kind-cluster commands is used.
- One of:
-
- v3.6.3
-
- Default: v3.6.3
-
- --executor EXECUTOR
- Executor to use in a kubernetes cluster.
- One of:
-
- KubernetesExecutor CeleryExecutor LocalExecutor CeleryKubernetesExecutor
- LocalKubernetesExecutor
-
- Default: KubernetesExecutor
-
- ****************************************************************************************************
- Manage mounting local files
-
- -l, --skip-mounting-local-sources
- Skips mounting local volume with sources - you get exactly what is in the
- docker image rather than your current local sources of Airflow.
-
- ****************************************************************************************************
- Assume answers to questions
-
- -y, --assume-yes
- Assume 'yes' answer to all questions.
-
- -n, --assume-no
- Assume 'no' answer to all questions.
-
- -q, --assume-quit
- Assume 'quit' answer to all questions.
-
- ****************************************************************************************************
- Install different Airflow version during PROD image build
-
- -a, --install-airflow-version INSTALL_AIRFLOW_VERSION
- Uses different version of Airflow when building PROD image.
-
- 2.0.2 2.0.1 2.0.0 wheel sdist
-
- -t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
- Installs Airflow directly from reference in GitHub when building PROD image.
- This can be a GitHub branch like main or v2-2-test, or a tag like 2.2.0rc1.
-
- --installation-method INSTALLATION_METHOD
- Method of installing Airflow in PROD image - either from the sources ('.')
- or from package 'apache-airflow' to install from PyPI.
- Default in Breeze is to install from sources. One of:
-
- . apache-airflow
-
- --upgrade-to-newer-dependencies
- Upgrades PIP packages to latest versions available without looking at the constraints.
-
- ****************************************************************************************************
- Use different Airflow version at runtime in CI image
-
- --use-airflow-version AIRFLOW_SPECIFICATION
- In CI image, installs Airflow at runtime from PIP released version or using
- the installation method specified (sdist, wheel, none). When 'none' is used,
- airflow is just removed. In this case airflow package should be added to dist folder
- and --use-packages-from-dist flag should be used.
-
- 2.0.2 2.0.1 2.0.0 wheel sdist none
-
- --use-packages-from-dist
- In CI image, if specified it will look for packages placed in dist folder and
- it will install the packages after entering the image.
- This is useful for testing provider packages.
-
- ****************************************************************************************************
- Credentials
-
- -f, --forward-credentials
- Forwards host credentials to docker container. Use with care as it will make
- your credentials available to everything you install in Docker.
-
- ****************************************************************************************************
- Flags for building Docker images (both CI and production)
-
- -F, --force-build-images
- Forces building of the local docker images. The images are rebuilt
- automatically for the first time or when changes are detected in
- package-related files, but you can force it using this flag.
-
- --cleanup-docker-context-files
- Removes whl and tar.gz files created in docker-context-files before running the command.
- In case there are some files there it unnecessarily increases the context size and
- makes the COPY . always invalidated - if you happen to have those files when you build your
- image.
-
- Customization options:
-
- -E, --extras EXTRAS
- Extras to pass to build images The default are different for CI and production images:
-
- CI image:
- devel_ci
-
- Production image:
- amazon,async,celery,cncf.kubernetes,dask,docker,elasticsearch,ftp,google,google_auth,
- grpc,hashicorp,http,ldap,microsoft.azure,mysql,odbc,pandas,postgres,redis,sendgrid,
- sftp,slack,ssh,statsd,virtualenv
-
- --image-tag TAG
- Additional tag in the image.
-
- --disable-pypi-when-building
- Disable installing Airflow from pypi when building. If you use this flag and want
- to install Airflow, you have to install it from packages placed in
- 'docker-context-files' and use --install-from-docker-context-files flag.
-
- --additional-extras ADDITIONAL_EXTRAS
- Additional extras to pass to build images The default is no additional extras.
-
- --additional-python-deps ADDITIONAL_PYTHON_DEPS
- Additional python dependencies to use when building the images.
-
- --dev-apt-command DEV_APT_COMMAND
- The basic command executed before dev apt deps are installed.
-
- --additional-dev-apt-command ADDITIONAL_DEV_APT_COMMAND
- Additional command executed before dev apt deps are installed.
-
- --additional-dev-apt-deps ADDITIONAL_DEV_APT_DEPS
- Additional apt dev dependencies to use when building the images.
-
- --dev-apt-deps DEV_APT_DEPS
- The basic apt dev dependencies to use when building the images.
-
- --additional-dev-apt-deps ADDITIONAL_DEV_DEPS
- Additional apt dev dependencies to use when building the images.
-
- --additional-dev-apt-envs ADDITIONAL_DEV_APT_ENVS
- Additional environment variables set when adding dev dependencies.
-
- --runtime-apt-command RUNTIME_APT_COMMAND
- The basic command executed before runtime apt deps are installed.
-
- --additional-runtime-apt-command ADDITIONAL_RUNTIME_APT_COMMAND
- Additional command executed before runtime apt deps are installed.
-
- --runtime-apt-deps ADDITIONAL_RUNTIME_APT_DEPS
- The basic apt runtime dependencies to use when building the images.
-
- --additional-runtime-apt-deps ADDITIONAL_RUNTIME_DEPS
- Additional apt runtime dependencies to use when building the images.
-
- --additional-runtime-apt-envs ADDITIONAL_RUNTIME_APT_DEPS
- Additional environment variables set when adding runtime dependencies.
-
- Build options:
-
- --disable-mysql-client-installation
- Disables installation of the mysql client which might be problematic if you are building
- image in controlled environment. Only valid for production image.
-
- --disable-mssql-client-installation
- Disables installation of the mssql client which might be problematic if you are building
- image in controlled environment. Only valid for production image.
-
- --constraints-location
- Url to the constraints file. In case of the production image it can also be a path to the
- constraint file placed in 'docker-context-files' folder, in which case it has to be
- in the form of '/docker-context-files/<NAME_OF_THE_FILE>'
-
- --disable-pip-cache
- Disables GitHub PIP cache during the build. Useful if GitHub is not reachable during build.
-
- --install-from-docker-context-files
- This flag is used during image building. If it is used additionally to installing
- Airflow from PyPI, the packages are installed from the .whl and .tar.gz packages placed
- in the 'docker-context-files' folder. The same flag can be used during entering the image in
- the CI image - in this case also the .whl and .tar.gz files will be installed automatically
-
- -C, --force-clean-images
- Force build images with cache disabled. This will remove the pulled or build images
- and start building images from scratch. This might take a long time.
-
- -r, --skip-rebuild-check
- Skips checking image for rebuilds. It will use whatever image is available locally/pulled.
-
- -L, --build-cache-local
- Uses local cache to build images. No pulled images will be used, but results of local
- builds in the Docker cache are used instead. This will take longer than when the pulled
- cache is used for the first time, but subsequent '--build-cache-local' builds will be
- faster as they will use mostly the locally build cache.
-
- This is default strategy used by the Production image builds.
-
- -U, --build-cache-pulled
- Uses images pulled from GitHub Container Registry to build images.
- Those builds are usually faster than when ''--build-cache-local'' with the exception if
- the registry images are not yet updated. The images are updated after successful merges
- to main.
-
- This is default strategy used by the CI image builds.
-
- -X, --build-cache-disabled
- Disables cache during docker builds. This is useful if you want to make sure you want to
- rebuild everything from scratch.
-
- This strategy is used by default for both Production and CI images for the scheduled
- (nightly) builds in CI.
-
- ****************************************************************************************************
- Flags for pulling/pushing Docker images (both CI and production)
-
- -g, --github-repository GITHUB_REPOSITORY
- GitHub repository used to pull, push images.
- Default: apache/airflow.
-
-
-
-
- -s, --github-image-id COMMIT_SHA
- <COMMIT_SHA> of the image. Images in GitHub registry are stored with those
- to be able to easily find the image for particular CI runs. Once you know the
- <COMMIT_SHA>, you can specify it in github-image-id flag and Breeze will
- automatically pull and use that image so that you can easily reproduce a problem
- that occurred in CI.
-
- Default: latest.
-
- ****************************************************************************************************
- Flags for running tests
-
- --test-type TEST_TYPE
- Type of the test to run. One of:
-
- All,Always,Core,Providers,API,CLI,Integration,Other,WWW,Postgres,MySQL,Helm,
- Quarantined
-
- Default: All
-
- ****************************************************************************************************
- Flags for generation of the provider packages
-
- -S, --version-suffix-for-pypi SUFFIX
- Adds optional suffix to the version in the generated provider package. It can be used
- to generate rc1/rc2 ... versions of the packages to be uploaded to PyPI.
-
- -N, --version-suffix-for-svn SUFFIX
- Adds optional suffix to the generated names of package. It can be used to generate
- rc1/rc2 ... versions of the packages to be uploaded to SVN.
-
- ****************************************************************************************************
- Increase verbosity of the scripts
-
- -v, --verbose
- Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
- debugging - when you run breeze with --verbose flags you will be able to see the commands
- executed under the hood and copy&paste them to your terminal to debug them more easily.
-
- Note that you can further increase verbosity and see all the commands executed by breeze
- by running 'export VERBOSE_COMMANDS="true"' before running breeze.
-
- --dry-run-docker
- Only show docker commands to execute instead of actually executing them. The docker
- commands are printed in yellow color.
-
- ****************************************************************************************************
- Print detailed help message
-
- -h, --help
- Shows detailed help message for the command specified.
-
- .. END BREEZE HELP MARKER
+Breeze uses built-in capability of ``rich`` to record and print the command help as an ``svg`` file.
+It's enabled by setting ``RECORD_BREEZE_OUTPUT_FILE`` to a file name where it will be recorded.
+By default it records the screenshots with default characters width and with "Breeze screenshot" title,
+but you can override it with ``RECORD_BREEZE_WIDTH`` and ``RECORD_BREEZE_TITLE`` variables respectively.
diff --git a/Breeze2 b/Breeze2
deleted file mode 100755
index 8da7ee869d..0000000000
--- a/Breeze2
+++ /dev/null
@@ -1,42 +0,0 @@
-#!/usr/bin/env bash
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-set -euo pipefail
-
-MY_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
-
-BREEZE_BINARY=Breeze2
-
-function check_breeze_installed() {
- if ! command -v ${BREEZE_BINARY} &> /dev/null; then
- echo
- echo "You need to install pipx and run:"
- echo
- echo " pipx install -e '${MY_DIR}/dev/breeze/'"
- echo
- echo
- if "${MY_DIR}/scripts/tools/confirm" "Installing pipx and Breeze?"; then
- python -m pip install pipx --upgrade
- pipx install -e "${MY_DIR}/dev/breeze/"
- ${BREEZE_BINARY} setup-autocomplete
- else
- exit
- fi
- fi
-}
-
-${BREEZE_BINARY} "${@}"
diff --git a/CI.rst b/CI.rst
index da16486224..10b1b666fc 100644
--- a/CI.rst
+++ b/CI.rst
@@ -680,7 +680,7 @@ cd27124534b46c9688a1d89e75fcd137ab5137e3, in python 3.8 environment you can run:
.. code-block:: bash
- ./breeze --github-image-id cd27124534b46c9688a1d89e75fcd137ab5137e3 --python 3.8
+ ./breeze-legacy --github-image-id cd27124534b46c9688a1d89e75fcd137ab5137e3 --python 3.8
You will be dropped into a shell with the exact version that was used during the CI run and you will
be able to run pytest tests manually, easily reproducing the environment that was used in CI. Note that in
@@ -707,13 +707,7 @@ In order to add a new version the following operations should be done (example u
.. code-block:: bash
- ./breeze build-image --python 3.10
-
-* push image as cache to GitHub:
-
-.. code-block:: bash
-
- ./breeze push-image --python 3.10
+ breeze build-image --python 3.10
* Find the 2 new images (prod, ci) created in
`GitHub Container registry <https://github.com/orgs/apache/packages?tab=packages&ecosystem=container&q=airflow>`_
diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
index f020066f67..5080424c99 100644
--- a/CONTRIBUTING.rst
+++ b/CONTRIBUTING.rst
@@ -245,7 +245,7 @@ or on macOS with `Homebrew <https://formulae.brew.sh/formula/jq>`_
.. code-block:: bash
- ./breeze
+ breeze
Breeze starts with downloading the Airflow CI image from
the Docker Hub and installing all required dependencies.
@@ -263,7 +263,7 @@ to make them immediately visible in the environment.
.. code-block:: bash
- ./breeze initialize-local-virtualenv --python 3.7
+ ./breeze-legacy initialize-local-virtualenv --python 3.7
6. Open your IDE (for example, PyCharm) and select the virtualenv you created
as the project's default virtualenv in your IDE.
@@ -554,6 +554,13 @@ All details about using and running Airflow Breeze can be found in
The Airflow Breeze solution is intended to ease your local development as "*It's
a Breeze to develop Airflow*".
+.. note::
+
+ We are in a process of switching to the new Python-based Breeze from a legacy Bash
+ Breeze. Not all functionality has been ported yet and the old Breeze is still available
+ until then as ``./breeze-legacy`` script. The documentation mentions when the old ./breeze-legacy
+ should be still used.
+
Benefits:
- Breeze is a complete environment that includes external components, such as
@@ -581,9 +588,14 @@ Limitations:
disk space and CPU. You can stop the environment manually after you use it
or even use a ``bare`` environment to decrease resource usage.
-**NOTE:** Breeze CI images are not supposed to be used in production environments.
-They are optimized for repeatability of tests, maintainability and speed of building rather
-than production performance. The production images are not yet officially published.
+
+
+.. note::
+
+ Breeze CI images are not supposed to be used in production environments.
+ They are optimized for repeatability of tests, maintainability and speed of building rather
+ than production performance. The production images are not yet officially published.
+
Airflow dependencies
diff --git a/CONTRIBUTORS_QUICK_START.rst b/CONTRIBUTORS_QUICK_START.rst
index 2b9d6367c2..2558d255c7 100644
--- a/CONTRIBUTORS_QUICK_START.rst
+++ b/CONTRIBUTORS_QUICK_START.rst
@@ -267,14 +267,13 @@ Setting up Breeze
.. code-block:: bash
- $ ./breeze setup-autocomplete
- $ source ~/.bash_completion.d/breeze-complete
+ $ breeze setup-autocomplete
3. Initialize breeze environment with required python version and backend. This may take a while for first time.
.. code-block:: bash
- $ ./breeze --python 3.8 --backend mysql
+ $ breeze --python 3.8 --backend mysql
.. note::
If you encounter an error like "docker.credentials.errors.InitializationError:
@@ -307,7 +306,7 @@ Setting up Breeze
.. code-block:: bash
- $ ./breeze stop
+ $ breeze stop
Installing airflow in the local virtual environment ``airflow-env`` with breeze.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -322,7 +321,7 @@ Installing airflow in the local virtual environment ``airflow-env`` with breeze.
.. code-block:: bash
- $ ./breeze initialize-local-virtualenv --python 3.8
+ $ ./breeze-legacy initialize-local-virtualenv --python 3.8
3. Add following line to ~/.bashrc in order to call breeze command from anywhere.
@@ -335,7 +334,7 @@ Using Breeze
~~~~~~~~~~~~
1. Starting breeze environment using ``breeze start-airflow`` starts Breeze environment with last configuration run(
- In this case python and backend will be picked up from last execution ``./breeze --python 3.8 --backend mysql``)
+ In this case python and backend will be picked up from last execution ``breeze --python 3.8 --backend mysql``)
It also automatically starts webserver, backend and scheduler. It drops you in tmux with scheduler in bottom left
and webserver in bottom right. Use ``[Ctrl + B] and Arrow keys`` to navigate.
@@ -1084,14 +1083,14 @@ Setting up Breeze
.. code-block:: bash
- $ ./breeze setup-autocomplete
+ $ breeze setup-autocomplete
$ source ~/.bash_completion.d/breeze-complete
3. Initialize breeze environment with required python version and backend. This may take a while for first time.
.. code-block:: bash
- $ ./breeze --python 3.8 --backend mysql
+ $ breeze --python 3.8 --backend mysql
.. note::
If you encounter an error like "docker.credentials.errors.InitializationError:
@@ -1123,7 +1122,7 @@ Setting up Breeze
.. code-block:: bash
- $ ./breeze stop
+ $ breeze stop
Installing airflow in the local virtual environment ``airflow-env`` with breeze.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -1133,7 +1132,7 @@ Installing airflow in the local virtual environment ``airflow-env`` with breeze.
.. code-block:: bash
$ sudo apt-get install sqlite libsqlite3-dev default-libmysqlclient-dev postgresql
- $ ./breeze initialize-local-virtualenv --python 3.8
+ $ ./breeze-legacy initialize-local-virtualenv --python 3.8
2. Add following line to ~/.bashrc in order to call breeze command from anywhere.
@@ -1147,7 +1146,7 @@ Using Breeze
~~~~~~~~~~~~
1. Starting breeze environment using ``breeze start-airflow`` starts Breeze environment with last configuration run(
- In this case python and backend will be picked up from last execution ``./breeze --python 3.8 --backend mysql``)
+ In this case python and backend will be picked up from last execution ``breeze --python 3.8 --backend mysql``)
It also automatically starts webserver, backend and scheduler. It drops you in tmux with scheduler in bottom left
and webserver in bottom right. Use ``[Ctrl + B] and Arrow keys`` to navigate.
@@ -1446,7 +1445,7 @@ All Tests are inside ./tests directory.
.. code-block:: bash
- $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All tests
+ $ ./breeze-legacy --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All tests
- Running specific test in container using shell scripts. Testing in container scripts are located in
@@ -1477,7 +1476,7 @@ All Tests are inside ./tests directory.
.. code-block:: bash
- $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core
+ $ ./breeze-legacy --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core
- Running Integration test for specific test type
@@ -1486,7 +1485,7 @@ All Tests are inside ./tests directory.
.. code-block:: bash
- $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All --integration mongo
+ $ ./breeze-legacy --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All --integration mongo
- For more information on Testing visit : |TESTING.rst|
@@ -1625,7 +1624,6 @@ To avoid burden on CI infrastructure and to save time, Pre-commit hooks can be r
$ pre-commit run --files airflow/decorators.py tests/utils/test_task_group.py
-
6. Running specific hook for selected files
.. code-block:: bash
@@ -1636,8 +1634,6 @@ To avoid burden on CI infrastructure and to save time, Pre-commit hooks can be r
Run flake8..........................................................Passed
-
-
7. Running specific checks in container using shell scripts. Scripts are located in ``./scripts/in_container``
directory.
@@ -1878,7 +1874,7 @@ Setting up Breeze
.. code-block:: bash
- $ ./breeze stop
+ $ breeze stop
Installing Airflow with Breeze.
@@ -2009,7 +2005,7 @@ All Tests are inside ``./tests`` directory.
.. code-block:: bash
- $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All tests
+ $ ./breeze-legacy --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All tests
- Running specific test in container using shell scripts. Testing in container scripts are located in
@@ -2048,7 +2044,7 @@ All Tests are inside ``./tests`` directory.
.. code-block:: bash
- $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core
+ $ ./breeze-legacy --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core
- Running Integration test for specific test type
@@ -2057,4 +2053,4 @@ All Tests are inside ``./tests`` directory.
.. code-block:: bash
- $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All --integration mongo
+ $ ./breeze-legacy --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All --integration mongo
diff --git a/IMAGES.rst b/IMAGES.rst
index 311ee4db5a..470e511100 100644
--- a/IMAGES.rst
+++ b/IMAGES.rst
@@ -85,33 +85,32 @@ You can build the CI image using current sources this command:
.. code-block:: bash
- ./breeze build-image
+ breeze build-image
You can build the PROD image using current sources with this command:
.. code-block:: bash
- ./breeze build-image --production-image
+ breeze build-prod-image
By adding ``--python <PYTHON_MAJOR_MINOR_VERSION>`` parameter you can build the
image version for the chosen Python version.
The images are build with default extras - different extras for CI and production image and you
can change the extras via the ``--extras`` parameters and add new ones with ``--additional-extras``.
-You can see default extras used via ``./breeze flags``.
For example if you want to build Python 3.7 version of production image with
"all" extras installed you should run this command:
.. code-block:: bash
- ./breeze build-image --python 3.7 --extras "all" --production-image
+ breeze build-prod-image --python 3.7 --extras "all"
If you just want to add new extras you can add them like that:
.. code-block:: bash
- ./breeze build-image --python 3.7 --additional-extras "all" --production-image
+ breeze build-prod-image --python 3.7 --additional-extras "all"
The command that builds the CI image is optimized to minimize the time needed to rebuild the image when
the source code of Airflow evolves. This means that if you already have the image locally downloaded and
@@ -129,8 +128,7 @@ parameter to Breeze:
.. code-block:: bash
- ./breeze build-image --python 3.7 --additional-extras=trino \
- --production-image --install-airflow-version=2.0.0
+ breeze build-prod-image --python 3.7 --additional-extras=trino --install-airflow-version=2.0.0
This will build the image using command similar to:
@@ -167,8 +165,8 @@ You can also skip installing airflow and install it from locally provided files
.. code-block:: bash
- ./breeze build-image --python 3.7 --additional-extras=trino \
- --production-image --disable-pypi-when-building --install-from-docker-context-files
+ breeze build-prod-image --python 3.7 --additional-extras=trino \
+ --disable-pypi-when-building --install-from-docker-context-files
In this case you airflow and all packages (.whl files) should be placed in ``docker-context-files`` folder.
@@ -195,21 +193,21 @@ even ``--build-cache-disabled`` flags when you run Breeze commands. For example:
.. code-block:: bash
- ./breeze build-image --python 3.7 --build-cache-local
+ breeze build-image --python 3.7 --docker-cache local
Will build the CI image using local build cache (note that it will take quite a long time the first
time you run it).
.. code-block:: bash
- ./breeze build-image --python 3.7 --production-image --build-cache-pulled
+ breeze build-prod-image --python 3.7 --docker-cache pulled
Will build the production image with pulled images as cache.
.. code-block:: bash
- ./breeze build-image --python 3.7 --production-image --build-cache-disabled
+ breeze build-prod-image --python 3.7 --docker-cache disabled
Will build the production image from the scratch.
@@ -285,7 +283,7 @@ For example this command will run the same Python 3.8 image as was used in build
.. code-block:: bash
- ./breeze --github-image-id 9a621eaa394c0a0a336f8e1b31b35eff4e4ee86e \
+ ./breeze-legacy --github-image-id 9a621eaa394c0a0a336f8e1b31b35eff4e4ee86e \
--python 3.8 --integration rabbitmq
You can see more details and examples in `Breeze <BREEZE.rst>`_
@@ -325,9 +323,7 @@ the same image can be built using ``breeze`` (it supports auto-completion of the
.. code-block:: bash
- ./breeze build-image -f Dockerfile.ci \
- --production-image --python 3.7 \
- --additional-extras=jdbc --additional-python-deps="pandas" \
+ breeze build-prod-image --python 3.7 --additional-extras=jdbc --additional-python-deps="pandas" \
--additional-dev-apt-deps="gcc g++" --additional-runtime-apt-deps="default-jre-headless"
You can customize more aspects of the image - such as additional commands executed before apt dependencies
diff --git a/LOCAL_VIRTUALENV.rst b/LOCAL_VIRTUALENV.rst
index ad0be26ce6..1cbf5b6b12 100644
--- a/LOCAL_VIRTUALENV.rst
+++ b/LOCAL_VIRTUALENV.rst
@@ -205,14 +205,14 @@ Activate your virtualenv, e.g. by using ``workon``, and once you are in it, run:
.. code-block:: bash
- ./breeze initialize-local-virtualenv
+ ./breeze-legacy initialize-local-virtualenv
By default Breeze installs the ``devel`` extra only. You can optionally control which extras are installed by exporting ``VIRTUALENV_EXTRAS`` before calling Breeze:
.. code-block:: bash
export VIRTUALENV_EXTRAS="devel,google,postgres"
- ./breeze initialize-local-virtualenv
+ ./breeze-legacy initialize-local-virtualenv
5. (optionally) run yarn build if you plan to run the webserver
diff --git a/STATIC_CODE_CHECKS.rst b/STATIC_CODE_CHECKS.rst
index 2fc0414345..41783dc71d 100644
--- a/STATIC_CODE_CHECKS.rst
+++ b/STATIC_CODE_CHECKS.rst
@@ -146,8 +146,6 @@ require Breeze Docker images to be installed locally.
------------------------------------ ---------------------------------------------------------------- ------------
``chart-schema-lint`` Lint chart/values.schema.json file
------------------------------------ ---------------------------------------------------------------- ------------
-``capitalized-breeze`` Breeze has to be Capitalized in Breeze2
------------------------------------- ---------------------------------------------------------------- ------------
``changelog-duplicates`` Checks for duplicate changelog entries
------------------------------------ ---------------------------------------------------------------- ------------
``check-2-1-compatibility`` Check that providers are 2.1-compatible
@@ -274,7 +272,9 @@ require Breeze Docker images to be installed locally.
------------------------------------ ---------------------------------------------------------------- ------------
``ui-lint`` Static checks of airflow/ui/ folder
------------------------------------ ---------------------------------------------------------------- ------------
-``update-breeze-file`` Update output of breeze command in BREEZE.rst
+``update-breeze-file`` Update output of breeze commands in BREEZE.rst
+------------------------------------ ---------------------------------------------------------------- ------------
+``limit-breeze-dependencies`` Breeze should have small number of top-level dependencies
------------------------------------ ---------------------------------------------------------------- ------------
``update-breeze-config-hash`` Update Breeze README.md with config files hash
------------------------------------ ---------------------------------------------------------------- ------------
@@ -362,7 +362,7 @@ Running static code checks via Breeze
The static code checks can be launched using the Breeze environment.
-You run the static code checks via ``./breeze static-check`` or commands.
+You run the static code checks via ``breeze static-check`` or commands.
You can see the list of available static checks either via ``--help`` flag or by using the autocomplete
option. Note that the ``all`` static check runs all configured static checks.
@@ -371,43 +371,43 @@ Run the ``mypy`` check for the currently staged changes:
.. code-block:: bash
- ./breeze static-check mypy
+ breeze static-check --type mypy
Run the ``mypy`` check for all files:
.. code-block:: bash
- ./breeze static-check mypy -- --all-files
+ breeze static-check --type mypy --all-files
Run the ``flake8`` check for the ``tests.core.py`` file with verbose output:
.. code-block:: bash
- ./breeze static-check flake8 -- --files tests/core.py --verbose
+ breeze static-check --type flake8 --files tests/core.py --verbose
Run the ``flake8`` check for the ``tests.core`` package with verbose output:
.. code-block:: bash
- ./breeze static-check flake8 -- --files tests/core/* --verbose
+ breeze static-check --type flake8 --files tests/core/* --verbose
Run all tests for the currently staged files:
.. code-block:: bash
- ./breeze static-check all
+ breeze static-check --type all
Run all tests for all files:
.. code-block:: bash
- ./breeze static-check all -- --all-files
+ breeze static-check --type all --all-files
Run all tests for last commit :
.. code-block:: bash
- ./breeze static-check all -- --from-ref HEAD^ --to-ref HEAD
+ breeze static-check --type all --last-commit
The ``license`` check is run via a separate script and a separate Docker image containing the
@@ -416,4 +416,4 @@ It does not take pre-commit parameters as extra arguments.
.. code-block:: bash
- ./breeze static-check licenses
+ breeze static-check licenses
diff --git a/TESTING.rst b/TESTING.rst
index 70273c2367..cb7b618e6f 100644
--- a/TESTING.rst
+++ b/TESTING.rst
@@ -168,19 +168,19 @@ to breeze.
.. code-block:: bash
- ./breeze tests tests/providers/http/hooks/test_http.py tests/core/test_core.py --db-reset -- --log-cli-level=DEBUG
+ ./breeze-legacy tests tests/providers/http/hooks/test_http.py tests/core/test_core.py --db-reset -- --log-cli-level=DEBUG
You can run the whole test suite without adding the test target:
.. code-block:: bash
- ./breeze tests --db-reset
+ ./breeze-legacy tests --db-reset
You can also specify individual tests or a group of tests:
.. code-block:: bash
- ./breeze tests --db-reset tests/core/test_core.py::TestCore
+ ./breeze-legacy tests --db-reset tests/core/test_core.py::TestCore
Running Tests of a specified type from the Host
@@ -201,13 +201,13 @@ kinds of test types:
.. code-block:: bash
- ./breeze --test-type Core --db-reset tests
+ ./breeze-legacy --test-type Core --db-reset tests
Runs all provider tests:
.. code-block:: bash
- ./breeze --test-type Providers --db-reset tests
+ ./breeze-legacy --test-type Providers --db-reset tests
* Special kinds of tests - Integration, Quarantined, Postgres, MySQL, which are marked with pytest
marks and for those you need to select the type using test-type switch. If you want to run such tests
@@ -219,13 +219,13 @@ kinds of test types:
.. code-block:: bash
- ./breeze --test-type Quarantined tests tests/cli/commands/test_task_command.py --db-reset
+ ./breeze-legacy --test-type Quarantined tests tests/cli/commands/test_task_command.py --db-reset
Run all Quarantined tests:
.. code-block:: bash
- ./breeze --test-type Quarantined tests --db-reset
+ ./breeze-legacy --test-type Quarantined tests --db-reset
Helm Unit Tests
===============
@@ -273,7 +273,7 @@ To run tests using breeze run the following command
.. code-block:: bash
- ./breeze --test-type Helm tests
+ ./breeze-legacy --test-type Helm tests
Airflow Integration Tests
=========================
@@ -327,19 +327,19 @@ To start the ``mongo`` integration only, enter:
.. code-block:: bash
- ./breeze --integration mongo
+ breeze --integration mongo
To start ``mongo`` and ``cassandra`` integrations, enter:
.. code-block:: bash
- ./breeze --integration mongo --integration cassandra
+ breeze --integration mongo --integration cassandra
To start all integrations, enter:
.. code-block:: bash
- ./breeze --integration all
+ breeze --integration all
In the CI environment, integrations can be enabled by specifying the ``ENABLED_INTEGRATIONS`` variable
storing a space-separated list of integrations to start. Thanks to that, we can run integration and
@@ -559,7 +559,7 @@ need to run the following steps:
.. code-block:: bash
- ./breeze prepare-provider-packages [PACKAGE ...]
+ ./breeze-legacy prepare-provider-packages [PACKAGE ...]
If you run this command without packages, you will prepare all packages. However, You can specify
providers that you would like to build if you just want to build few provider packages.
@@ -570,7 +570,7 @@ before running, so you should run it before generating ``apache-airflow`` packag
.. code-block:: bash
- ./breeze prepare-airflow-packages
+ ./breeze-legacy prepare-airflow-packages
This prepares airflow .whl package in the dist folder.
@@ -580,7 +580,7 @@ This installs airflow and enters
.. code-block:: bash
- ./breeze --use-airflow-version wheel --use-packages-from-dist --skip-mounting-local-sources
+ ./breeze-legacy --use-airflow-version wheel --use-packages-from-dist --skip-mounting-local-sources
@@ -602,7 +602,7 @@ For your testing, you manage Kind cluster with ``kind-cluster`` breeze command:
.. code-block:: bash
- ./breeze kind-cluster [ start | stop | recreate | status | deploy | test | shell | k9s ]
+ ./breeze-legacy kind-cluster [ start | stop | recreate | status | deploy | test | shell | k9s ]
The command allows you to start/stop/recreate/status Kind Kubernetes cluster, deploy Airflow via Helm
chart as well as interact with the cluster (via test and shell commands).
@@ -623,7 +623,7 @@ Deploying Airflow to the Kubernetes cluster created is also done via ``kind-clus
.. code-block:: bash
- ./breeze kind-cluster deploy
+ ./breeze-legacy kind-cluster deploy
The deploy command performs those steps:
@@ -640,10 +640,10 @@ You can also specify a different executor by providing the ``--executor`` option
.. code-block:: bash
- ./breeze kind-cluster deploy --executor CeleryExecutor
+ ./breeze-legacy kind-cluster deploy --executor CeleryExecutor
Note that when you specify the ``--executor`` option, it becomes the default. Therefore, every other operations
-on ``./breeze kind-cluster`` will default to using this executor. To change that, use the ``--executor`` option on the
+on ``./breeze-legacy kind-cluster`` will default to using this executor. To change that, use the ``--executor`` option on the
subsequent commands too.
@@ -667,15 +667,15 @@ Running Kubernetes tests via breeze:
.. code-block:: bash
- ./breeze kind-cluster test
- ./breeze kind-cluster test -- TEST TEST [TEST ...]
+ ./breeze-legacy kind-cluster test
+ ./breeze-legacy kind-cluster test -- TEST TEST [TEST ...]
Optionally add ``--executor``:
.. code-block:: bash
- ./breeze kind-cluster test --executor CeleryExecutor
- ./breeze kind-cluster test -- TEST TEST [TEST ...] --executor CeleryExecutor
+ ./breeze-legacy kind-cluster test --executor CeleryExecutor
+ ./breeze-legacy kind-cluster test -- TEST TEST [TEST ...] --executor CeleryExecutor
Entering shell with Kubernetes Cluster
--------------------------------------
@@ -701,13 +701,13 @@ You can enter the shell via those scripts
.. code-block:: bash
- ./breeze kind-cluster shell
+ ./breeze-legacy kind-cluster shell
Optionally add ``--executor``:
.. code-block:: bash
- ./breeze kind-cluster shell --executor CeleryExecutor
+ ./breeze-legacy kind-cluster shell --executor CeleryExecutor
K9s CLI - debug Kubernetes in style!
@@ -734,7 +734,7 @@ You can enter the k9s tool via breeze (after you deployed Airflow):
.. code-block:: bash
- ./breeze kind-cluster k9s
+ ./breeze-legacy kind-cluster k9s
You can exit k9s by pressing Ctrl-C.
@@ -747,7 +747,7 @@ The typical session for tests with Kubernetes looks like follows:
.. code-block:: bash
- ./breeze kind-cluster start
+ ./breeze-legacy kind-cluster start
Starts Kind Kubernetes cluster
@@ -787,7 +787,7 @@ The typical session for tests with Kubernetes looks like follows:
.. code-block:: bash
- ./breeze kind-cluster status
+ ./breeze-legacy kind-cluster status
Checks status of Kind Kubernetes cluster
@@ -807,7 +807,7 @@ The typical session for tests with Kubernetes looks like follows:
.. code-block:: bash
- ./breeze kind-cluster deploy
+ ./breeze-legacy kind-cluster deploy
4. Run Kubernetes tests
@@ -821,7 +821,7 @@ The virtualenv required will be created automatically when the scripts are run.
.. code-block:: bash
- ./breeze kind-cluster test
+ ./breeze-legacy kind-cluster test
4b) You can enter an interactive shell to run tests one-by-one
@@ -830,7 +830,7 @@ This prepares and enters the virtualenv in ``.build/.kubernetes_venv_<YOUR_CURRE
.. code-block:: bash
- ./breeze kind-cluster shell
+ ./breeze-legacy kind-cluster shell
Once you enter the environment, you receive this information:
@@ -852,7 +852,7 @@ In a separate terminal you can open the k9s CLI:
.. code-block:: bash
- ./breeze kind-cluster k9s
+ ./breeze-legacy kind-cluster k9s
Use it to observe what's going on in your cluster.
@@ -860,7 +860,7 @@ Use it to observe what's going on in your cluster.
It is very easy to running/debug Kubernetes tests with IntelliJ/PyCharm. Unlike the regular tests they are
in ``kubernetes_tests`` folder and if you followed the previous steps and entered the shell using
-``./breeze kind-cluster shell`` command, you can setup your IDE very easy to run (and debug) your
+``./breeze-legacy kind-cluster shell`` command, you can setup your IDE very easy to run (and debug) your
tests using the standard IntelliJ Run/Debug feature. You just need a few steps:
a) Add the virtualenv as interpreter for the project:
@@ -889,7 +889,7 @@ c) Run/Debug tests using standard "Run/Debug" feature of IntelliJ
NOTE! The first time you run it, it will likely fail with
``kubernetes.config.config_exception.ConfigException``:
``Invalid kube-config file. Expected key current-context in kube-config``. You need to add KUBECONFIG
-environment variable copying it from the result of "./breeze kind-cluster test":
+environment variable copying it from the result of "./breeze-legacy kind-cluster test":
.. code-block:: bash
@@ -947,14 +947,14 @@ Airflow to the cluster.
.. code-block:: bash
- ./breeze kind-cluster deploy
+ ./breeze-legacy kind-cluster deploy
7. Stop KinD cluster when you are done
.. code-block:: bash
- ./breeze kind-cluster stop
+ ./breeze-legacy kind-cluster stop
Airflow System Tests
@@ -1080,12 +1080,12 @@ Preparing provider packages for System Tests for Airflow 1.10.* series
----------------------------------------------------------------------
To run system tests with the older Airflow version, you need to prepare provider packages. This
-can be done by running ``./breeze prepare-provider-packages <PACKAGES TO BUILD>``. For
+can be done by running ``./breeze-legacy prepare-provider-packages <PACKAGES TO BUILD>``. For
example, the below command will build google, postgres and mysql wheel packages:
.. code-block:: bash
- ./breeze prepare-provider-packages -- google postgres mysql
+ ./breeze-legacy prepare-provider-packages -- google postgres mysql
Those packages will be prepared in ./dist folder. This folder is mapped to /dist folder
when you enter Breeze, so it is easy to automate installing those packages for testing.
@@ -1099,14 +1099,16 @@ Here is the typical session that you need to do to run system tests:
.. code-block:: bash
- ./breeze --python 3.6 --db-reset --forward-credentials restart
+ breeze stop
+ breeze --python 3.7 --db-reset --forward-credentials
This will:
-* restarts the whole environment (i.e. recreates metadata database from the scratch)
-* run Breeze with python 3.6 version
-* reset the Airflow database
-* forward your local credentials to Breeze
+* stop the whole environment (i.e. recreates metadata database from the scratch)
+* run Breeze with:
+ * python 3.7 version
+ * resetting the Airflow database
+ * forward your local credentials to Breeze
3. Run the tests:
@@ -1115,7 +1117,6 @@ This will:
pytest -o faulthandler_timeout=2400 \
--system=google tests/providers/google/cloud/operators/test_compute_system.py
-
Iteration with System Tests if your resources are slow to create
----------------------------------------------------------------
@@ -1154,7 +1155,8 @@ Breeze session. They are usually expensive to run.
.. code-block:: bash
- ./breeze --python 3.6 --db-reset --forward-credentials restart
+ breeze stop
+ breeze --python 3.6 --db-reset --forward-credentials
2. Run create action in helper (to create slowly created resources):
diff --git a/breeze b/breeze
index 27ef65cb8d..b433fac7f9 100755
--- a/breeze
+++ b/breeze
@@ -1,5 +1,4 @@
#!/usr/bin/env bash
-
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -18,3670 +17,65 @@
# under the License.
set -euo pipefail
+MY_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
-AIRFLOW_SOURCES="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-
-if [[ ${BREEZE_REDIRECT=} == "" ]]; then
- set +u
- mkdir -p "${AIRFLOW_SOURCES}"/logs
- export BREEZE_REDIRECT="true"
- if [[ "$(uname)" == "Darwin" ]]; then
- exec script -q "${AIRFLOW_SOURCES}"/logs/breeze.out "$(command -v bash)" -c "$(printf "%q " "${0}" "${@}")"
- else
- exec script --return --quiet "${AIRFLOW_SOURCES}"/logs/breeze.out -c "$(printf "%q " "${0}" "${@}")"
- fi
- set -u
-fi
-
-export AIRFLOW_SOURCES
-readonly AIRFLOW_SOURCES
-
-
-# Bash arrays need to be defined outside of functions unfortunately :(
-# Because on Mac OS Bash 3.4 defining arrays inside functions does not work
-# Array with extra options for Docker compose
-declare -a EXTRA_DC_OPTIONS
-export EXTRA_DC_OPTIONS
-
-# Array with selected integrations
-declare -a INTEGRATIONS
-export INTEGRATIONS
-
-# This is where remaining args are passed
-declare -a REMAINING_ARGS
-export REMAINING_ARGS
-
-# This is where static check options are defined
-declare -a EXTRA_STATIC_CHECK_OPTIONS
-export EXTRA_STATIC_CHECK_OPTIONS
-
-#######################################################################################################
-# Sets up all the default variables for Breeze. They are needed by all other functions
-# All those variables are exported. They are not set to read-only because those
-# defaults can be modified later on when command line arguments are parsed
-# and variables stored in .build directory (stored in the previous run) are read
-#
-# Used globals:
-# FORCE_SCREEN_WIDTH
-#
-# Modified globals (constants or candidates for constants after we override them via appropriate flags):
-#
-# BREEZE
-# SUPPRESS_CHEATSHEET_FILE
-# SUPPRESS_ASCIIART_FILE
-# MAX_SCREEN_WIDTH
-# SCREEN_WIDTH
-# MOUNT_SELECTED_LOCAL_SOURCES
-# FORWARD_CREDENTIALS
-# DB_RESET
-# START_AIRFLOW
-# INSTALL_AIRFLOW_VERSION
-# INSTALL_AIRFLOW_REFERENCE
-# FORCE_BUILD_IMAGES
-# PRODUCTION_IMAGE
-# PYTHON_MAJOR_MINOR_VERSION
-#
-# Global variables:
-#
-# command_to_run
-# second_command_to_run
-# docker_compose_command
-#
-# Also it sets the variables and globals set by common initialization functions from
-# scripts/ci/libraries/_initialization.sh and breeze-complete script (which sets-up auto-complete).
-#
-#######################################################################################################
-function breeze::setup_default_breeze_constants() {
- # Indicates that we are inside Breeze environment
- export BREEZE=true
- readonly BREEZE
-
- # If those files are present, the ASCII-art/cheat-sheet are suppressed
- SUPPRESS_CHEATSHEET_FILE="${AIRFLOW_SOURCES}/.build/.suppress_cheatsheet"
- readonly SUPPRESS_CHEATSHEET_FILE
-
- SUPPRESS_ASCIIART_FILE="${AIRFLOW_SOURCES}/.build/.suppress_asciiart"
- readonly SUPPRESS_ASCIIART_FILE
-
- # Maximum screen indented_screen_width to print the lines spanning the whole terminal indented_screen_width
- export MAX_SCREEN_WIDTH=100
- readonly MAX_SCREEN_WIDTH
-
- # By default we mount selected local Airflow sources
- export MOUNT_SELECTED_LOCAL_SOURCES="true"
-
- # By default we do not mount all local Airflow sources
- export MOUNT_ALL_LOCAL_SOURCES="false"
-
- # Forward common host credentials to docker (gcloud, aws etc.).
- export FORWARD_CREDENTIALS="false"
-
- # If set to true, the database will be reset at entry. Works for Postgres and MySQL
- export DB_RESET="false"
-
- # If set to true, the database will be initialized, a user created and webserver and scheduler started
- export START_AIRFLOW="false"
-
- # If set to true, the test connections will be created
- export LOAD_DEFAULT_CONNECTIONS="false"
-
- # If set to true, the sample dags will be used
- export LOAD_EXAMPLES="false"
-
- # If set to true, Breeze db volumes will be preserved when breeze is stopped and reused next time
- # Which means that you do not have to start from scratch
- export PRESERVE_VOLUMES="false"
-
- # Sources by default are installed from local sources when using breeze
- AIRFLOW_SOURCES_FROM=${AIRFLOW_SOURCES_FROM:="."}
- export AIRFLOW_SOURCES_FROM
-
- # They are copied to /opt/airflow by default in breeze
- AIRFLOW_SOURCES_TO=${AIRFLOW_SOURCES_TO:="/opt/airflow"}
- export AIRFLOW_SOURCES_TO
-
- # Sources by default are installed from local sources when using breeze
- AIRFLOW_SOURCES_WWW_FROM=${AIRFLOW_SOURCES_WWW_FROM:="./airflow/www"}
- export AIRFLOW_SOURCES_WWW_FROM
-
- # They are copied to /opt/airflow by default in breeze
- AIRFLOW_SOURCES_WWW_TO=${AIRFLOW_SOURCES_WWW_TO:="/opt/airflow/airflow/www"}
- export AIRFLOW_SOURCES_WWW_TO
+BREEZE_BINARY=breeze
+COLOR_RED=$'\e[31m'
+COLOR_RESET=$'\e[0m'
+COLOR_YELLOW=$'\e[33m'
- # Unlike in CI scripts, in breeze by default production image is installed from sources
- export AIRFLOW_INSTALLATION_METHOD="."
-
- # If it set is set to specified version, then the source version of Airflow
- # is removed and the specified version of Airflow is installed from PyPi
- export INSTALL_AIRFLOW_VERSION=""
-
- # If it is set to specified reference (tag/branch), then the source version
- # of Airflow is removed and the specified version of Airflow is installed from GitHub
- export INSTALL_AIRFLOW_REFERENCE=""
-
- # Determines whether to force build without checking if it is needed
- # Can be overridden by '--force-build-images' flag.
- export FORCE_BUILD_IMAGES="false"
-
- # Determines whether to reinstall airflow at entering the image.
- export USE_AIRFLOW_VERSION=""
- # if set to true, the ci image will look for wheel packages in dist folder and will install them
- # during entering the container
- export USE_PACKAGES_FROM_DIST="false"
-
- # load all the common functions here - those are the functions that are shared between Breeze
- # and CI scripts. The CI scripts do not use Breeze as driving script - they read all configuration
- # from the environment variables. That's why we keep all the common initialization in those libs
- # shellcheck source=scripts/ci/libraries/_all_libs.sh
- . "${AIRFLOW_SOURCES}/scripts/ci/libraries/_all_libs.sh"
-
- # When we generate documentation for README files, we want to force the indented_screen_width of terminal so that
- # No matter who is running the documentation generation gets the same output
- if [[ ${FORCE_SCREEN_WIDTH:="false"} != "true" ]]; then
- # Sets indented_screen_width of the screen from terminal
- SCREEN_WIDTH="$(tput cols)"
- if [[ -z ${SCREEN_WIDTH=} ]]; then
- SCREEN_WIDTH=${MAX_SCREEN_WIDTH}
- fi
- if ((SCREEN_WIDTH > MAX_SCREEN_WIDTH)); then
- SCREEN_WIDTH=${MAX_SCREEN_WIDTH}
- fi
- else
- SCREEN_WIDTH=${MAX_SCREEN_WIDTH}
- fi
-
- export SCREEN_WIDTH
- readonly SCREEN_WIDTH
-
- # for Breeze default tests executed are "All"
- export TEST_TYPE=${TEST_TYPE:="All"}
-
- # Update short and long options in the breeze-complete script
- # This way autocomplete will work automatically with all options available
- # shellcheck source=breeze-complete
- . "${AIRFLOW_SOURCES}/breeze-complete"
-
- # Default command to run - entering breeze environment
- command_to_run="enter_breeze"
-
- # In some cases we also want to run two commands in a row (for example when we restart the environment)
- second_command_to_run=""
-
- # Determines if help should be run (set to true by --help flag)
- run_help="false"
-
- # Holds docker compose command if the `docker-compose` command is used.
- docker_compose_command=""
+function manual_instructions() {
+ echo "Please run those commands manually (you might need to restart shell between them):"
+ echo
+ echo " pip -m install pipx"
+ echo " pipx ensurepath"
+ echo " pipx install -e '${MY_DIR}/dev/breeze/'"
+ echo " breeze setup-autocomplete --force"
+ echo
+ echo " After that, both pipx and breeze should be available ono your path"
+ exit
}
-#######################################################################################################
-#
-# Initializes development-friendly virtualenv if you are already in such env. It installs all the necessary
-# packages from PyPI and it case of problems it provides useful hints on what prerequisites should be
-# installed. It also removes and resets the existing AIRFLOW_HOME installation to make sure that you
-# have it synchronized with the version of airflow installed. It resets the airflow's sqlite database to
-# a clean state. You can use this function if your virtualenv is broken, to clean it up
-#
-# Used globals:
-# PYTHON_MAJOR_MINOR_VERSION
-# AIRFLOW_HOME_DIR
-# AIRFLOW_SOURCES
-# VIRTUALENV_EXTRAS
-# DEFAULT_CONSTRAINTS_BRANCH
-# OSTYPE
-#
-#######################################################################################################
-function breeze::initialize_virtualenv() {
- # Check if we are inside virtualenv
+function check_breeze_installed() {
set +e
- echo -e "import sys\nif not hasattr(sys,'base_prefix'):\n sys.exit(1)" |
- python"${PYTHON_MAJOR_MINOR_VERSION}"
- local res=$?
+ command -v "${BREEZE_BINARY}" >/dev/null
+ local breeze_on_path=$?
+ command -v "pipx" >/dev/null
+ local pipx_on_path=$?
set -e
- if [[ ${res} != "0" ]]; then
- echo
- echo "${COLOR_RED}ERROR: Initializing local virtualenv only works when you have virtualenv activated ${COLOR_RESET}"
- echo
- echo "Please enter your local virtualenv before (for example using 'pyenv activate' or 'workon') "
- echo
- exit 1
- else
- echo
- echo "Initializing the virtualenv: $(command -v python)!"
- echo
- echo "Extras to be installed: ${VIRTUALENV_EXTRAS}"
- echo
- echo "This will wipe out ${AIRFLOW_HOME_DIR} and reset all the databases!"
- echo
- "${AIRFLOW_SOURCES}/scripts/tools/confirm" "Proceeding with the initialization"
- echo
- pushd "${AIRFLOW_SOURCES}" >/dev/null 2>&1 || exit 1
- set +e
- pip install -e ".[${VIRTUALENV_EXTRAS}]" \
- --constraint "https://raw.githubusercontent.com/${CONSTRAINTS_GITHUB_REPOSITORY}/${DEFAULT_CONSTRAINTS_BRANCH}/constraints-source-providers-${PYTHON_MAJOR_MINOR_VERSION}.txt"
- res=$?
- set -e
- popd
- if [[ ${res} != "0" ]]; then
- echo "#######################################################################"
- echo " You had some troubles installing the venv !!!!!"
- echo " Try running the command below and rerun virtualenv installation"
- echo
- if [[ ${OSTYPE} == "darwin"* ]]; then
- echo " brew install sqlite mysql postgresql openssl"
- echo " export LDFLAGS=\"-L/usr/local/opt/openssl/lib\""
- echo " export CPPFLAGS=\"-I/usr/local/opt/openssl/include\""
+ if [[ ${breeze_on_path} != "0" || ${pipx_on_path} != "0" ]]; then
+ echo
+ if [[ ${pipx_on_path} != 0 ]]; then
+ echo "${COLOR_RED}The 'pipx' is not on path. It should be installed and 'pipx' should be available on your PATH.${COLOR_RESET}"
+ export TIMEOUT=0
+ if "${MY_DIR}/scripts/tools/confirm" "Installing pipx?"; then
+ python -m pip install pipx --upgrade
+ echo
+ echo "${COLOR_YELLOW}Please close and re-open the shell and retry. You might need to add 'pipx' to the PATH!${COLOR_RESET}"
+ echo
+ exit
else
- echo " sudo apt install build-essential python3-dev libsqlite3-dev openssl \\"
- echo " sqlite default-libmysqlclient-dev libmysqlclient-dev postgresql"
+ manual_instructions
fi
- echo
- echo "#######################################################################"
- exit ${res}
- fi
- echo
- echo "Wiping and recreating ${AIRFLOW_HOME_DIR}"
- echo
- if [[ "${AIRFLOW_SOURCES}" == "${AIRFLOW_HOME_DIR}" ]]; then
- echo "AIRFLOW_HOME and Source code for Apache Airflow resides in the same path ${AIRFLOW_HOME_DIR}"
- echo "When running this command it will delete all the files in the path ${AIRFLOW_HOME_DIR} to clear dynamic files like config/logs/db"
- echo "Move your source code for Apache Airflow to different folder to avoid deletion"
- exit 1
- fi
- rm -rvf "${AIRFLOW_HOME_DIR}"
- mkdir -p "${AIRFLOW_HOME_DIR}"
- echo
- echo "Resetting AIRFLOW sqlite database"
- echo
- AIRFLOW__CORE__LOAD_EXAMPLES="False" \
- AIRFLOW__CORE__UNIT_TEST_MODE="False" \
- AIRFLOW__CORE__SQL_ALCHEMY_POOL_ENABLED="False" \
- AIRFLOW__CORE__DAGS_FOLDER="${AIRFLOW_SOURCES}/empty" \
- AIRFLOW__CORE__PLUGINS_FOLDER="${AIRFLOW_SOURCES}/empty" \
- airflow db reset -y
- echo
- echo "Resetting AIRFLOW sqlite unit test database"
- echo
- AIRFLOW__CORE__LOAD_EXAMPLES="False" \
- AIRFLOW__CORE__UNIT_TEST_MODE="True" \
- AIRFLOW__CORE__SQL_ALCHEMY_POOL_ENABLED="False" \
- AIRFLOW__CORE__DAGS_FOLDER="${AIRFLOW_SOURCES}/empty" \
- AIRFLOW__CORE__PLUGINS_FOLDER="${AIRFLOW_SOURCES}/empty" \
- airflow db reset -y
- echo
- echo "Initialization of virtualenv was successful! Go ahead and develop Airflow!"
- echo
- exit 0
- fi
-}
-
-#######################################################################################################
-#
-# Sets up autocomplete for Breeze for both - bash and zsh
-#
-# Used globals:
-#
-# AIRFLOW_SOURCES
-# HOME
-# OSTYPE
-#
-#######################################################################################################
-function breeze::setup_autocomplete() {
- echo "Installing bash/zsh completion for local user"
- echo
- "${AIRFLOW_SOURCES}/scripts/tools/confirm" "This will create ~/.bash_completion.d/ directory and modify ~/.*rc files"
- echo
- echo
- mkdir -pv ~/.bash_completion.d
- ln -sf "${AIRFLOW_SOURCES}/breeze-complete" "${HOME}/.bash_completion.d/"
- echo
- echo "Breeze Bash completion is now linked to: ${AIRFLOW_SOURCES}/breeze-complete"
- echo
- local breeze_comment="Added by Airflow Breeze autocomplete setup"
- if ! grep "${breeze_comment}" "${HOME}/.bashrc" >/dev/null 2>&1; then
- touch ~/.bashrc
- # shellcheck disable=SC2129
- echo "# START: ${breeze_comment}" >>~/.bashrc
- cat <<"EOF" >>~/.bashrc
-for bcfile in ~/.bash_completion.d/* ; do
- . ${bcfile}
-done
-EOF
- echo "# END: ${breeze_comment}" >>~/.bashrc
- echo
- echo "The ${HOME}/.bashrc has been modified"
- echo
- else
- echo
- echo "The ${HOME}/.bashrc was already modified before. Not changing it."
- echo
- fi
- if ! grep "${breeze_comment}" "${HOME}/.zshrc" >/dev/null 2>&1; then
- # shellcheck disable=SC2129
- echo "# START: ${breeze_comment}" >>~/.zshrc
- cat <<"EOF" >>~/.zshrc
-autoload compinit && compinit
-autoload bashcompinit && bashcompinit
-source ~/.bash_completion.d/breeze-complete
-EOF
- echo "# END: ${breeze_comment}" >>~/.zshrc
- echo
- echo "The ${HOME}/.zshrc has been modified"
- echo
- else
- echo
- echo "The ${HOME}/.zshrc was already modified before. Not changing it."
- echo
- fi
- if [[ "${OSTYPE}" == "darwin"* ]]; then
- # For MacOS we have to handle the special case where terminal app DOES NOT run .bashrc by default
- # But re-runs .bash_profile :(
- # See https://scriptingosx.com/2017/04/about-bash_profile-and-bashrc-on-macos/
- if ! grep "${breeze_comment}" "${HOME}/.bash_profile"; then
- # shellcheck disable=SC2129
- echo "# START: ${breeze_comment}" >>~/.bash_profile
- cat <<"EOF" >>~/.bash_profile
-if [ -r ~/.bashrc ]; then
- source ~/.bashrc
-fi
-EOF
- echo "# END: ${breeze_comment}" >>~/.bash_profile
- echo
- echo "The ${HOME}/.bash_profile has been modified"
- echo
- else
- echo
- echo "The ${HOME}/.bash_profile was already modified before. Not changing it."
- echo
- fi
- fi
- echo
- echo
- echo "Breeze completion is installed to ~/.bash_completion.d/breeze-complete"
- echo
- echo "Please exit and re-enter your shell or run:"
- echo
- if [[ "${OSTYPE}" == "darwin"* ]]; then
- if grep "${breeze_comment}" "${HOME}/.zshrc" >/dev/null 2>&1; then
- echo " source ~/.zshrc"
- echo
- echo " source ~/.bash_completion.d/breeze-complete"
- echo
- exec zsh
- exit 0
- fi
- fi
- echo " source ~/.bash_completion.d/breeze-complete"
- echo
- exit 0
-}
-
-#######################################################################################################
-#
-# Prints information about the current configuration of Breeze - if you enter breeze interactively
-# and you did not suppress cheatsheet or asciiart, it also prints those. It also prints values
-# of constants set by breeze::read_saved_environment_variables() function and other initialization functions.
-#
-# Used globals:
-#
-# BACKEND
-# POSTGRES_VERSION
-# MYSQL_VERSION
-# SUPPRESS_CHEATSHEET_FILE
-# SUPPRESS_ASCIIART_FILE
-# PRODUCTION_IMAGE
-# BRANCH_NAME
-# AIRFLOW_CI_IMAGE
-# AIRFLOW_PROD_IMAGE
-# AIRFLOW_VERSION
-# INSTALL_AIRFLOW_VERSION
-# INSTALL_AIRFLOW_REFERENCE
-#
-# Outputs:
-# Prints the information about the build to stdout.
-#
-#######################################################################################################
-function breeze::print_badge() {
- local backend_version=""
- if [[ ${BACKEND} == "postgres" ]]; then
- backend_version="${POSTGRES_VERSION}"
- elif [[ ${BACKEND} == "mysql" ]]; then
- backend_version="${MYSQL_VERSION}"
- fi
- if [[ ! -f "${SUPPRESS_ASCIIART_FILE}" && ${command_to_run} == "enter_breeze" ]]; then
- cat <<EOF
-
-
-
-
- @&&&&&&@
- @&&&&&&&&&&&@
- &&&&&&&&&&&&&&&&
- &&&&&&&&&&
- &&&&&&&
- &&&&&&&
- @@@@@@@@@@@@@@@@ &&&&&&
- @&&&&&&&&&&&&&&&&&&&&&&&&&&
- &&&&&&&&&&&&&&&&&&&&&&&&&&&&
- &&&&&&&&&&&&
- &&&&&&&&&
- &&&&&&&&&&&&
- @@&&&&&&&&&&&&&&&@
- @&&&&&&&&&&&&&&&&&&&&&&&&&&&& &&&&&&
- &&&&&&&&&&&&&&&&&&&&&&&&&&&& &&&&&&
- &&&&&&&&&&&&&&&&&&&&&&&& &&&&&&
- &&&&&&
- &&&&&&&
- @&&&&&&&&
- @&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
- &&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
- &&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
-
-
-
- @&&&@ && @&&&&&&&&&&& &&&&&&&&&&&& && &&&&&&&&&& &&& &&& &&&
- &&& &&& && @&& &&& && && &&& &&&@ &&& &&&&& &&&
- &&& &&& && @&&&&&&&&&&&& &&&&&&&&&&& && && &&& &&& &&& &&@ &&&
- &&&&&&&&&&& && @&&&&&&&&& && && &&@ &&& &&@&& &&@&&
- &&& &&& && @&& &&&@ && &&&&&&&&&&& &&&&&&&&&&&& &&&& &&&&
-
-&&&&&&&&&&&& &&&&&&&&&&&& &&&&&&&&&&&@ &&&&&&&&&&&& &&&&&&&&&&& &&&&&&&&&&&
-&&& &&& && &&& && &&& &&&& &&
-&&&&&&&&&&&&@ &&&&&&&&&&&& &&&&&&&&&&& &&&&&&&&&&& &&&& &&&&&&&&&&
-&&& && && &&&& && &&& &&&& &&
-&&&&&&&&&&&&& && &&&&@ &&&&&&&&&&&@ &&&&&&&&&&&& @&&&&&&&&&&& &&&&&&&&&&&
-
-EOF
- if [[ ${PRODUCTION_IMAGE} == "true" ]]; then
- cat <<EOF
-
- Use production image.
-
- Branch name: ${BRANCH_NAME}
- Docker image: ${AIRFLOW_PROD_IMAGE}
- Platform: ${PLATFORM}
- Airflow source version: $(build_images::get_airflow_version_from_production_image)
-EOF
- else
- cat <<EOF
-
- Use CI image.
-
- Branch name: ${BRANCH_NAME}
- Docker image: ${AIRFLOW_CI_IMAGE_WITH_TAG}
- Platform: ${PLATFORM}
- Airflow source version: ${AIRFLOW_VERSION}
-EOF
- fi
- cat <<EOF
- Python version: ${PYTHON_MAJOR_MINOR_VERSION}
- Debian version: ${DEBIAN_VERSION}
- Backend: ${BACKEND} ${backend_version}
-EOF
- if [[ -n ${USE_AIRFLOW_VERSION=} ]]; then
- cat <<EOF
-
- Airflow used at runtime: ${USE_AIRFLOW_VERSION=}
-
-EOF
- fi
- else
- if [[ ${PRODUCTION_IMAGE} == "true" ]]; then
- cat <<EOF
-
- Production image.
-
- Branch name: ${BRANCH_NAME}
- Docker image: ${AIRFLOW_PROD_IMAGE}
- Platform: ${PLATFORM}
-EOF
- else
- cat <<EOF
-
- CI image.
-
- Branch name: ${BRANCH_NAME}
- Docker image: ${AIRFLOW_CI_IMAGE}
- Platform: ${PLATFORM}
-
-EOF
- fi
- if [[ -n ${INSTALL_AIRFLOW_VERSION=} || -n ${INSTALL_AIRFLOW_REFERENCE=} ]]; then
- cat <<EOF
- Airflow version installed: ${INSTALL_AIRFLOW_VERSION=}${INSTALL_AIRFLOW_REFERENCE=}
-
-EOF
- fi
- cat <<EOF
-
- Airflow source version: ${AIRFLOW_VERSION}
- Python version: ${PYTHON_MAJOR_MINOR_VERSION}
- Backend: ${BACKEND} ${backend_version}
-EOF
- fi
- if [[ ${VERBOSE} == "true" ]]; then
- initialization::summarize_build_environment
- fi
-
-}
-
-#######################################################################################################
-#
-# Prepares command file that can be used to easily run the docker commands outside of Breeze.
-#
-# The command file generated in cache ./build directory is a standalone script that contains
-# All the environment variables and docker-compose configuration to run the command.
-# This is because depending on configuration of Breeze we might have different compose files
-# used and different env variables set.
-#
-# Those are a convenience scripts that you might use to debug command execution although
-# In most cases they are used internally by Breeze.
-#
-# Arguments:
-#
-# file to prepare
-# command to run
-# compose_file to use
-# airflow_image to use
-#
-# Outputs:
-# Creates the convenience command file that can be run to use the docker command.
-#
-#######################################################################################################
-function breeze::prepare_command_file() {
- local file="${1}"
- local command="${2}"
- local compose_file="${3}"
- cat <<EOF >"${file}"
-#!/usr/bin/env bash
-docker_compose_version=\$(docker-compose --version || true)
-if [[ \${docker_compose_version} =~ .*([0-9]+)\.([0-9]+)\.([0-9]+).* ]]; then
- major=\${BASH_REMATCH[1]}
- minor=\${BASH_REMATCH[2]}
- patch=\${BASH_REMATCH[3]}
- if [[ \${major} == "1" ]]; then
- if (( minor < 29 )); then
- echo
- echo "${COLOR_RED}You have too old version of docker-compose: \${major}.\${minor}.\${patch}! At least 1.29 is needed! Please upgrade!${COLOR_RESET}"
- echo "${COLOR_RED}See https://docs.docker.com/compose/install/ for instructions. Make sure docker-compose you install is first on the PATH variable of yours.${COLOR_RESET}"
- echo
- exit 1
- fi
- fi
- echo
- echo "${COLOR_GREEN}Good version of docker-compose: \${major}.\${minor}.\${patch}${COLOR_RESET}"
- echo
-else
- echo
- echo "${COLOR_YELLOW}Unknown docker-compose version. At least 1.29 is needed! If Breeze fails - upgrade to latest available docker-compose version${COLOR_RESET}"
- echo
-fi
-
-if [[ \${VERBOSE} == "true" ]]; then
- echo
- echo "Executing script:"
- echo
- echo "${COLOR_CYAN}${file} \${@}${COLOR_RESET}"
- echo
- set -x
-fi
-cd "\$( dirname "\${BASH_SOURCE[0]}" )" || exit
-export HOST_USER_ID=${HOST_USER_ID}
-export HOST_GROUP_ID=${HOST_GROUP_ID}
-export COMPOSE_FILE="${compose_file}"
-export PYTHON_MAJOR_MINOR_VERSION="${PYTHON_MAJOR_MINOR_VERSION}"
-export DEBIAN_VERSION="${DEBIAN_VERSION}"
-export BACKEND="${BACKEND}"
-export AIRFLOW_VERSION="${AIRFLOW_VERSION}"
-export INSTALL_AIRFLOW_VERSION="${INSTALL_AIRFLOW_VERSION}"
-export PLATFORM="${PLATFORM}"
-export SSH_PORT="${SSH_PORT}"
-export WEBSERVER_HOST_PORT="${WEBSERVER_HOST_PORT}"
-export FLOWER_HOST_PORT="${FLOWER_HOST_PORT}"
-export REDIS_HOST_PORT="${REDIS_HOST_PORT}"
-export POSTGRES_HOST_PORT="${POSTGRES_HOST_PORT}"
-export POSTGRES_VERSION="${POSTGRES_VERSION}"
-export MYSQL_HOST_PORT="${MYSQL_HOST_PORT}"
-export MYSQL_VERSION="${MYSQL_VERSION}"
-export MSSQL_HOST_PORT="${MSSQL_HOST_PORT}"
-export MSSQL_VERSION="${MSSQL_VERSION}"
-export AIRFLOW_SOURCES="${AIRFLOW_SOURCES}"
-export AIRFLOW_CI_IMAGE="${AIRFLOW_CI_IMAGE}"
-export AIRFLOW_CI_IMAGE_WITH_TAG="${AIRFLOW_CI_IMAGE_WITH_TAG}"
-export AIRFLOW_PROD_IMAGE="${AIRFLOW_PROD_IMAGE}"
-export AIRFLOW_IMAGE_KUBERNETES="${AIRFLOW_IMAGE_KUBERNETES}"
-export SQLITE_URL="${SQLITE_URL}"
-export USE_AIRFLOW_VERSION="${USE_AIRFLOW_VERSION}"
-export SKIP_TWINE_CHECK="${SKIP_TWINE_CHECK}"
-export USE_PACKAGES_FROM_DIST="${USE_PACKAGES_FROM_DIST}"
-export EXECUTOR="${EXECUTOR}"
-export START_AIRFLOW="${START_AIRFLOW}"
-export ENABLED_INTEGRATIONS="${ENABLED_INTEGRATIONS}"
-export ENABLED_SYSTEMS="${ENABLED_SYSTEMS}"
-export GITHUB_ACTIONS="${GITHUB_ACTIONS}"
-export ISSUE_ID="${ISSUE_ID}"
-export NUM_RUNS="${NUM_RUNS}"
-export VERSION_SUFFIX_FOR_SVN="${VERSION_SUFFIX_FOR_SVN}"
-export VERSION_SUFFIX_FOR_PYPI="${VERSION_SUFFIX_FOR_PYPI}"
-docker-compose ${command}
-EOF
- chmod u+x "${file}"
-}
-
-#######################################################################################################
-#
-# Prepare all command files that we are using. We use dc_ci - to run docker compose command for CI image
-#
-# Global constants set:
-#
-# PYTHON_BASE_IMAGE
-# AIRFLOW_CI_IMAGE
-# BUILT_CI_IMAGE_FLAG_FILE
-#
-#######################################################################################################
-function breeze::prepare_command_files() {
- local main_ci_docker_compose_file=${SCRIPTS_CI_DIR}/docker-compose/base.yml
- if [[ ${BACKEND} == "mssql" ]]; then
- local backend_docker_compose_file=${SCRIPTS_CI_DIR}/docker-compose/backend-${BACKEND}-${DEBIAN_VERSION}.yml
- else
- local backend_docker_compose_file=${SCRIPTS_CI_DIR}/docker-compose/backend-${BACKEND}.yml
- fi
-
- local backend_port_docker_compose_file=${SCRIPTS_CI_DIR}/docker-compose/backend-${BACKEND}-port.yml
- local local_docker_compose_file=${SCRIPTS_CI_DIR}/docker-compose/local.yml
- local local_all_sources_docker_compose_file=${SCRIPTS_CI_DIR}/docker-compose/local-all-sources.yml
- local files_docker_compose_file=${SCRIPTS_CI_DIR}/docker-compose/files.yml
- local remove_sources_docker_compose_file=${SCRIPTS_CI_DIR}/docker-compose/remove-sources.yml
- local forward_credentials_docker_compose_file=${SCRIPTS_CI_DIR}/docker-compose/forward-credentials.yml
-
-
- if [[ ${BACKEND} == "mssql" ]]; then
- local docker_filesystem
- docker_filesystem=$(stat "-f" "-c" "%T" /var/lib/docker 2>/dev/null || echo "unknown")
- if [[ ${docker_filesystem} == "tmpfs" ]]; then
- # In case of tmpfs backend for docker, mssql fails because TMPFS does not support
- # O_DIRECT parameter for direct writing to the filesystem
- # https://github.com/microsoft/mssql-docker/issues/13
- # so we need to mount an external volume for its db location
- # specified by MSSQL_DATA_VOLUME
- backend_docker_compose_file="${backend_docker_compose_file}:${SCRIPTS_CI_DIR}/docker-compose/backend-mssql-bind-volume.yml"
- else
- backend_docker_compose_file="${backend_docker_compose_file}:${SCRIPTS_CI_DIR}/docker-compose/backend-mssql-docker-volume.yml"
fi
- fi
-
-
- local compose_ci_file=${main_ci_docker_compose_file}:${backend_docker_compose_file}:${files_docker_compose_file}
-
- if [[ "${MOUNT_SELECTED_LOCAL_SOURCES}" != "false" ]]; then
- compose_ci_file=${compose_ci_file}:${local_docker_compose_file}:${backend_port_docker_compose_file}
- fi
-
- if [[ "${MOUNT_ALL_LOCAL_SOURCES}" != "false" ]]; then
- compose_ci_file=${compose_ci_file}:${local_all_sources_docker_compose_file}:${backend_port_docker_compose_file}
- fi
-
- if [[ ${FORWARD_CREDENTIALS} == "true" ]]; then
- compose_ci_file=${compose_ci_file}:${forward_credentials_docker_compose_file}
- fi
-
- if [[ -n ${INSTALL_AIRFLOW_VERSION=} || -n ${INSTALL_AIRFLOW_REFERENCE} || -n ${USE_AIRFLOW_VERSION=} ]]; then
- compose_ci_file=${compose_ci_file}:${remove_sources_docker_compose_file}
- fi
-
- set +u
- local unique_integrations
- # shellcheck disable=SC2207
- unique_integrations=($(echo "${INTEGRATIONS[@]}" | tr ' ' '\n' | sort -u | tr '\n' ' '))
-
- local integration
- for integration in "${unique_integrations[@]}"; do
- compose_ci_file=${compose_ci_file}:${SCRIPTS_CI_DIR}/docker-compose/integration-${integration}.yml
- done
- set -u
-
- export DOCKER_COMPOSE_RUN_SCRIPT_FOR_CI="dc_ci"
- readonly DOCKER_COMPOSE_RUN_SCRIPT_FOR_CI
-
- breeze::prepare_command_file "${BUILD_CACHE_DIR}/${DOCKER_COMPOSE_RUN_SCRIPT_FOR_CI}" \
- "\"\${@}\"" "${compose_ci_file}"
-}
-
-#######################################################################################################
-#
-# Prints detailed help for all commands and flags. Used to generate documentation added to BREEZE.rst
-# automatically.
-#
-# Used global variables:
-# _breeze_all_commands
-#
-# Outputs:
-# Prints detailed help for all commands to stdout.
-#
-#######################################################################################################
-function breeze::do_help_all() {
- echo
- breeze::print_line
- breeze::usage
- breeze::print_line
- echo
- echo
- echo "Detailed usage"
- echo
- breeze::print_line
- echo
- local subcommand
- # shellcheck disable=SC2154
- for subcommand in ${_breeze_all_commands}; do
- breeze::detailed_usage "${subcommand}"
- breeze::print_line
- echo
- done
- echo
- breeze::flags
-}
-
-#######################################################################################################
-#
-# Parses all arguments that can be passed to Breeze command - that includes command to run and flags.
-#
-# Used global variables:
-# _breeze_getopt_short_options
-# _breeze_getopt_long_options
-# _breeze_allowed_integrations
-#
-# Updated global constants:
-# By the end of this function, all the constants from `initialization::make_constants_read_only`
-# function are set and they are set as read-only.
-#
-#######################################################################################################
-function breeze::parse_arguments() {
- set -u
- local params
- if ! params=$(getopt \
- -o "${_breeze_getopt_short_options:=}" \
- -l "${_breeze_getopt_long_options:=}" \
- --name "$CMDNAME" -- "$@"); then
- breeze::flags
- exit 1
- fi
-
- eval set -- "${params}"
- unset params
-
- # Parse Flags.
- # Please update short and long options in the breeze-complete script
- # This way autocomplete will work out-of-the-box
- while true; do
- case "${1}" in
- -h | --help)
- run_help="true"
- shift
- ;;
- -p | --python)
- export PYTHON_MAJOR_MINOR_VERSION="${2}"
- echo "Python version: ${PYTHON_MAJOR_MINOR_VERSION}"
- echo
- shift 2
- ;;
- -d | --debian-version)
- export DEBIAN_VERSION="${2}"
- echo "Debian version: ${DEBIAN_VERSION}"
- echo
- shift 2
- ;;
- -b | --backend)
- export BACKEND="${2}"
- echo "Backend: ${BACKEND}"
- echo
- shift 2
- ;;
- --platform)
- export PLATFORM="${2}"
- echo "Backend: ${PLATFORM}"
- echo
- shift 2
- ;;
- -i | --integration)
- local INTEGRATION=${2}
- parameters::check_and_save_allowed_param "INTEGRATION" "integration" "--integration"
- echo "Integration: ${INTEGRATION}"
- if [[ ${INTEGRATION} == "all" ]]; then
- # shellcheck disable=SC2154
- for INTEGRATION in ${_breeze_allowed_integrations}; do
- if [[ ${INTEGRATION} != "all" ]]; then
- echo "${INTEGRATION}"
- INTEGRATIONS+=("${INTEGRATION}")
- fi
- done
- else
- INTEGRATIONS+=("${INTEGRATION}")
- fi
- if [[ " ${INTEGRATIONS[*]} " =~ " trino " ]]; then
- INTEGRATIONS+=("kerberos");
- fi
- echo
- shift 2
- ;;
- -K | --kubernetes-mode)
- export KUBERNETES_MODE="${2}"
- echo "Kubernetes mode: ${KUBERNETES_MODE}"
- echo
- shift 2
- ;;
- -V | --kubernetes-version)
- export KUBERNETES_VERSION="${2}"
- echo "Kubernetes version: ${KUBERNETES_VERSION}"
- echo
- shift 2
- ;;
- --kind-version)
- export KIND_VERSION="${2}"
- echo "Kind version: ${KIND_VERSION}"
- echo
- shift 2
- ;;
- --executor)
- export EXECUTOR="${2}"
- echo "Using ${EXECUTOR} in cluster"
- echo
- shift 2
- ;;
- --helm-version)
- export HELM_VERSION="${2}"
- echo "Helm version: ${HELM_VERSION}"
- echo
- shift 2
- ;;
- --postgres-version)
- export POSTGRES_VERSION="${2}"
- echo "Postgres version: ${POSTGRES_VERSION}"
- echo
- shift 2
- ;;
- --mysql-version)
- export MYSQL_VERSION="${2}"
- echo "MySQL version: ${MYSQL_VERSION}"
- echo
- shift 2
- ;;
- --mssql-version)
- export MSSQL_VERSION="${2}"
- echo "MSSQL version: ${MSSQL_VERSION}"
- echo
- shift 2
- ;;
- -l | --skip-mounting-local-sources)
- MOUNT_SELECTED_LOCAL_SOURCES="false"
- MOUNT_ALL_LOCAL_SOURCES="false"
- echo "Mount selected local sources: ${MOUNT_SELECTED_LOCAL_SOURCES}"
- echo "Mount all local sources: ${MOUNT_ALL_LOCAL_SOURCES}"
- echo
- shift
- ;;
- --mount-all-local-sources)
- MOUNT_ALL_LOCAL_SOURCES="true"
- MOUNT_SELECTED_LOCAL_SOURCES="false"
- echo "Mount selected local sources: ${MOUNT_SELECTED_LOCAL_SOURCES}"
- echo "Mount all local sources: ${MOUNT_ALL_LOCAL_SOURCES}"
- echo
- shift
- ;;
- -a | --install-airflow-version)
- INSTALL_AIRFLOW_VERSION="${2}"
- # Reference is mutually exclusive with version
- INSTALL_AIRFLOW_REFERENCE=""
- echo "Installs version of Airflow: ${INSTALL_AIRFLOW_VERSION}"
- echo
- shift 2
- ;;
- -t | --install-airflow-reference)
- INSTALL_AIRFLOW_REFERENCE="${2}"
- # Reference is mutually exclusive with version
- INSTALL_AIRFLOW_VERSION=""
- # Skip mounting local sources when airflow is installed from remote
- INSTALL_PROVIDERS_FROM_SOURCES="false"
- echo "Installs Airflow from reference: ${INSTALL_AIRFLOW_REFERENCE}"
- echo
- shift 2
- ;;
- --use-airflow-version)
- USE_AIRFLOW_VERSION="${2}"
- MOUNT_SELECTED_LOCAL_SOURCES="false"
- MOUNT_ALL_LOCAL_SOURCES="false"
- DB_RESET="true"
- echo "Uses Airflow at runtime from: ${USE_AIRFLOW_VERSION}, reset DB and skip mounting local sources"
- echo
- shift 2
- ;;
- --db-reset)
- echo "Resetting the DB!"
- echo
- export DB_RESET="true"
- shift
- ;;
- -v | --verbose)
- export VERBOSE="true"
- echo "Verbose output"
- echo
- shift
- ;;
- -y | --assume-yes)
- export FORCE_ANSWER_TO_QUESTIONS="yes"
- echo "Assuming 'yes' answer to all questions."
- echo
- shift
- ;;
- -n | --assume-no)
- export FORCE_ANSWER_TO_QUESTIONS="no"
- echo "Assuming 'no' answer to all questions."
- echo
- shift
- ;;
- -q | --assume-quit)
- export FORCE_ANSWER_TO_QUESTIONS="quit"
- echo "Assuming 'quit' answer to all questions."
- echo
- shift
- ;;
- -F | --force-build-images)
- echo "Force build images"
- echo
- export FORCE_BUILD_IMAGES="true"
- # if you want to force build an image - assume you want to build it :)
- export FORCE_ANSWER_TO_QUESTIONS="yes"
- shift
- ;;
- -C | --force-clean-images)
- echo "Clean build of images without cache"
- echo
- export DOCKER_CACHE="disabled"
- # if not set here, docker cached is determined later, depending on type of image to be built
- export FORCE_BUILD_IMAGES="true"
- shift
- ;;
- -r | --skip-rebuild-check)
- echo "Skips checking image for rebuilds"
- echo
- export CHECK_IMAGE_FOR_REBUILD="false"
- export SKIP_BUILDING_PROD_IMAGE="true"
- shift
- ;;
- -L | --build-cache-local)
- echo "Use local cache to build images"
- echo
- export DOCKER_CACHE="local"
- # if not set here, docker cached is determined later, depending on type of image to be built
- shift
- ;;
- -U | --build-cache-pulled)
- echo "Use pulled cache to build images"
- echo
- export DOCKER_CACHE="pulled"
- # if not set here, docker cached is determined later, depending on type of image to be built
- shift
- ;;
- -X | --build-cache-disabled)
- echo "Use disabled cache to build images"
- echo
- export DOCKER_CACHE="disabled"
- # if not set here, docker cached is determined later, depending on type of image to be built
- shift
- ;;
- -I | --production-image)
- export PRODUCTION_IMAGE="true"
- export SQLITE_URL=
- echo
- echo "*************** PRODUCTION IMAGE *************************"
- echo
- shift
- ;;
- --disable-pypi-when-building)
- export INSTALL_FROM_PYPI="false"
- export AIRFLOW_PRE_CACHED_PIP_PACKAGES="false"
- echo "Disable installing airflow from PYPI"
- shift
- ;;
- -E | --extras)
- export AIRFLOW_EXTRAS="${2}"
- echo "Extras : ${AIRFLOW_EXTRAS}"
- shift 2
- ;;
- --additional-extras)
- export ADDITIONAL_AIRFLOW_EXTRAS="${2}"
- echo "Additional extras : ${ADDITIONAL_AIRFLOW_EXTRAS}"
- shift 2
- ;;
- --additional-python-deps)
- export ADDITIONAL_PYTHON_DEPS="${2}"
- echo "Additional python dependencies: ${ADDITIONAL_PYTHON_DEPS}"
- shift 2
- ;;
- --dev-apt-deps)
- export DEV_APT_DEPS="${2}"
- echo "Apt dev dependencies: ${DEV_APT_DEPS}"
- shift 2
- ;;
- --additional-dev-apt-deps)
- export ADDITIONAL_DEV_APT_DEPS="${2}"
- echo "Additional apt dev dependencies: ${ADDITIONAL_DEV_APT_DEPS}"
- shift 2
- ;;
- --dev-apt-command)
- export DEV_APT_COMMAND="${2}"
- echo "Apt dev command: ${DEV_APT_COMMAND}"
- shift 2
- ;;
- --additional-dev-apt-command)
- export ADDITIONAL_DEV_APT_COMMAND="${2}"
- echo "Additional Apt dev command: ${ADDITIONAL_DEV_APT_COMMAND}"
- shift 2
- ;;
- --additional-dev-apt-env)
- export ADDITIONAL_DEV_APT_ENV="${2}"
- echo "Additional Apt dev environment variables: ${ADDITIONAL_DEV_APT_ENV}"
- shift 2
- ;;
- --runtime-apt-deps)
- export RUNTIME_APT_DEPS="${2}"
- echo "Apt runtime dependencies: ${RUNTIME_APT_DEPS}"
- shift 2
- ;;
- --additional-runtime-apt-deps)
- export ADDITIONAL_RUNTIME_APT_DEPS="${2}"
- echo "Additional apt runtime dependencies: ${ADDITIONAL_RUNTIME_APT_DEPS}"
- shift 2
- ;;
- --runtime-apt-command)
- export RUNTIME_APT_COMMAND="${2}"
- echo "Apt runtime command: ${RUNTIME_APT_COMMAND}"
- shift 2
- ;;
- --additional-runtime-apt-command)
- export ADDITIONAL_RUNTIME_APT_COMMAND="${2}"
- echo "Additional Apt runtime command: ${ADDITIONAL_RUNTIME_APT_COMMAND}"
- shift 2
- ;;
- --additional-runtime-apt-env)
- export ADDITIONAL_RUNTIME_APT_ENV="${2}"
- echo "Additional Apt runtime environment variables: ${ADDITIONAL_RUNTIME_APT_ENV}"
- shift 2
- ;;
- --disable-mysql-client-installation)
- export INSTALL_MYSQL_CLIENT="false"
- echo "Install MySQL client: ${INSTALL_MYSQL_CLIENT}"
- shift
- ;;
- --disable-mssql-client-installation)
- export INSTALL_MSSQL_CLIENT="false"
- echo "Install MsSQL client: ${INSTALL_MSSQL_CLIENT}"
- shift
- ;;
- --disable-postgres-client-installation)
- export INSTALL_POSTGRES_CLIENT="false"
- echo "Install Postgres client: ${INSTALL_POSTGRES_CLIENT}"
- shift
- ;;
- --constraints-location)
- export AIRFLOW_CONSTRAINTS_LOCATION="${2}"
- echo "Constraints location: ${AIRFLOW_CONSTRAINTS_LOCATION}"
- shift 2
- ;;
- --disable-pip-cache)
- echo "Disable PIP cache during build"
- echo
- export AIRFLOW_PRE_CACHED_PIP_PACKAGES="false"
- shift
- ;;
- --install-from-docker-context-files)
- export INSTALL_FROM_DOCKER_CONTEXT_FILES="true"
- echo "Install wheels from local docker-context-files when building image"
- shift
- ;;
- --image-tag)
- export IMAGE_TAG="${2}"
- echo "Tag to add to the image: ${IMAGE_TAG}"
- shift 2
- ;;
- -f | --forward-credentials)
- echo "Forwarding credentials. Be careful as your credentials are available in the container!"
- echo
- export FORWARD_CREDENTIALS="true"
- shift
- ;;
- -g | --github-repository)
- echo
- echo "Using GitHub registry."
- echo "GitHub repository: ${2}"
- echo
- export GITHUB_REPOSITORY="${2}"
- shift 2
- ;;
- -s | --github-image-id)
- echo
- echo "GitHub image id: ${2}"
- echo
- echo "Force pulling the image, using GitHub registry and skip mounting local sources."
- echo "This is in order to get the exact same version as used in CI environment for SHA!."
- echo "You can specify --skip-mounting-local-sources to not mount local sources to get exact. "
- echo "behaviour as in the CI environment."
- echo
- export GITHUB_REGISTRY_PULL_IMAGE_TAG="${2}"
- export CHECK_IMAGE_FOR_REBUILD="false"
- export SKIP_BUILDING_PROD_IMAGE="true"
- export SKIP_CHECK_REMOTE_IMAGE="true"
- shift 2
- ;;
- --init-script)
- export INIT_SCRIPT_FILE="${2}"
- echo "The initialization file is in ${INIT_SCRIPT_FILE}"
- echo
- shift 2
- ;;
- -S | --version-suffix-for-pypi)
- export VERSION_SUFFIX_FOR_PYPI="${2}"
- echo "Version suffix for PyPI ${VERSION_SUFFIX_FOR_PYPI}"
- echo
- shift 2
- ;;
- --load-example-dags)
- export LOAD_EXAMPLES="true"
- echo "Include Airflow sample dags"
- echo
- shift
- ;;
- --load-default-connections)
- export LOAD_DEFAULT_CONNECTIONS="true"
- echo "Include Airflow default connections"
- echo
- shift
- ;;
- --preserve-volumes)
- export PRESERVE_VOLUMES="true"
- echo "Preserves data volumes when stopping airflow"
- echo
- shift
- ;;
- --cleanup-docker-context-files)
- export CLEANUP_DOCKER_CONTEXT_FILES="true"
- echo "Preserves data volumes when stopping airflow"
- echo
- shift
- ;;
- --use-packages-from-dist)
- export USE_PACKAGES_FROM_DIST="true"
- echo "Install packages found in dist folder when entering breeze."
- echo
- shift
- ;;
- --upgrade-to-newer-dependencies)
- export UPGRADE_TO_NEWER_DEPENDENCIES="true"
- echo "Upgrade packages to latest versions."
- echo
- shift
- ;;
- --package-format)
- export PACKAGE_FORMAT="${2}"
- echo "Selected package type: ${PACKAGE_FORMAT}"
- echo
- shift 2
- ;;
- --non-interactive)
- export NON_INTERACTIVE="true"
- echo "Running in non-interactive mode"
- echo
- shift
- ;;
- --generate-providers-issue)
- export GENERATE_PROVIDERS_ISSUE="true"
- export FORWARD_CREDENTIALS="true"
- echo "Generating providers issue (includes forwarding credentials)"
- echo
- shift
- ;;
- --installation-method)
- export AIRFLOW_INSTALLATION_METHOD="${2}"
- echo "Airflow installation method: ${AIRFLOW_INSTALLATION_METHOD}"
- echo
- shift 2
- ;;
- --generate-constraints-mode)
- export GENERATE_CONSTRAINTS_MODE="${2}"
- echo "Generate constraints mode: ${GENERATE_CONSTRAINTS_MODE}"
- echo
- shift 2
- ;;
- --test-type)
- export TEST_TYPE="${2}"
- echo "Selected test type: ${TEST_TYPE}"
- echo
- shift 2
- ;;
- --dry-run-docker)
- export DRY_RUN_DOCKER="true"
- echo "Dry run mode"
- echo
- shift
- ;;
- --)
- shift
- break
- ;;
- *)
- breeze::flags
- echo
- echo "${COLOR_RED}ERROR: Unknown flag ${COLOR_RESET}"
- echo
- exit 1
- ;;
- esac
- done
- local last_subcommand=""
- # Parse commands
- if [[ "$#" -ne 0 ]]; then
- case "${1}" in
- shell)
- last_subcommand="${1}"
- shift
- ;;
- exec)
- last_subcommand="${1}"
- command_to_run="run_exec"
- shift
- ;;
- build-docs)
- last_subcommand="${1}"
- command_to_run="build_docs"
- shift
- ;;
- build-image)
- last_subcommand="${1}"
- command_to_run="build_image"
- # if you want to build an image - assume you want to build it :)
- export FORCE_ANSWER_TO_QUESTIONS="yes"
- # and assume you want to build it no matter if it is needed
- export FORCE_BUILD_IMAGES="true"
- echo "Build image"
- echo
- shift
- ;;
- prepare-build-cache)
- last_subcommand="${1}"
- command_to_run="prepare_build_cache"
- export FORCE_ANSWER_TO_QUESTIONS="yes"
- # and assume you want to build it no matter if it is needed
- export FORCE_BUILD_IMAGES="true"
- export PREPARE_BUILDX_CACHE="true"
- export DOCKER_CACHE="pulled"
- export CLEANUP_DOCKER_CONTEXT_FILES="true"
- echo "Prepare buildx cache"
- echo
- shift
- ;;
- cleanup-image)
- last_subcommand="${1}"
- echo "Cleanup the image"
- echo
- command_to_run="cleanup_image"
- shift
- ;;
- docker-compose)
- last_subcommand="${1}"
- if [[ $# -lt 2 ]]; then
- echo "You should specify docker compose command to run"
- shift
- run_help="true"
- else
- docker_compose_command="${2}"
- shift 2
- fi
- command_to_run="run_docker_compose"
- ;;
- generate-constraints)
- last_subcommand="${1}"
- command_to_run="perform_generate_constraints"
- export FORCE_ANSWER_TO_QUESTIONS="yes"
- export FORCE_BUILD_IMAGES="true"
- export SKIP_CHECK_REMOTE_IMAGE="true"
- export UPGRADE_TO_NEWER_DEPENDENCIES="true"
- shift
- ;;
- prepare-airflow-packages)
- last_subcommand="${1}"
- command_to_run="perform_prepare_airflow_packages"
- export INSTALL_PROVIDERS_FROM_SOURCES="false"
- shift
- ;;
- prepare-provider-packages)
- last_subcommand="${1}"
- command_to_run="perform_prepare_provider_packages"
- shift
- ;;
- prepare-provider-documentation)
- last_subcommand="${1}"
- command_to_run="perform_prepare_provider_documentation"
- shift
- ;;
- initialize-local-virtualenv)
- last_subcommand="${1}"
- echo "Initializing local virtualenv"
- echo
- command_to_run="perform_initialize_local_virtualenv"
- export INSTALL_PROVIDERS_FROM_SOURCES="true"
- shift
- ;;
- kind-cluster)
- last_subcommand="${1}"
- # Switch to production image for all kind operations
- export PRODUCTION_IMAGE="true"
- command_to_run="manage_kind_cluster"
- export KIND_CLUSTER_OPERATION="${2:-}"
- if [[ -n ${KIND_CLUSTER_OPERATION=} ]]; then
- shift 2
+ if [[ ${breeze_on_path} != 0 ]]; then
+ echo "${COLOR_RED}The '${BREEZE_BINARY}' is not on path. Breeze should be installed and 'breeze' should be available on your PATH!${COLOR_RESET}"
+ export TIMEOUT=0
+ if "${MY_DIR}/scripts/tools/confirm" "Installing breeze?"; then
+ pipx ensurepath --force
+ pipx install -e "${MY_DIR}/dev/breeze/" --force
+ ${BREEZE_BINARY} setup-autocomplete --force --answer yes
+ echo
+ echo "${COLOR_YELLOW}Please close and re-open the shell and retry. Then rerun your last command!${COLOR_RESET}"
+ echo
+ exit
else
- shift
+ manual_instructions
fi
- ;;
- setup-autocomplete)
- last_subcommand="${1}"
- echo "Setting up autocomplete"
- echo
- command_to_run="perform_setup_autocomplete"
- shift
- ;;
- static-check)
- last_subcommand="${1}"
- command_to_run="perform_static_checks"
- if [[ "$#" -lt 2 ]]; then
- if [[ ${run_help} != "true" ]]; then
- echo "You should specify static check that you would like to run or 'all' to run all checks."
- echo
- echo "One of :"
- echo
- echo "${_breeze_allowed_static_checks:=}"
- echo
- echo "For example:"
- echo
- echo "${CMDNAME} static-check mypy"
- echo
- exit 1
- else
- shift
- fi
- else
- export PYTHON_MAJOR_MINOR_VERSION=${DEFAULT_PYTHON_MAJOR_MINOR_VERSION}
- static_check="${2:-}"
- EXTRA_STATIC_CHECK_OPTIONS+=("--show-diff-on-failure")
- shift 2
- fi
- ;;
- start-airflow)
- last_subcommand="${1}"
- export START_AIRFLOW="true"
- shift
- ;;
- stop)
- last_subcommand="${1}"
- command_to_run="run_docker_compose"
- docker_compose_command="down"
- EXTRA_DC_OPTIONS+=("--remove-orphans")
- for INTEGRATION in ${_breeze_allowed_integrations}; do
- if [[ ${INTEGRATION} != "all" ]]; then
- INTEGRATIONS+=("${INTEGRATION}")
- fi
- done
- # In case of tmpfs backend for docker, mssql fails because TMPFS does not support
- # O_DIRECT parameter for direct writing to the filesystem
- # https://github.com/microsoft/mssql-docker/issues/13
- # so we need to mount an external volume for its db location
- # the external db must allow for parallel testing so external volume is mapped
- # to the data volume. Stop should also clean the volume
- rm -rf "${MSSQL_DATA_VOLUME:?"MSSQL_DATA_VOLUME should never be empty!"}"/*
- shift
- ;;
- restart)
- last_subcommand="${1}"
- command_to_run="run_docker_compose"
- docker_compose_command="down"
- EXTRA_DC_OPTIONS+=("--remove-orphans")
- second_command_to_run="enter_breeze"
- echo "Restarts the environment. Includes emptying the databases."
- shift
- ;;
- tests)
- last_subcommand="${1}"
- command_to_run="run_tests"
- shift
- ;;
- toggle-suppress-cheatsheet)
- last_subcommand="${1}"
- if [[ -f "${SUPPRESS_CHEATSHEET_FILE}" ]]; then
- rm -f "${SUPPRESS_CHEATSHEET_FILE}"
- else
- touch "${SUPPRESS_CHEATSHEET_FILE}"
- fi
- echo "Toggle suppress cheatsheet"
- echo
- shift
- command_to_run="toggle_suppress_cheatsheet"
- ;;
- toggle-suppress-asciiart)
- last_subcommand="${1}"
- if [[ -f "${SUPPRESS_ASCIIART_FILE}" ]]; then
- rm -f "${SUPPRESS_ASCIIART_FILE}"
- else
- touch "${SUPPRESS_ASCIIART_FILE}"
- fi
- echo "Toggle suppress asciiart"
- echo
- shift
- command_to_run="toggle_suppress_asciiart"
- ;;
- flags)
- breeze::flags
- exit 0
- ;;
- help)
- breeze::usage
- exit 0
- ;;
- help-all)
- breeze::do_help_all
- exit 0
- ;;
- *)
- breeze::usage
- echo
- echo "${COLOR_RED}ERROR: Unknown command ${COLOR_RESET}"
- echo
- exit 1
- ;;
- esac
- else
- :
- # By default, start interactive terminal
- fi
-
- if [[ ${run_help} == "true" ]]; then
- if [[ ${last_subcommand} == "" ]]; then
- breeze::usage
- breeze::flag_footer
- else
- breeze::detailed_usage "${last_subcommand}"
fi
- exit 0
- fi
-
- if [[ ${PRESERVE_VOLUMES} != "true" ]]; then
- EXTRA_DC_OPTIONS+=("--volumes")
- fi
- # EXTRA_DC_OPTIONS is only used by Breeze. It's value is set here as well.
- readonly EXTRA_DC_OPTIONS
-
- # Also Remaining args are set here and set as read-only - no more changes to it.
- REMAINING_ARGS+=("$@")
- export REMAINING_ARGS
- readonly REMAINING_ARGS
-}
-
-#######################################################################################################
-#
-# Prepares nicely formatted versions of list of allowed and default values defined in Breeze.
-# It is used in help command to print the lists in a readable format and fold the lists
-# so that they fit the screen indented_screen_width.
-#
-# Used global variables:
-# _breeze_allowed_*
-#
-# Updated global constants:
-# FORMATTED_* constant variables that can be used in Breeze Help output
-#
-#######################################################################################################
-function breeze::prepare_formatted_versions() {
- if [[ -n "${_breeze_formatted_versions_prepared:-}" ]]; then
- return
fi
-
- _breeze_formatted_versions_prepared=1
-
- local indent=15
- local list_prefix
- list_prefix=$(printf "%-${indent}s" " ")
- local indented_screen_width=$((SCREEN_WIDTH - indent))
-
- FORMATTED_PYTHON_MAJOR_MINOR_VERSIONS=$(echo "${_breeze_allowed_python_major_minor_versions=""}" | tr '\n' ' ' |
- fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
- readonly FORMATTED_PYTHON_MAJOR_MINOR_VERSIONS
-
- FORMATTED_DEBIAN_VERSIONS=$(echo "${_breeze_allowed_debian_versions=""}" | tr '\n' ' ' |
- fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
- readonly FORMATTED_DEBIAN_VERSIONS
-
- FORMATTED_BACKENDS=$(echo "${_breeze_allowed_backends=""}" | tr '\n' ' ' |
- fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
- readonly FORMATTED_BACKENDS
-
- FORMATTED_PLATFORMS=$(echo "${_breeze_allowed_platforms=""}" |
- tr '\n' ' ' | fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
- readonly FORMATTED_PLATFORMS
-
- FORMATTED_STATIC_CHECKS=$(echo "${_breeze_allowed_static_checks=""}" | tr '\n' ' ' |
- fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
- readonly FORMATTED_STATIC_CHECKS
-
- FORMATTED_INTEGRATIONS=$(echo "${_breeze_allowed_integrations=""}" | tr '\n' ' ' |
- fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
- readonly FORMATTED_INTEGRATIONS
-
- FORMATTED_KUBERNETES_MODES=$(echo "${_breeze_allowed_kubernetes_modes=""}" | tr '\n' ' ' |
- fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
- readonly FORMATTED_KUBERNETES_MODES
-
- FORMATTED_KUBERNETES_VERSIONS=$(echo "${_breeze_allowed_kubernetes_versions=""}" | tr '\n' ' ' |
- fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
- readonly FORMATTED_KUBERNETES_VERSIONS
-
- FORMATTED_KIND_VERSIONS=$(echo "${_breeze_allowed_kind_versions=""}" | tr '\n' ' ' |
- fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
- readonly FORMATTED_KIND_VERSIONS
-
- FORMATTED_HELM_VERSIONS=$(echo "${_breeze_allowed_helm_versions=""}" | tr '\n' ' ' |
- fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
- readonly FORMATTED_HELM_VERSIONS
-
- FORMATTED_EXECUTORS=$(echo "${_breeze_allowed_executors=""}" | tr '\n' ' ' |
- fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
- readonly FORMATTED_EXECUTORS
-
- FORMATTED_KIND_OPERATIONS=$(echo "${_breeze_allowed_kind_operations=""}" | tr '\n' ' ' |
- fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
- readonly FORMATTED_KIND_OPERATIONS
-
- FORMATTED_INSTALL_AIRFLOW_VERSIONS=$(echo "${_breeze_allowed_install_airflow_versions=""}" |
- tr '\n' ' ' | fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
- readonly FORMATTED_INSTALL_AIRFLOW_VERSIONS
-
- FORMATTED_INSTALLATION_METHOD=$(echo "${_breeze_allowed_installation_methods=""}" |
- tr '\n' ' ' | fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
- readonly FORMATTED_INSTALLATION_METHOD
-
- FORMATTED_GENERATE_CONSTRAINTS_MODE=$(echo "${_breeze_allowed_generate_constraints_modes=""}" |
- tr '\n' ' ' | fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
- readonly FORMATTED_GENERATE_CONSTRAINTS_MODE
-
- FORMATTED_POSTGRES_VERSIONS=$(echo "${_breeze_allowed_postgres_versions=""}" |
- tr '\n' ' ' | fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
- readonly FORMATTED_POSTGRES_VERSIONS
-
- FORMATTED_MYSQL_VERSIONS=$(echo "${_breeze_allowed_mysql_versions=""}" |
- tr '\n' ' ' | fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
- readonly FORMATTED_MYSQL_VERSIONS
-
- FORMATTED_MSSQL_VERSIONS=$(echo "${_breeze_allowed_mssql_versions=""}" |
- tr '\n' ' ' | fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
- readonly FORMATTED_MSSQL_VERSIONS
-
- FORMATTED_DEFAULT_CI_EXTRAS=$(echo "${DEFAULT_CI_EXTRAS=}" |
- tr ',' ' ' | fold -w "${indented_screen_width}" -s | sed "s/ /,/g; s/^/${list_prefix}/")
- readonly FORMATTED_DEFAULT_CI_EXTRAS
-
- FORMATTED_DEFAULT_PROD_EXTRAS=$(echo "${DEFAULT_PROD_EXTRAS=}" |
- tr ',' ' ' | fold -w "${indented_screen_width}" -s | sed "s/ /,/g; s/^/${list_prefix}/")
- readonly FORMATTED_DEFAULT_PROD_EXTRAS
-
- FORMATTED_TEST_TYPES=$(echo "${_breeze_allowed_test_types=""}" |
- fold -w "${indented_screen_width}" -s | sed "s/ /,/g; s/^/${list_prefix}/")
- readonly FORMATTED_TEST_TYPES
-
- FORMATTED_PACKAGE_FORMATS=$(echo "${_breeze_allowed_package_formats=""}" |
- fold -w "${indented_screen_width}" -s | sed "s/ /,/g; s/^/${list_prefix}/")
- readonly FORMATTED_PACKAGE_FORMATS
-
- FORMATTED_USE_AIRFLOW_VERSION=$(echo "${_breeze_allowed_USE_AIRFLOW_VERSION=""}" |
- tr '\n' ' ' | fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
- readonly FORMATTED_USE_AIRFLOW_VERSION
-
}
-#######################################################################################################
-#
-# Prepares usage information for all the commands in Breeze.
-# Those usage commands are stored in appropriate environment variables.
-#
-# Created constants:
-# USAGE_*
-# DETAILED_USAGE_*
-#
-#######################################################################################################
-
-# shellcheck disable=SC2034,SC2090,SC2089,SC2155
-
-function breeze::prepare_usage() {
- if [[ -n "${_breeze_usage_prepared:-}" ]]; then
- return
- fi
-
- _breeze_usage_prepared=1
-
- breeze::prepare_formatted_versions
-
- # Note that MacOS uses Bash 3.* and we cannot use associative arrays
- export USAGE_SHELL="[Default] Enters interactive shell in the container"
- readonly USAGE_SHELL
- export USAGE_EXEC="Execs into running breeze container in new terminal"
- readonly USAGE_EXEC
- export USAGE_BUILD_DOCS="Builds documentation in the container"
- readonly USAGE_BUILD_DOCS
- export USAGE_BUILD_IMAGE="Builds CI or Production docker image"
- readonly USAGE_BUILD_DOCS
- export USAGE_PREPARE_BUILD_CACHE="Prepares CI or Production build cache"
- readonly USAGE_PREPARE_BUILD_CACHE
- export USAGE_CLEANUP_IMAGE="Cleans up the container image created"
- readonly USAGE_BUILD_DOCS
- export USAGE_DOCKER_COMPOSE="Executes specified docker-compose command"
- readonly USAGE_DOCKER_COMPOSE
- export USAGE_FLAGS="Shows all breeze's flags"
- readonly USAGE_FLAGS
- export USAGE_GENERATE_CONSTRAINTS="Generates pinned constraint files"
- readonly USAGE_GENERATE_CONSTRAINTS
- export USAGE_INITIALIZE_LOCAL_VIRTUALENV="Initializes local virtualenv"
- readonly USAGE_INITIALIZE_LOCAL_VIRTUALENV
- export USAGE_PREPARE_PROVIDER_DOCUMENTATION="Prepares provider packages documentation"
- readonly USAGE_PREPARE_PROVIDER_DOCUMENTATION
- export USAGE_PREPARE_AIRFLOW_PACKAGES="Prepares airflow packages"
- readonly USAGE_PREPARE_AIRFLOW_PACKAGES
- export USAGE_PREPARE_PROVIDER_PACKAGES="Prepares provider packages"
- readonly USAGE_PREPARE_PROVIDER_PACKAGES
- export USAGE_PUSH_IMAGE="Pushes images to registry"
- readonly USAGE_PUSH_IMAGE
- export USAGE_KIND_CLUSTER="Manages KinD cluster on the host"
- readonly USAGE_KIND_CLUSTER
- export USAGE_SETUP_AUTOCOMPLETE="Sets up autocomplete for breeze"
- readonly USAGE_SETUP_AUTOCOMPLETE
- export USAGE_START_AIRFLOW="Starts Scheduler and Webserver and enters the shell"
- readonly USAGE_START_AIRFLOW
- export USAGE_STOP="Stops the docker-compose environment"
- readonly USAGE_STOP
- export USAGE_RESTART="Stops the docker-compose environment including DB cleanup"
- readonly USAGE_RESTART
- export USAGE_STATIC_CHECK="Performs selected static check for changed files"
- readonly USAGE_STATIC_CHECK
- export USAGE_TOGGLE_SUPPRESS_CHEATSHEET="Toggles on/off cheatsheet"
- readonly USAGE_TOGGLE_SUPPRESS_CHEATSHEET
- export USAGE_TOGGLE_SUPPRESS_ASCIIART="Toggles on/off asciiart"
- readonly USAGE_TOGGLE_SUPPRESS_ASCIIART
- export USAGE_TESTS="Runs selected tests in the container"
- readonly USAGE_TESTS
- export USAGE_HELP="Shows this help message"
- readonly USAGE_HELP
- export USAGE_HELP_ALL="Shows detailed help for all commands and flags"
- readonly USAGE_HELP_ALL
-
- export DETAILED_USAGE_SHELL="
-${CMDNAME} shell [FLAGS] [-- <EXTRA_ARGS>]
-
- This is default subcommand if no subcommand is used.
-
- Enters interactive shell where you can run all tests, start Airflow webserver, scheduler,
- workers, interact with the database, run DAGs etc. It is the default command if no command
- is selected. The shell is executed in the container and in case integrations are chosen,
- the integrations will be started as separated docker containers - under the docker-compose
- supervision. Local sources are by default mounted to within the container so you can edit
- them locally and run tests immediately in the container. Several folders ('files', 'dist')
- are also mounted so that you can exchange files between the host and container.
-
- The 'files/airflow-breeze-config/variables.env' file can contain additional variables
- and setup. This file is automatically sourced when you enter the container. Database
- and webserver ports are forwarded to appropriate database/webserver so that you can
- connect to it from your host environment.
-
- You can also pass <EXTRA_ARGS> after -- they will be passed as bash parameters, this is
- especially useful to pass bash options, for example -c to execute command:
-
- '${CMDNAME} shell -- -c \"ls -la\"'
- '${CMDNAME} -- -c \"ls -la\"'
-
- For GitHub repository, the --github-repository flag can be used to specify the repository
- to pull and push images. You can also use --github-image-id <COMMIT_SHA> in case
- you want to pull the image with specific COMMIT_SHA tag.
-
- '${CMDNAME} shell \\
- --github-image-id 9a621eaa394c0a0a336f8e1b31b35eff4e4ee86e' - pull/use image with SHA
- '${CMDNAME} \\
- --github-image-id 9a621eaa394c0a0a336f8e1b31b35eff4e4ee86e' - pull/use image with SHA
-
-Most flags are applicable to the shell command as it will run build when needed.
-"
- readonly DETAILED_USAGE_SHELL
- export DETAILED_USAGE_EXEC="
-${CMDNAME} exec [-- <EXTRA_ARGS>]
-
- Execs into interactive shell to an already running container. The container mus be started
- already by breeze shell command. If you are not familiar with tmux, this is the best
- way to run multiple processes in the same container at the same time for example scheduler,
- webserver, workers, database console and interactive terminal.
-"
- export DETAILED_USAGE_BUILD_DOCS="
-${CMDNAME} build-docs [-- <EXTRA_ARGS>]
-
- Builds Airflow documentation. The documentation is build inside docker container - to
- maintain the same build environment for everyone. Appropriate sources are mapped from
- the host to the container so that latest sources are used. The folders where documentation
- is generated ('docs/_build') are also mounted to the container - this way results of
- the documentation build is available in the host.
-
- The possible extra args are: --docs-only, --spellcheck-only, --package-filter, --help
-"
- readonly DETAILED_USAGE_BUILD_DOCS
- export DETAILED_USAGE_BUILD_IMAGE="
-${CMDNAME} build-image [FLAGS]
-
- Builds docker image (CI or production) without entering the container. You can pass
- additional options to this command, such as:
-
- Choosing python version:
- '--python'
-
- Choosing cache option:
- '--build-cache-local' or '-build-cache-pulled', or '--build-cache-none'
-
- Choosing whether to force pull images or force build the image:
- '--force-build-image'
-
- You can also pass '--production-image' flag to build production image rather than CI image.
-
- For GitHub repository, the '--github-repository' can be used to choose repository
- to pull/push images.
-
-Flags:
-$(breeze::flag_airflow_variants)
-$(breeze::flag_build_different_airflow_version)
-$(breeze::flag_production_image)
-$(breeze::flag_build_docker_images)
-$(breeze::flag_pull_push_docker_images "no_show_sha")
-$(breeze::flag_verbosity)
-"
- readonly DETAILED_USAGE_BUILD_IMAGE
- export DETAILED_USAGE_PREPARE_BUILD_CACHE="
-${CMDNAME} prepare-build-cache [FLAGS]
-
- Prepares build cache (CI or production) without entering the container. You can pass
- additional options to this command, such as:
-
- Choosing python version:
- '--python'
-
- You can also pass '--production-image' flag to build production image rather than CI image.
-
- For GitHub repository, the '--github-repository' can be used to choose repository
- to pull/push images. Cleanup docker context files and pull cache are forced. This command
- requires buildx to be installed.
-
-Flags:
-$(breeze::flag_airflow_variants)
-$(breeze::flag_build_different_airflow_version)
-$(breeze::flag_production_image)
-$(breeze::flag_pull_push_docker_images "no_show_sha")
-$(breeze::flag_verbosity)
-"
- readonly DETAILED_USAGE_PREPARE_BUILD_CACHE
- export DETAILED_USAGE_CLEANUP_IMAGE="
-${CMDNAME} cleanup-image [FLAGS]
-
- Removes the breeze-related images created in your local docker image cache. This will
- not reclaim space in docker cache. You need to 'docker system prune' (optionally
- with --all) to reclaim that space.
-
-Flags:
-$(breeze::flag_airflow_variants)
-$(breeze::flag_production_image)
-$(breeze::flag_verbosity)
-"
- readonly DETAILED_USAGE_CLEANUP_IMAGE
- export DETAILED_USAGE_DOCKER_COMPOSE="
-${CMDNAME} docker-compose [FLAGS] COMMAND [-- <EXTRA_ARGS>]
-
- Run docker-compose command instead of entering the environment. Use 'help' as command
- to see available commands. The <EXTRA_ARGS> passed after -- are treated
- as additional options passed to docker-compose. For example
-
- '${CMDNAME} docker-compose pull -- --ignore-pull-failures'
-
-Flags:
-$(breeze::flag_airflow_variants)
-$(breeze::flag_backend_variants)
-$(breeze::flag_verbosity)
-"
- readonly DETAILED_USAGE_DOCKER_COMPOSE
- export DETAILED_USAGE_FLAGS="
- Explains in detail all the flags that can be used with breeze.
-"
- readonly DETAILED_USAGE_FLAGS
- export DETAILED_USAGE_PREPARE_PROVIDER_DOCUMENTATION="
-${CMDNAME} prepare-provider-documentation [FLAGS] [PACKAGE_ID ...]
-
- Prepares documentation files for provider packages.
-
- The command is optionally followed by the list of packages to generate readme for.
- If the first parameter is not formatted as a date, then today is regenerated.
- If no packages are specified, readme for all packages are generated.
- If no date is specified, current date + 3 days is used (allowing for PMC votes to pass).
-
- Examples:
-
- '${CMDNAME} prepare-provider-documentation' or
- '${CMDNAME} prepare-provider-documentation --version-suffix-for-pypi rc1'
-
- General form:
-
- '${CMDNAME} prepare-provider-documentation <PACKAGE_ID> ...'
-
- * <PACKAGE_ID> is usually directory in the airflow/providers folder (for example
- 'google' but in several cases, it might be one level deeper separated with
- '.' for example 'apache.hive'
-
-Flags:
-$(breeze::flag_version_suffix)
-$(breeze::flag_packages)
-$(breeze::prepare_providers_documentation)
-$(breeze::flag_verbosity)
-"
- readonly DETAILED_USAGE_PREPARE_PROVIDER_DOCUMENTATION
- export DETAILED_USAGE_GENERATE_CONSTRAINTS="
-${CMDNAME} generate-constraints [FLAGS]
-
- Generates pinned constraint files with all extras from setup.py. Those files are generated in
- files folder - separate files for different python version. Those constraint files when
- pushed to orphan constraints-main, constraints-2-0 branches are used
- to generate repeatable CI test runs as well as run repeatable production image builds and
- upgrades when you want to include installing or updating some of the released providers
- released at the time particular airflow version was released. You can use those
- constraints to predictably install released Airflow versions. This is mainly used to test
- the constraint generation or manually fix them - constraints are pushed to the orphan
- branches by a successful scheduled CRON job in CI automatically, but sometimes manual fix
- might be needed.
-
-Flags:
-$(breeze::flag_generate_constraints)
-$(breeze::flag_airflow_variants)
-$(breeze::flag_verbosity)
-"
- readonly DETAILED_USAGE_GENERATE_CONSTRAINTS
- export DETAILED_USAGE_INITIALIZE_LOCAL_VIRTUALENV="
-${CMDNAME} initialize-local-virtualenv [FLAGS]
-
- Initializes locally created virtualenv installing all dependencies of Airflow
- taking into account the constraints for the version specified.
- This local virtualenv can be used to aid auto-completion and IDE support as
- well as run unit tests directly from the IDE. You need to have virtualenv
- activated before running this command.
-
-Flags:
-$(breeze::flag_airflow_variants)
-"
- readonly DETAILED_USAGE_INITIALIZE_LOCAL_VIRTUALENV
- export DETAILED_USAGE_PREPARE_AIRFLOW_PACKAGES="
-${CMDNAME} prepare-airflow-packages [FLAGS]
-
- Prepares airflow packages (sdist and wheel) in dist folder. Note that
- prepare-provider-packages command cleans up the dist folder, so if you want also
- to generate provider packages, make sure you run prepare-provider-packages first,
- and prepare-airflow-packages second. You can specify optional
- --version-suffix-for-pypi flag to generate rc candidates for PyPI packages.
- The packages are prepared in dist folder
-
- Examples:
-
- '${CMDNAME} prepare-airflow-packages --package-format wheel' or
- '${CMDNAME} prepare-airflow-packages --version-suffix-for-pypi rc1'
-
-Flags:
-$(breeze::flag_packages)
-$(breeze::flag_version_suffix)
-$(breeze::flag_verbosity)
-"
- readonly DETAILED_USAGE_PREPARE_AIRFLOW_PACKAGES
- export DETAILED_USAGE_PREPARE_PROVIDER_PACKAGES="
-${CMDNAME} prepare-provider-packages [FLAGS] [PACKAGE_ID ...]
-
- Prepares provider packages. You can provide (after --) optional list of packages to prepare.
- If no packages are specified, readme for all packages are generated. You can specify optional
- --version-suffix-for-svn flag to generate rc candidate packages to upload to SVN or
- --version-suffix-for-pypi flag to generate rc candidates for PyPI packages. You can also
- provide both suffixes in case you prepare alpha/beta versions. The packages are prepared in
- dist folder. Note that this command also cleans up dist folder before generating the packages
- so that you do not have accidental files there. This will delete airflow package if it is
- prepared there so make sure you run prepare-provider-packages first,
- and prepare-airflow-packages second.
-
- Examples:
-
- '${CMDNAME} prepare-provider-packages' or
- '${CMDNAME} prepare-provider-packages google' or
- '${CMDNAME} prepare-provider-packages --package-format wheel google' or
- '${CMDNAME} prepare-provider-packages --version-suffix-for-svn rc1 http google amazon' or
- '${CMDNAME} prepare-provider-packages --version-suffix-for-pypi rc1 http google amazon'
- '${CMDNAME} prepare-provider-packages --version-suffix-for-pypi a1
- --version-suffix-for-svn a1 http google amazon'
-
- General form:
-
- '${CMDNAME} prepare-provider-packages [--package-format PACKAGE_FORMAT] \\
- [--version-suffix-for-svn|--version-suffix-for-pypi] <PACKAGE_ID> ...'
-
- * <PACKAGE_ID> is usually directory in the airflow/providers folder (for example
- 'google'), but in several cases, it might be one level deeper separated with '.'
- for example 'apache.hive'
-
-Flags:
-$(breeze::flag_packages)
-$(breeze::flag_version_suffix)
-$(breeze::flag_verbosity)
-"
- readonly DETAILED_USAGE_PREPARE_PROVIDER_PACKAGES
- export DETAILED_USAGE_KIND_CLUSTER="
-${CMDNAME} kind-cluster [FLAGS] OPERATION
-
- Manages host-side Kind Kubernetes cluster that is used to run Kubernetes integration tests.
- It allows to start/stop/restart/status the Kind Kubernetes cluster and deploy Airflow to it.
- This enables you to run tests inside the breeze environment with latest airflow images.
- Note that in case of deploying airflow, the first step is to rebuild the image and loading it
- to the cluster so you can also pass appropriate build image flags that will influence
- rebuilding the production image. Operation is one of:
-
-${FORMATTED_KIND_OPERATIONS}
-
- The last two operations - shell and k9s allow you to perform interactive testing with
- kubernetes tests. You can enter the shell from which you can run kubernetes tests and in
- another terminal you can start the k9s CLI to debug kubernetes instance. It is an easy
- way to debug the kubernetes deployments.
-
- You can read more about k9s at https://k9scli.io/
-
-Flags:
-$(breeze::flag_airflow_variants)
-$(breeze::flag_build_docker_images)
-"
- readonly DETAILED_USAGE_KIND_CLUSTER
- export DETAILED_USAGE_SETUP_AUTOCOMPLETE="
-${CMDNAME} setup-autocomplete
-
- Sets up autocomplete for breeze commands. Once you do it you need to re-enter the bash
- shell and when typing breeze command <TAB> will provide autocomplete for
- parameters and values.
-"
- readonly DETAILED_USAGE_SETUP_AUTOCOMPLETE
- export DETAILED_USAGE_START_AIRFLOW="
-${CMDNAME} start-airflow
-
- Like the Shell command this will enter the interactive shell, but it will also start
- automatically the Scheduler and the Webserver. It will leave you in a tmux session where you
- can also observe what is happening in your Airflow.
-
- This is a convenient way to setup a development environment. Your dags will be loaded from the
- folder 'files/dags' on your host machine (it could take some times).
-
- If you want to load default connections and example dags you can use the dedicated flags.
-
-Flags:
-$(breeze::flag_use_different_airflow_version)
-$(breeze::flag_start_airflow)
-"
- readonly DETAILED_USAGE_START_AIRFLOW
- export DETAILED_USAGE_STOP="
-${CMDNAME} stop
-
- Brings down running docker compose environment. When you start the environment, the docker
- containers will continue running so that startup time is shorter. But they take quite a lot of
- memory and CPU. This command stops all running containers from the environment.
-
-Flags:
-$(breeze::flag_stop_airflow)
-"
- readonly DETAILED_USAGE_STOP
- export DETAILED_USAGE_RESTART="
-${CMDNAME} restart [FLAGS]
-
- Restarts running docker compose environment. When you restart the environment, the docker
- containers will be restarted. That includes cleaning up the databases. This is
- especially useful if you switch between different versions of Airflow.
-
-Flags:
-$(breeze::flag_stop_airflow)
-"
- readonly DETAILED_USAGE_RESTART
- export DETAILED_USAGE_STATIC_CHECK="
-${CMDNAME} static-check [FLAGS] static_check [-- <EXTRA_ARGS>]
-
- Run selected static checks for currently changed files. You should specify static check that
- you would like to run or 'all' to run all checks. One of:
-
-${FORMATTED_STATIC_CHECKS}
-
- You can pass extra arguments including options to the pre-commit framework as
- <EXTRA_ARGS> passed after --. For example:
-
- '${CMDNAME} static-check mypy' or
- '${CMDNAME} static-check mypy -- --files tests/core.py'
- '${CMDNAME} static-check mypy -- --all-files'
-
- To check all files that differ between you current branch and main run:
-
- '${CMDNAME} static-check all -- --from-ref \$(git merge-base main HEAD) --to-ref HEAD'
-
- To check all files that are in the HEAD commit run:
-
- '${CMDNAME} static-check mypy -- --from-ref HEAD^ --to-ref HEAD'
-
-
- You can see all the options by adding --help EXTRA_ARG:
-
- '${CMDNAME} static-check mypy -- --help'
-"
- readonly DETAILED_USAGE_STATIC_CHECK
- export DETAILED_USAGE_TESTS="
-${CMDNAME} tests [FLAGS] [TEST_TARGET ..] [-- <EXTRA_ARGS>]
-
- Run the specified unit test target. There might be multiple
- targets specified separated with comas. The <EXTRA_ARGS> passed after -- are treated
- as additional options passed to pytest. You can pass 'tests' as target to
- run all tests. For example:
-
- '${CMDNAME} tests tests/core/test_core.py -- --logging-level=DEBUG'
- '${CMDNAME} tests tests
-
-Flags:
-$(breeze::flag_tests)
-"
- readonly DETAILED_USAGE_TESTS
- export DETAILED_USAGE_TOGGLE_SUPPRESS_CHEATSHEET="
-${CMDNAME} toggle-suppress-cheatsheet
-
- Toggles on/off cheatsheet displayed before starting bash shell.
-"
- readonly DETAILED_USAGE_TOGGLE_SUPPRESS_CHEATSHEET
- export DETAILED_USAGE_TOGGLE_SUPPRESS_ASCIIART="
-${CMDNAME} toggle-suppress-asciiart
-
- Toggles on/off asciiart displayed before starting bash shell.
-"
- readonly DETAILED_USAGE_TOGGLE_SUPPRESS_ASCIIART
- export DETAILED_USAGE_HELP="
-${CMDNAME} help
-
- Shows general help message for all commands.
-"
- readonly DETAILED_USAGE_HELP
- export DETAILED_USAGE_HELP_ALL="
-${CMDNAME} help-all
-
- Shows detailed help for all commands and flags.
-"
- readonly DETAILED_USAGE_HELP_ALL
-}
-
-# shellcheck enable=all
-
-#######################################################################################################
-#
-# Gets environment variable value converting the lowercase name of command into variable name
-# Arguments:
-# prefix for the variable to add
-# name of the variable
-# Outputs:
-# Writes the capitalized name of the variable to stdout
-#######################################################################################################
-function breeze::get_variable_from_lowercase_name() {
- local prefix="${1}"
- local name="${2}"
- local suffix
- suffix="$(echo "${name}" | tr "[:lower:]-" "[:upper:]_")"
- local variable_name="${prefix}_${suffix}"
- echo "${!variable_name}"
-}
-
-#######################################################################################################
-#
-# Gets usage information from lowercase command
-# Arguments:
-# lowercase command name
-# Outputs:
-# usage information for the command.
-#######################################################################################################
-function breeze::get_usage() {
- breeze::prepare_usage
- breeze::get_variable_from_lowercase_name "USAGE" "${1}"
-}
-
-#######################################################################################################
-#
-# Gets detailed usage information from lowercase command
-# Arguments:
-# lowercase command name
-# Outputs:
-# Detailed usage information for the command.
-#######################################################################################################
-function breeze::get_detailed_usage() {
- breeze::prepare_usage
- breeze::get_variable_from_lowercase_name "DETAILED_USAGE" "${1}"
-}
-
-#######################################################################################################
-#
-# Prints general usage information for all commands.
-#
-# Globals used:
-# CMDNAME
-# _breeze_commands
-# _breeze_extra_arg_commands
-# _breeze_help_commands
-#
-# Outputs:
-# General usage information for all commands.
-#######################################################################################################
-function breeze::usage() {
- breeze::prepare_usage
- echo "
-
-usage: ${CMDNAME} [FLAGS] [COMMAND] -- <EXTRA_ARGS>
-
-By default the script enters the CI container and drops you to bash shell, but you can choose
-one of the commands to run specific actions instead.
-
-Add --help after each command to see details:
-
-Commands without arguments:
-"
- for subcommand in ${_breeze_commands}; do
- printf " %-40s %s\n" "${subcommand}" "$(breeze::get_usage "${subcommand}")"
- done
- echo "
-Commands with arguments:
-"
- # shellcheck disable=SC2154
- for subcommand in ${_breeze_extra_arg_commands}; do
- printf " %-35s%-10s %s\n" "${subcommand}" "<ARG>" "$(breeze::get_usage "${subcommand}")"
- done
- echo "
-Help commands:
-"
- # shellcheck disable=SC2154
- for subcommand in ${_breeze_help_commands}; do
- printf " %-40s %s\n" "${subcommand}" "$(breeze::get_usage "${subcommand}")"
- done
- echo
-}
-
-#######################################################################################################
-#
-# Prints detailed usage for command specified
-#
-# Argument:
-# Command name.
-#
-# Outputs:
-# Detailed usage information for the command
-#######################################################################################################
-# Prints detailed usage for command specified
-function breeze::detailed_usage() {
- subcommand=${1}
- echo "
-
-Detailed usage for command: ${subcommand}
-
-$(breeze::get_detailed_usage "${subcommand}")
-
-"
-}
-
-#######################################################################################################
-#
-# Prints flag footer - common to all commands..
-#
-# Outputs:
-# Footer common for all commands.
-#######################################################################################################
-function breeze::flag_footer() {
- echo "
-Run '${CMDNAME} flags' to see all applicable flags.
-"
-}
-
-#######################################################################################################
-#
-# Prints flags for different variants of airflow to use
-#
-# Global constants used:
-# FORMATTED_PYTHON_MAJOR_MINOR_VERSIONS
-#
-# Outputs:
-# Flag information.
-#######################################################################################################
-function breeze::flag_airflow_variants() {
- echo "
--p, --python PYTHON_MAJOR_MINOR_VERSION
- Python version used for the image. This is always major/minor version.
-
- One of:
-
-${FORMATTED_PYTHON_MAJOR_MINOR_VERSIONS}
-
---platform PLATFORM
- Builds image for the platform specified.
-
- One of:
-
-${FORMATTED_PLATFORMS}
-
-
--d, --debian DEBIAN_VERSION
- Debian version used for the image. This is always name of the debian distribution version.
-
- One of:
-
-${FORMATTED_DEBIAN_VERSIONS}
-
-"
-}
-
-#######################################################################################################
-#
-# Prints flags for different backend to use
-#
-# Global constants used:
-# FORMATTED_BACKENDS
-# FORMATTED_POSTGRES_VERSIONS
-# FORMATTED_MYSQL_VERSIONS
-#
-# Outputs:
-# Flag information.
-#######################################################################################################
-function breeze::flag_backend_variants() {
- echo "
--b, --backend BACKEND
- Backend to use for tests - it determines which database is used.
- One of:
-
-${FORMATTED_BACKENDS}
-
- Default: ${_breeze_default_backend:=}
-
---postgres-version POSTGRES_VERSION
- Postgres version used. One of:
-
-${FORMATTED_POSTGRES_VERSIONS}
-
---mysql-version MYSQL_VERSION
- MySql version used. One of:
-
-${FORMATTED_MYSQL_VERSIONS}
-
---mssql-version MSSQL_VERSION
- MSSql version used. One of:
-
-${FORMATTED_MSSQL_VERSIONS}
-
-"
-}
-
-#######################################################################################################
-#
-# Prints production image flags
-#
-# Outputs:
-# Flag information.
-#######################################################################################################
-function breeze::flag_production_image() {
- echo "
--I, --production-image
- Use production image for entering the environment and builds (not for tests).
-"
-}
-
-#######################################################################################################
-#
-# Prints additional breeze action flags
-#
-# Global constants used:
-# FORMATTED_INTEGRATIONS
-#
-#
-# Outputs:
-# Flag information.
-#######################################################################################################
-function breeze::flag_breeze_actions() {
- echo "
--d, --db-reset
- Resets the database at entry to the environment. It will drop all the tables
- and data and recreate the DB from scratch even if 'restart' command was not used.
- Combined with 'restart' command it enters the environment in the state that is
- ready to start Airflow webserver/scheduler/worker. Without the switch, the database
- does not have any tables and you need to run reset db manually.
-
--i, --integration INTEGRATION
- Integration to start during tests - it determines which integrations are started
- for integration tests. There can be more than one integration started, or all to
- start all integrations. Selected integrations are not saved for future execution.
- One of:
-
-${FORMATTED_INTEGRATIONS}
-
---init-script INIT_SCRIPT_FILE
- Initialization script name - Sourced from files/airflow-breeze-config. Default value
- init.sh. It will be executed after the environment is configured and started.
-
-"
-}
-
-#######################################################################################################
-#
-# Prints Kubernetes action flags
-#
-# Global constants used:
-# FORMATTED_KUBERNETES_MODE
-# FORMATTED_KUBERNETES_VERSIONS
-# FORMATTED_KIND_VERSIONS
-# FORMATTED_HELM_VERSIONS
-# FORMATTED_EXECUTORS
-#
-#
-# Outputs:
-# Flag information.
-#######################################################################################################
-function breeze::flag_kubernetes_configuration() {
- echo "
-Configuration for the KinD Kubernetes cluster and tests:
-
--K, --kubernetes-mode KUBERNETES_MODE
- Kubernetes mode - only used in case one of kind-cluster commands is used.
- One of:
-
-${FORMATTED_KUBERNETES_MODES}
-
- Default: ${_breeze_default_kubernetes_mode:=}
-
--V, --kubernetes-version KUBERNETES_VERSION
- Kubernetes version - only used in case one of kind-cluster commands is used.
- One of:
-
-${FORMATTED_KUBERNETES_VERSIONS}
-
- Default: ${_breeze_default_kubernetes_version:=}
-
---kind-version KIND_VERSION
- Kind version - only used in case one of kind-cluster commands is used.
- One of:
-
-${FORMATTED_KIND_VERSIONS}
-
- Default: ${_breeze_default_kind_version:=}
-
---helm-version HELM_VERSION
- Helm version - only used in case one of kind-cluster commands is used.
- One of:
-
-${FORMATTED_HELM_VERSIONS}
-
- Default: ${_breeze_default_helm_version:=}
-
---executor EXECUTOR
- Executor to use in a kubernetes cluster.
- One of:
-
-${FORMATTED_EXECUTORS}
-
- Default: ${_breeze_default_executor:=}
-
-"
-}
-
-#######################################################################################################
-#
-# Prints flags that determine what is the source mounting scheme
-#
-# Outputs:
-# Flag information.
-#######################################################################################################
-function breeze::flag_local_file_mounting() {
- echo "
--l, --skip-mounting-local-sources
- Skips mounting local volume with sources - you get exactly what is in the
- docker image rather than your current local sources of Airflow.
-
-"
-}
-
-#######################################################################################################
-#
-# Prints flags that allow to build different airflow variants
-#
-# Global constants used:
-# FORMATTED_INSTALL_AIRFLOW_VERSIONS
-#
-#
-# Outputs:
-# Flag information.
-#######################################################################################################
-function breeze::flag_build_different_airflow_version() {
- echo "
--a, --install-airflow-version INSTALL_AIRFLOW_VERSION
- Uses different version of Airflow when building PROD image.
-
-${FORMATTED_INSTALL_AIRFLOW_VERSIONS}
-
--t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
- Installs Airflow directly from reference in GitHub when building PROD image.
- This can be a GitHub branch like main or v2-2-test, or a tag like 2.2.0rc1.
-
---installation-method INSTALLATION_METHOD
- Method of installing Airflow in PROD image - either from the sources ('.')
- or from package 'apache-airflow' to install from PyPI.
- Default in Breeze is to install from sources. One of:
-
-${FORMATTED_INSTALLATION_METHOD}
-
---upgrade-to-newer-dependencies
- Upgrades PIP packages to latest versions available without looking at the constraints.
-
-"
-}
-
-
-#######################################################################################################
-#
-# Prints flags that allow to build different airflow variants
-#
-# Global constants used:
-# FORMATTED_INSTALL_AIRFLOW_VERSIONS
-#
-#
-# Outputs:
-# Flag information.
-#######################################################################################################
-function breeze::flag_use_different_airflow_version() {
- echo "
---use-airflow-version AIRFLOW_SPECIFICATION
- In CI image, installs Airflow at runtime from PIP released version or using
- the installation method specified (sdist, wheel, none). When 'none' is used,
- airflow is just removed. In this case airflow package should be added to dist folder
- and --use-packages-from-dist flag should be used.
-
-${FORMATTED_USE_AIRFLOW_VERSION}
-
---use-packages-from-dist
- In CI image, if specified it will look for packages placed in dist folder and
- it will install the packages after entering the image.
- This is useful for testing provider packages.
-
-"
-}
-
-
-#######################################################################################################
-#
-# Prints flags that allow to choose variants of constraint generation
-#
-# Global constants used:
-# GENERATE_CONSTRAINTS_MODES
-#
-# Outputs:
-# Flag information.
-#######################################################################################################
-function breeze::flag_generate_constraints() {
- echo "
---generate-constraints-mode GENERATE_CONSTRAINTS_MODE
- Mode of generating constraints - determines whether providers are installed when generating
- constraints and which version of them (either the ones from sources are used or the ones
- from pypi.
-
- One of:
-
-${FORMATTED_GENERATE_CONSTRAINTS_MODE}
-
-"
-}
-
-
-#######################################################################################################
-#
-# Prints flags that allow to set assumed answers to questions
-#
-# Outputs:
-# Flag information.
-#######################################################################################################
-function breeze::flag_assume_answers_to_questions() {
- echo "
--y, --assume-yes
- Assume 'yes' answer to all questions.
-
--n, --assume-no
- Assume 'no' answer to all questions.
-
--q, --assume-quit
- Assume 'quit' answer to all questions.
-"
-}
-
-#######################################################################################################
-#
-# Prints flags that are used for credential forwarding
-#
-# Outputs:
-# Flag information.
-#######################################################################################################
-function breeze::flag_credentials() {
- echo "
--f, --forward-credentials
- Forwards host credentials to docker container. Use with care as it will make
- your credentials available to everything you install in Docker.
-"
-}
-
-#######################################################################################################
-#
-# Prints flags that control package preparation
-#
-# Outputs:
-# Flag information.
-#######################################################################################################
-function breeze::flag_packages() {
- echo "
---package-format PACKAGE_FORMAT
-
- Chooses format of packages to prepare.
-
- One of:
-
-${FORMATTED_PACKAGE_FORMATS}
-
- Default: ${_breeze_default_package_format:=}
-
-"
-}
-
-#######################################################################################################
-#
-# Prints flags for prepare-providers-documentation
-#
-# Outputs:
-# Flag information.
-#######################################################################################################
-function breeze::prepare_providers_documentation() {
- echo "
---non-interactive
-
- Runs the command in non-interactive mode.
-
---generate-providers-issue
-
- Generate providers issue that should be created.
-
-"
-}
-
-#######################################################################################################
-#
-# Prints flags that control verbosity
-#
-# Outputs:
-# Flag information.
-#######################################################################################################
-function breeze::flag_verbosity() {
- echo "
--v, --verbose
- Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
- debugging - when you run breeze with --verbose flags you will be able to see the commands
- executed under the hood and copy&paste them to your terminal to debug them more easily.
-
- Note that you can further increase verbosity and see all the commands executed by breeze
- by running 'export VERBOSE_COMMANDS=\"true\"' before running breeze.
-
---dry-run-docker
- Only show docker commands to execute instead of actually executing them. The docker
- commands are printed in yellow color.
-
-"
-}
-
-#######################################################################################################
-#
-# Prints information about help flag
-#
-# Outputs:
-# Flag information.
-#######################################################################################################
-function breeze::flag_help() {
- echo "
--h, --help
- Shows detailed help message for the command specified.
-"
-}
-
-#######################################################################################################
-#
-# Prints flags controlling docker build process
-#
-# Global constants used:
-# FORMATTED_DEFAULT_CI_EXTRAS
-# FORMATTED_DEFAULT_PROD_EXTRAS
-#
-# Outputs:
-# Flag information.
-#######################################################################################################
-function breeze::flag_build_docker_images() {
- echo "
--F, --force-build-images
- Forces building of the local docker images. The images are rebuilt
- automatically for the first time or when changes are detected in
- package-related files, but you can force it using this flag.
-
---cleanup-docker-context-files
- Removes whl and tar.gz files created in docker-context-files before running the command.
- In case there are some files there it unnecessarily increases the context size and
- makes the COPY . always invalidated - if you happen to have those files when you build your
- image.
-
-Customization options:
-
--E, --extras EXTRAS
- Extras to pass to build images The default are different for CI and production images:
-
- CI image:
-${FORMATTED_DEFAULT_CI_EXTRAS}
-
- Production image:
-${FORMATTED_DEFAULT_PROD_EXTRAS}
-
---image-tag TAG
- Additional tag in the image.
-
---disable-pypi-when-building
- Disable installing Airflow from pypi when building. If you use this flag and want
- to install Airflow, you have to install it from packages placed in
- 'docker-context-files' and use --install-from-docker-context-files flag.
-
---additional-extras ADDITIONAL_EXTRAS
- Additional extras to pass to build images The default is no additional extras.
-
---additional-python-deps ADDITIONAL_PYTHON_DEPS
- Additional python dependencies to use when building the images.
-
---dev-apt-command DEV_APT_COMMAND
- The basic command executed before dev apt deps are installed.
-
---additional-dev-apt-command ADDITIONAL_DEV_APT_COMMAND
- Additional command executed before dev apt deps are installed.
-
---additional-dev-apt-deps ADDITIONAL_DEV_APT_DEPS
- Additional apt dev dependencies to use when building the images.
-
---dev-apt-deps DEV_APT_DEPS
- The basic apt dev dependencies to use when building the images.
-
---additional-dev-apt-deps ADDITIONAL_DEV_DEPS
- Additional apt dev dependencies to use when building the images.
-
---additional-dev-apt-envs ADDITIONAL_DEV_APT_ENVS
- Additional environment variables set when adding dev dependencies.
-
---runtime-apt-command RUNTIME_APT_COMMAND
- The basic command executed before runtime apt deps are installed.
-
---additional-runtime-apt-command ADDITIONAL_RUNTIME_APT_COMMAND
- Additional command executed before runtime apt deps are installed.
-
---runtime-apt-deps ADDITIONAL_RUNTIME_APT_DEPS
- The basic apt runtime dependencies to use when building the images.
-
---additional-runtime-apt-deps ADDITIONAL_RUNTIME_DEPS
- Additional apt runtime dependencies to use when building the images.
-
---additional-runtime-apt-envs ADDITIONAL_RUNTIME_APT_DEPS
- Additional environment variables set when adding runtime dependencies.
-
-Build options:
-
---disable-mysql-client-installation
- Disables installation of the mysql client which might be problematic if you are building
- image in controlled environment. Only valid for production image.
-
---disable-mssql-client-installation
- Disables installation of the mssql client which might be problematic if you are building
- image in controlled environment. Only valid for production image.
-
---constraints-location
- Url to the constraints file. In case of the production image it can also be a path to the
- constraint file placed in 'docker-context-files' folder, in which case it has to be
- in the form of '/docker-context-files/<NAME_OF_THE_FILE>'
-
---disable-pip-cache
- Disables GitHub PIP cache during the build. Useful if GitHub is not reachable during build.
-
---install-from-docker-context-files
- This flag is used during image building. If it is used additionally to installing
- Airflow from PyPI, the packages are installed from the .whl and .tar.gz packages placed
- in the 'docker-context-files' folder. The same flag can be used during entering the image in
- the CI image - in this case also the .whl and .tar.gz files will be installed automatically
-
--C, --force-clean-images
- Force build images with cache disabled. This will remove the pulled or build images
- and start building images from scratch. This might take a long time.
-
--r, --skip-rebuild-check
- Skips checking image for rebuilds. It will use whatever image is available locally/pulled.
-
--L, --build-cache-local
- Uses local cache to build images. No pulled images will be used, but results of local
- builds in the Docker cache are used instead. This will take longer than when the pulled
- cache is used for the first time, but subsequent '--build-cache-local' builds will be
- faster as they will use mostly the locally build cache.
-
- This is default strategy used by the Production image builds.
-
--U, --build-cache-pulled
- Uses images pulled from GitHub Container Registry to build images.
- Those builds are usually faster than when ''--build-cache-local'' with the exception if
- the registry images are not yet updated. The images are updated after successful merges
- to main.
-
- This is default strategy used by the CI image builds.
-
--X, --build-cache-disabled
- Disables cache during docker builds. This is useful if you want to make sure you want to
- rebuild everything from scratch.
-
- This strategy is used by default for both Production and CI images for the scheduled
- (nightly) builds in CI.
-
-"
-}
-
-#######################################################################################################
-#
-# Prints flags controlling docker pull and push process
-#
-# Global constants used:
-# _breeze_default_github_repository
-# _breeze_default_github_image_id
-#
-# Outputs:
-# Flag information.
-#######################################################################################################
-function breeze::flag_pull_push_docker_images() {
- local show_sha="${1-'show_sha'}"
- echo "
--g, --github-repository GITHUB_REPOSITORY
- GitHub repository used to pull, push images.
- Default: ${_breeze_default_github_repository:=}.
-
-"
-
- [[ "$show_sha" != "no_show_sha" ]] && echo "
-
--s, --github-image-id COMMIT_SHA
- <COMMIT_SHA> of the image. Images in GitHub registry are stored with those
- to be able to easily find the image for particular CI runs. Once you know the
- <COMMIT_SHA>, you can specify it in github-image-id flag and Breeze will
- automatically pull and use that image so that you can easily reproduce a problem
- that occurred in CI.
-
- Default: ${_breeze_default_github_image_id:=}.
-
-"
-}
-
-#######################################################################################################
-#
-# Prints flags that control version of generated packages
-#
-# Outputs:
-# Flag information.
-#######################################################################################################
-function breeze::flag_version_suffix() {
- echo "
--S, --version-suffix-for-pypi SUFFIX
- Adds optional suffix to the version in the generated provider package. It can be used
- to generate rc1/rc2 ... versions of the packages to be uploaded to PyPI.
-
--N, --version-suffix-for-svn SUFFIX
- Adds optional suffix to the generated names of package. It can be used to generate
- rc1/rc2 ... versions of the packages to be uploaded to SVN.
-"
-}
-#####################################################################################################
-#
-# Prints flags that control how Airflow should be populated with the command start-airflow
-#
-# Outputs:
-# Flag information.
-#######################################################################################################
-function breeze::flag_start_airflow() {
- echo "
---load-example-dags
- Include Airflow example dags.
-
---load-default-connections
- Include Airflow Default Connections.
-"
-}
-
-#####################################################################################################
-#
-# Prints flags that control how Airflow should be populated with the command stop-airflow
-#
-# Outputs:
-# Flag information.
-#######################################################################################################
-function breeze::flag_stop_airflow() {
- echo "
---preserve-volumes
- Use this flag if you would like to preserve data volumes from the databases used
- by the integrations. By default, those volumes are deleted, so when you run 'stop'
- or 'restart' commands you start from scratch, but by using this flag you can
- preserve them. If you want to delete those volumes after stopping Breeze, just
- run the 'breeze stop' again without this flag.
-"
-}
-
-
-#####################################################################################################
-#
-# Prints flags that control tests
-#
-# Outputs:
-# Flag information.
-#######################################################################################################
-function breeze::flag_tests() {
- echo "
---test-type TEST_TYPE
- Type of the test to run. One of:
-
-${FORMATTED_TEST_TYPES}
-
- Default: ${_breeze_default_test_type:=}
-
-"
-}
-
-#######################################################################################################
-#
-# Prints all flags
-#
-# Outputs:
-# Flag information.
-#######################################################################################################
-function breeze::flags() {
- breeze::prepare_formatted_versions
-
- echo "
-$(breeze::print_line)
-
-Summary of all flags supported by Breeze:
-
-$(breeze::print_star_line)
- Choose Airflow variant
-$(breeze::flag_airflow_variants)
-
-$(breeze::print_star_line)
- Choose backend to run for Airflow
-$(breeze::flag_backend_variants)
-
-$(breeze::print_star_line)
- Enable production image
-$(breeze::flag_production_image)
-
-$(breeze::print_star_line)
- Additional actions executed while entering breeze
-$(breeze::flag_breeze_actions)
-
-$(breeze::print_star_line)
- Additional actions executed while starting Airflow
-$(breeze::flag_start_airflow)
-
-$(breeze::print_star_line)
- Cleanup options when stopping Airflow
-$(breeze::flag_stop_airflow)
-
-$(breeze::print_star_line)
- Kind kubernetes and Kubernetes tests configuration(optional)
-$(breeze::flag_kubernetes_configuration)
-
-$(breeze::print_star_line)
- Manage mounting local files
-$(breeze::flag_local_file_mounting)
-
-$(breeze::print_star_line)
- Assume answers to questions
-$(breeze::flag_assume_answers_to_questions)
-
-$(breeze::print_star_line)
- Install different Airflow version during PROD image build
-$(breeze::flag_build_different_airflow_version)
-
-$(breeze::print_star_line)
- Use different Airflow version at runtime in CI image
-$(breeze::flag_use_different_airflow_version)
-
-$(breeze::print_star_line)
- Credentials
-$(breeze::flag_credentials)
-
-$(breeze::print_star_line)
- Flags for building Docker images (both CI and production)
-$(breeze::flag_build_docker_images)
-
-$(breeze::print_star_line)
- Flags for pulling/pushing Docker images (both CI and production)
-$(breeze::flag_pull_push_docker_images "show_sha")
-
-$(breeze::print_star_line)
- Flags for running tests
-$(breeze::flag_tests)
-
-$(breeze::print_star_line)
- Flags for generation of the provider packages
-$(breeze::flag_version_suffix)
-
-$(breeze::print_star_line)
- Increase verbosity of the scripts
-$(breeze::flag_verbosity)
-
-$(breeze::print_star_line)
- Print detailed help message
-$(breeze::flag_help)
-"
-}
-
-#######################################################################################################
-#
-# Prints header line filling screen indented_screen_width - only when VERBOSE is set
-#
-# Outputs:
-# Prints header line.
-#######################################################################################################
-function breeze::print_header_line() {
- if [ ${VERBOSE:="false"} == "true" ]; then
- echo
- printf '=%.0s' $(seq "${SCREEN_WIDTH}")
- echo
- fi
-}
-
-#######################################################################################################
-#
-# Prints separation line filling screen indented_screen_width
-#
-# Outputs:
-# Prints line.
-#######################################################################################################
-function breeze::print_line() {
- printf '#%.0s' $(seq "${SCREEN_WIDTH}")
-}
-
-# Prints star line filling screen indented_screen_width
-function breeze::print_star_line() {
- printf '*%.0s' $(seq "${SCREEN_WIDTH}")
-}
-
-#######################################################################################################
-#
-# Reads saved environment variables. Some of the variables are stored across session so that once
-# you use them you do not have to use it next time. That makes those flags persistent.
-#
-# An example of it is `--backend` or '--kubernetes-mode' flags.
-#
-# Note that PYTHON_MAJOR_MINOR_VERSION is not read here - it is read at the
-# `setup_default_breeze_variables` method because it is needed
-# to determine the right images to use and set several variables that depend on the Python version
-#
-# Uses:
-# _breeze_default_* corresponding defaults for variables it reads
-#
-# Modified and used global constants:
-#
-# BACKEND
-# KUBERNETES_MODE
-# KUBERNETES_VERSION
-# KIND_VERSION
-# HELM_VERSION
-# EXECUTOR
-# POSTGRES_VERSION
-# MYSQL_VERSION
-#
-#######################################################################################################
-
-function breeze::read_saved_environment_variables() {
- BACKEND="${BACKEND:=$(parameters::read_from_file BACKEND)}"
- BACKEND=${BACKEND:-${_breeze_default_backend}}
-
- KUBERNETES_MODE="${KUBERNETES_MODE:=$(parameters::read_from_file KUBERNETES_MODE)}"
- KUBERNETES_MODE=${KUBERNETES_MODE:=${_breeze_default_kubernetes_mode}}
-
- KUBERNETES_VERSION="${KUBERNETES_VERSION:=$(parameters::read_from_file KUBERNETES_VERSION)}"
- KUBERNETES_VERSION=${KUBERNETES_VERSION:=${_breeze_default_kubernetes_version}}
-
- KIND_VERSION="${KIND_VERSION:=$(parameters::read_from_file KIND_VERSION)}"
- KIND_VERSION=${KIND_VERSION:=${_breeze_default_kind_version}}
-
- HELM_VERSION="${HELM_VERSION:=$(parameters::read_from_file HELM_VERSION)}"
- HELM_VERSION=${HELM_VERSION:=${_breeze_default_helm_version}}
-
- EXECUTOR="${EXECUTOR:=$(parameters::read_from_file EXECUTOR)}"
- EXECUTOR=${EXECUTOR:-${_breeze_default_executor}}
-
- POSTGRES_VERSION="${POSTGRES_VERSION:=$(parameters::read_from_file POSTGRES_VERSION)}"
- POSTGRES_VERSION=${POSTGRES_VERSION:=${_breeze_default_postgres_version}}
-
- MYSQL_VERSION="${MYSQL_VERSION:=$(parameters::read_from_file MYSQL_VERSION)}"
- MYSQL_VERSION=${MYSQL_VERSION:=${_breeze_default_mysql_version}}
-
- MSSQL_VERSION="${MSSQL_VERSION:=$(parameters::read_from_file MSSQL_VERSION)}"
- MSSQL_VERSION=${MSSQL_VERSION:=${_breeze_default_mssql_version}}
-}
-
-#######################################################################################################
-#
-# Checks if variables are correctly set and if they are - saves them so that they can be used across
-# sessions.
-#
-# In case the variables are matching expected values they are saved in ".build/VARIABLE_NAME" for
-# later reuse. If not, error is printed and the saved file is cleaned, so that next time
-# default value can be used.
-#
-# Used Global constants:
-# INSTALL_AIRFLOW_REFERENCE
-# INSTALL_AIRFLOW_VERSION
-# BACKEND
-# KUBERNETES_MODE
-# KUBERNETES_VERSION
-# KIND_VERSION
-# HELM_VERSION
-# EXECUTOR
-# POSTGRES_VERSION
-# MYSQL_VERSION
-# MSSQL_VERSION
-#
-# Updated Global constants:
-# BRANCH_NAME
-#
-# Output: saved variable files in .build,
-#######################################################################################################
-function breeze::check_and_save_all_params() {
- parameters::check_and_save_allowed_param "PYTHON_MAJOR_MINOR_VERSION" "Python version" "--python"
- # TODO: Generalize me (make it automatically use matching versions)
- if [[ -n "${INSTALL_AIRFLOW_REFERENCE=}" ]]; then
- if [[ ${INSTALL_AIRFLOW_REFERENCE=} == *2_0* ]]; then
- export BRANCH_NAME="v2-0-test"
- elif [[ ${INSTALL_AIRFLOW_REFERENCE=} == *2_1* ]]; then
- export BRANCH_NAME="v2-1-test"
- elif [[ ${INSTALL_AIRFLOW_REFERENCE=} == *2_2* ]]; then
- export BRANCH_NAME="v2-2-test"
- fi
- elif [[ -n "${INSTALL_AIRFLOW_VERSION=}" ]]; then
- if [[ ${INSTALL_AIRFLOW_VERSION=} == *2.0* ]]; then
- export BRANCH_NAME="v2-0-test"
- elif [[ ${INSTALL_AIRFLOW_VERSION=} == *2.1* ]]; then
- export BRANCH_NAME="v2-1-test"
- elif [[ ${INSTALL_AIRFLOW_VERSION=} == *2.2* ]]; then
- export BRANCH_NAME="v2-2-test"
- fi
- fi
-
- parameters::check_and_save_allowed_param "BACKEND" "backend" "--backend"
- parameters::check_and_save_allowed_param "PLATFORM" "platform" "--platform"
- parameters::check_and_save_allowed_param "KUBERNETES_MODE" "Kubernetes mode" "--kubernetes-mode"
- parameters::check_and_save_allowed_param "KUBERNETES_VERSION" "Kubernetes version" "--kubernetes-version"
- parameters::check_and_save_allowed_param "KIND_VERSION" "KinD version" "--kind-version"
- parameters::check_and_save_allowed_param "HELM_VERSION" "Helm version" "--helm-version"
- parameters::check_and_save_allowed_param "EXECUTOR" "Executors" "--executor"
- parameters::check_and_save_allowed_param "POSTGRES_VERSION" "Postgres version" "--postgres-version"
- parameters::check_and_save_allowed_param "MYSQL_VERSION" "Mysql version" "--mysql-version"
- parameters::check_and_save_allowed_param "MSSQL_VERSION" "MSSql version" "--mssql-version"
-
- parameters::check_allowed_param TEST_TYPE "Type of tests" "--test-type"
- parameters::check_allowed_param PACKAGE_FORMAT "Format of packages to build" "--package-format"
-}
-
-#######################################################################################################
-#
-# Prints cheatsheet if it is not suppressed
-#
-# Used global constants:
-#
-# AIRFLOW_SOURCES
-# SSH_PORT
-# WEBSERVER_HOST_PORT
-# POSTGRES_HOST_PORT
-# MYSQL_HOST_PORT
-# MSSQL_HOST_PORT
-#
-#######################################################################################################
-function breeze::print_cheatsheet() {
- if [[ ! -f ${SUPPRESS_CHEATSHEET_FILE} && ${command_to_run} == "enter_breeze" ]]; then
-
- echo
- breeze::print_line
- echo
- echo " Airflow Breeze CHEATSHEET"
- echo
- set +e
- if ! command -v breeze; then
- breeze::print_line
- echo
- echo " Adding breeze to your path:"
- echo " When you exit the environment, you can add sources of Airflow to the path - you can"
- echo " run breeze or the scripts above from any directory by calling 'breeze' commands directly"
- echo
- echo " export PATH=\${PATH}:\"${AIRFLOW_SOURCES}\""
- echo
- fi
- set -e
- breeze::print_line
-
- echo
- echo " Port forwarding:"
- echo
- echo " Ports are forwarded to the running docker containers for webserver and database"
- echo " * ${SSH_PORT} -> forwarded to Airflow ssh server -> airflow:22"
- echo " * ${WEBSERVER_HOST_PORT} -> forwarded to Airflow webserver -> airflow:8080"
- echo " * ${FLOWER_HOST_PORT} -> forwarded to Flower dashboard -> airflow:5555"
- echo " * ${POSTGRES_HOST_PORT} -> forwarded to Postgres database -> postgres:5432"
- echo " * ${MYSQL_HOST_PORT} -> forwarded to MySQL database -> mysql:3306"
- echo " * ${MSSQL_HOST_PORT} -> forwarded to MSSQL database -> mssql:1443"
- echo " * ${REDIS_HOST_PORT} -> forwarded to Redis broker -> redis:6379"
- echo
- echo " Here are links to those services that you can use on host:"
- echo " * ssh connection for remote debugging: ssh -p ${SSH_PORT} airflow@127.0.0.1 pw: airflow"
- echo " * Webserver: http://127.0.0.1:${WEBSERVER_HOST_PORT}"
- echo " * Flower: http://127.0.0.1:${FLOWER_HOST_PORT}"
- echo " * Postgres: jdbc:postgresql://127.0.0.1:${POSTGRES_HOST_PORT}/airflow?user=postgres&password=airflow"
- echo " * Mysql: jdbc:mysql://127.0.0.1:${MYSQL_HOST_PORT}/airflow?user=root"
- echo " * Redis: redis://127.0.0.1:${REDIS_HOST_PORT}/0"
- else
- echo
- fi
-}
-
-#######################################################################################################
-#
-# Prints setup instruction in case we find that autocomplete is not set
-# also prints how to toggle asciiart/cheatsheet
-#
-# Used global constants:
-# CMDNAME
-#######################################################################################################
-function breeze::print_setup_instructions() {
- if [[ ${command_to_run} == "enter_breeze" ]]; then
- # shellcheck disable=SC2034 # Unused variables left for comp_breeze usage
- if ! typeset -f "_comp_breeze" >/dev/null; then
- breeze::print_line
- echo
- echo " You can setup autocomplete by running '${CMDNAME} setup-autocomplete'"
- echo
- echo
- fi
- breeze::print_line
- echo
- echo " You can toggle ascii/cheatsheet by running:"
- echo " * ${CMDNAME} toggle-suppress-cheatsheet"
- echo " * ${CMDNAME} toggle-suppress-asciiart"
- echo
- breeze::print_line
- echo
- echo
- echo
- echo
- fi
-}
-
-#######################################################################################################
-#
-# Checks that pre-commit is installed and upgrades it if needed
-# this is used in case static check command is used.
-#
-#######################################################################################################
-function breeze::make_sure_precommit_is_installed() {
- echo
- echo "Making sure pre-commit is installed"
- echo
- local pip_binary
- if command -v pip3 >/dev/null; then
- pip_binary=pip3
- elif command -v pip >/dev/null; then
- pip_binary=pip
- else
- echo
- echo "${COLOR_RED}ERROR: You need to have pip or pip3 in your PATH ${COLOR_RESET}"
- echo
- exit 1
- fi
- "${pip_binary}" install --upgrade pre-commit >/dev/null 2>&1
- # Add ~/.local/bin to the path in case pip is run outside of virtualenv
- export PATH="${PATH}":~/.local/bin
-}
-
-#######################################################################################################
-#
-# Removes CI and PROD images and cleans up the flag that indicates that the image was already built
-#
-# Used global constants:
-# PYTHON_BASE_IMAGE
-# AIRFLOW_CI_IMAGE
-# AIRFLOW_PROD_IMAGE
-# BUILT_CI_IMAGE_FLAG_FILE
-#
-#######################################################################################################
-function breeze::remove_images() {
- # shellcheck disable=SC2086
- docker rmi --force ${PYTHON_BASE_IMAGE} \
- ${AIRFLOW_PYTHON_BASE_IMAGE} \
- ${AIRFLOW_CI_LOCAL_MANIFEST_IMAGE} \
- ${AIRFLOW_CI_REMOTE_MANIFEST_IMAGE} \
- ${AIRFLOW_CI_IMAGE} \
- ${AIRFLOW_PROD_IMAGE} \
- ${AIRFLOW_PROD_BUILD_IMAGE} \
- 2>/dev/null >/dev/null || true
- docker system prune || true
- echo
- echo "#########################################################################################"
- echo "NOTE!! Removed Airflow images for Python version ${PYTHON_MAJOR_MINOR_VERSION} and pruned cache."
- echo "#########################################################################################"
- echo
- rm -f "${BUILT_CI_IMAGE_FLAG_FILE}"
-}
-
-#######################################################################################################
-#
-# Runs chosen static checks
-#
-# Uses variables:
-# static_check
-#
-#######################################################################################################
-function breeze::run_static_checks() {
- if [[ ${static_check} == "all" ]]; then
- echo
- echo "Running: pre-commit run" "${EXTRA_STATIC_CHECK_OPTIONS[@]}" "$@"
- echo
- pre-commit run "${EXTRA_STATIC_CHECK_OPTIONS[@]}" "$@"
- else
- if [[ ${static_check} == *"mypy"* || ${static_check} == *"flake8"* ]]; then
- echo
- echo "Running build pre-commit before running the requested static check"
- echo
- pre-commit run "build"
- fi
- echo
- echo "Running: pre-commit run" "${static_check}" "${EXTRA_STATIC_CHECK_OPTIONS[@]}" "$@"
- echo
- pre-commit run "${static_check}" "${EXTRA_STATIC_CHECK_OPTIONS[@]}" "$@"
- fi
-}
-
-#######################################################################################################
-#
-# Runs Build before a command if needed - the right build command will be determined and used
-# depending on which command you are trying ton
-#
-# Used global constants:
-# PRODUCTION_IMAGE
-# KIND_CLUSTER_OPERATION
-# FORMATTED_KIND_OPERATIONS
-#
-# Used variables:
-# command_to_run
-#
-#######################################################################################################
-function breeze::run_build_command() {
- case "${command_to_run}" in
- run_tests | run_docker_compose)
- build_images::prepare_ci_build
- build_images::rebuild_ci_image_if_needed
- ;;
- enter_breeze)
- if [[ ${PRODUCTION_IMAGE} == "true" ]]; then
- build_images::prepare_prod_build
- else
- build_images::prepare_ci_build
- build_images::rebuild_ci_image_if_needed
- fi
- ;;
- build_docs | perform_static_checks | perform_generate_constraints | \
- perform_prepare_provider_documentation | perform_prepare_provider_packages | \
- perform_prepare_airflow_packages)
- build_images::prepare_ci_build
- build_images::rebuild_ci_image_if_needed
- ;;
- build_image)
- if [[ ${CLEANUP_DOCKER_CONTEXT_FILES} == "true" ]]; then
- build_images::cleanup_docker_context_files
- fi
- build_images::check_for_docker_context_files
- if [[ ${PRODUCTION_IMAGE} == "true" ]]; then
- build_images::prepare_prod_build
- build_images::build_prod_images
- else
-
- build_images::prepare_ci_build
- build_images::rebuild_ci_image_if_needed
- fi
- ;;
- prepare_build_cache)
- if [[ ${CLEANUP_DOCKER_CONTEXT_FILES} == "true" ]]; then
- build_images::cleanup_docker_context_files
- fi
- build_images::check_for_docker_context_files
-
- if [[ ${PRODUCTION_IMAGE} == "true" ]]; then
- build_images::prepare_prod_build
- build_images::build_prod_images
- else
- build_images::prepare_ci_build
- build_images::rebuild_ci_image_if_needed
- fi
- ;;
- cleanup_image | run_exec)
- if [[ ${PRODUCTION_IMAGE} == "true" ]]; then
- build_images::prepare_prod_build
- else
- build_images::prepare_ci_build
- fi
- ;;
- perform_initialize_local_virtualenv | perform_setup_autocomplete | \
- toggle_suppress_cheatsheet | toggle_suppress_asciiart )
- ;;
- manage_kind_cluster)
- if [[ ${KIND_CLUSTER_OPERATION} == "start" ]]; then
- echo "Starts KinD cluster"
- elif [[ ${KIND_CLUSTER_OPERATION} == "stop" ]]; then
- echo "Stops KinD cluster"
- elif [[ ${KIND_CLUSTER_OPERATION} == "restart" ]]; then
- echo "Restarts KinD cluster"
- elif [[ ${KIND_CLUSTER_OPERATION} == "recreate" ]]; then
- echo "Recreates KinD cluster"
- elif [[ ${KIND_CLUSTER_OPERATION} == "status" ]]; then
- echo "Checks status of KinD cluster"
- elif [[ ${KIND_CLUSTER_OPERATION} == "deploy" ]]; then
- echo "Deploys Airflow to KinD cluster"
- build_images::prepare_prod_build
- build_images::build_prod_images
- elif [[ ${KIND_CLUSTER_OPERATION} == "test" ]]; then
- echo "Run Kubernetes tests with the KinD cluster "
- elif [[ ${KIND_CLUSTER_OPERATION} == "shell" ]]; then
- echo "Enter an interactive shell for kubernetes testing"
- elif [[ ${KIND_CLUSTER_OPERATION} == "k9s" ]]; then
- echo "Run k9s cli to debug in style"
- elif [[ -z ${KIND_CLUSTER_OPERATION=} ]]; then
- echo
- echo "Please provide an operation to run"
- echo
- echo "Should be one of:"
- echo "${FORMATTED_KIND_OPERATIONS}"
- echo
- exit 1
- else
- echo
- echo "ERROR: Unknown Kind Kubernetes cluster operation: '${KIND_CLUSTER_OPERATION}'"
- echo
- echo "Should be one of:"
- echo "${FORMATTED_KIND_OPERATIONS}"
- echo
- exit 1
- fi
- ;;
- *)
- echo
- echo "${COLOR_RED}ERROR: Unknown command to run ${command_to_run} ${COLOR_RESET}"
- echo
- exit 1
- ;;
- esac
-}
-
-# executes command
-function breeze::run_command() {
- "${@}"
-}
-
-
-# print command instead of executing
-function breeze::print_command() {
- echo
- echo "${COLOR_YELLOW}" "${@}" "${COLOR_RESET}"
- echo
-}
-
-
-#######################################################################################################
-#
-# Runs the actual command - depending on the command chosen it will use the right
-# Convenient script and run the right command for the script.
-#
-# Used variables:
-# command_to_run
-#
-# Used global constants:
-# PRODUCTION_IMAGE
-# SCRIPTS_CI_DIR
-# BUILD_CACHE_DIR
-# KIND_CLUSTER_OPERATION
-# EXTRA_DC_OPTIONS
-#
-# Set Global variables:
-# RUN_TESTS
-#######################################################################################################
-function breeze::run_breeze_command() {
- set +u
- local dc_run_file
- local run_command="breeze::run_command"
- if [[ ${DRY_RUN_DOCKER=} != "false" ]]; then
- run_command="breeze::print_command"
- fi
- if [[ ${PRODUCTION_IMAGE} == "true" ]]; then
- dc_run_file="${BUILD_CACHE_DIR}/${DOCKER_COMPOSE_RUN_SCRIPT_FOR_PROD}"
- else
- dc_run_file="${BUILD_CACHE_DIR}/${DOCKER_COMPOSE_RUN_SCRIPT_FOR_CI}"
- fi
-
- case "${command_to_run}" in
- enter_breeze)
- docker_engine_resources::check_all_resources
- if [[ $(uname -m) == "arm64" || $(uname -m) == "aarch64" ]]; then
- if [[ ${BACKEND} == "mysql" || ${BACKEND} == "mssql" ]]; then
- echo "${COLOR_RED}MacOS with ARM processor is not supported for ${BACKEND} backend. Exiting.${COLOR_RESET}"
- exit 1
- fi
- fi
- if [[ ${PRODUCTION_IMAGE} == "true" ]]; then
- echo "${COLOR_RED}ERROR: Entering production image via breeze is not supported${COLOR_RESET}"
- echo
- echo "Please run ${COLOR_BLUE}docker run -it apache/airflow:latest bash${COLOR_RESET} instead"
- echo
- echo "See https://airflow.apache.org/docs/docker-stack/entrypoint.html#executing-commands for details"
- exit 1
- else
- ${run_command} "${dc_run_file}" run --service-ports --rm airflow "${@}"
- fi
- ;;
- run_exec)
- docker_engine_resources::check_all_resources
- # Unfortunately `docker-compose exec` does not support exec'ing into containers started with run :(
- # so we have to find it manually
- set +e
- local airflow_testing_container
- airflow_testing_container=$("${dc_run_file}" ps | grep airflow | awk '{print $1}' 2>/dev/null)
- : "${airflow_testing_container:?"ERROR! Breeze must be running in order to exec into running container"}"
- set -e
- docker exec -it "${airflow_testing_container}" \
- "/opt/airflow/scripts/docker/entrypoint_exec.sh" "${@}"
- ;;
- run_tests)
- docker_engine_resources::check_all_resources
- export RUN_TESTS="true"
- readonly RUN_TESTS
- export ENABLED_INTEGRATIONS="${INTEGRATIONS[*]}"
- export LIST_OF_INTEGRATION_TESTS_TO_RUN="${INTEGRATIONS[*]}"
- ${run_command} "${BUILD_CACHE_DIR}/${DOCKER_COMPOSE_RUN_SCRIPT_FOR_CI}" run --service-ports --rm airflow "$@"
- ;;
- run_docker_compose)
- docker_engine_resources::check_all_resources
- set +u
- ${run_command} "${dc_run_file}" "${docker_compose_command}" "${EXTRA_DC_OPTIONS[@]}" "$@"
- set -u
- ;;
- perform_static_checks)
- docker_engine_resources::check_all_resources
- breeze::make_sure_precommit_is_installed
- breeze::run_static_checks "${@}"
- ;;
- build_image) ;;
- prepare_build_cache) ;;
- cleanup_image)
- breeze::remove_images
- ;;
- perform_generate_constraints)
- docker_engine_resources::check_all_resources
- runs::run_generate_constraints
- ;;
- perform_prepare_airflow_packages)
- docker_engine_resources::check_all_resources
- runs::run_prepare_airflow_packages
- ;;
- perform_prepare_provider_packages)
- docker_engine_resources::check_all_resources
- runs::run_prepare_provider_packages "${@}"
- ;;
- perform_prepare_provider_documentation)
- docker_engine_resources::check_all_resources
- runs::run_prepare_provider_documentation "${@}"
- ;;
- perform_initialize_local_virtualenv)
- breeze::initialize_virtualenv
- ;;
- perform_setup_autocomplete)
- breeze::setup_autocomplete
- ;;
- manage_kind_cluster)
- docker_engine_resources::check_all_resources
- kind::make_sure_kubernetes_tools_are_installed
- kind::get_kind_cluster_name
- kind::perform_kind_cluster_operation "${KIND_CLUSTER_OPERATION}"
- ;;
- build_docs)
- docker_engine_resources::check_all_resources
- runs::run_docs "${@}"
- ;;
- toggle_suppress_cheatsheet)
- if [[ -f "${SUPPRESS_CHEATSHEET_FILE}" ]]; then
- echo
- echo "Cheatsheet disabled"
- echo
- else
- echo
- echo "Cheatsheet enabled"
- echo
- fi
- ;;
- toggle_suppress_asciiart)
- if [[ -f "${SUPPRESS_ASCIIART_FILE}" ]]; then
- echo
- echo "ASCIIart disabled"
- echo
- else
- echo
- echo "ASCIIart enabled"
- echo
- fi
- ;;
- *)
- echo
- echo "${COLOR_RED}ERROR: Unknown command to run ${command_to_run} ${COLOR_RESET}"
- echo
- ;;
- esac
- set -u
-}
-
-#######################################################################################################
-#
-# We have different versions of images depending on the python version used. We keep up with the
-# Latest patch-level changes in Python (this is done automatically during CI test runs) so we have
-# To only take into account MAJOR and MINOR version of python. This variable keeps the major/minor
-# version of python in X.Y format (3.7, 3.8, 3.9, 3.10).
-#
-# In Breeze the precedence of setting the version is as follows:
-# 1. --python flag (if set, it will explicitly override it in the next step)
-# 2. PYTHON_MAJOR_MINOR_VERSION exported from outside
-# 3. last used version stored in ./build/PYTHON_MAJOR_MINOR_VERSION
-# 4. DEFAULT_PYTHON_MAJOR_MINOR_VERSION from scripts/ci/libraries/_initialization.sh
-#
-# Here points 2. and 3. are realized. If result is empty string , the 4. will be set in
-# the next step (sanity_checks::basic_sanity_checks() is called and the version is still not set by then)
-# finally, if --python flag is specified, it will override whatever is set above.
-#
-# We need to run after initialization::initialize_common_environment (so that parameters::read_from_file function is present)
-# But before we set the default value for Python
-#
-# Used and modified global constants:
-# PYTHON_MAJOR_MINOR_VERSION
-#######################################################################################################
-function breeze::determine_python_version_to_use_in_breeze() {
- PYTHON_MAJOR_MINOR_VERSION="${PYTHON_MAJOR_MINOR_VERSION:=$(parameters::read_from_file PYTHON_MAJOR_MINOR_VERSION)}"
- export PYTHON_MAJOR_MINOR_VERSION
-}
-
-breeze::setup_default_breeze_constants
-
-initialization::create_directories
-
-breeze::read_saved_environment_variables
-
-initialization::initialize_common_environment
-
-initialization::check_docker_version
-
-initialization::get_environment_for_builds_on_ci
-
-breeze::determine_python_version_to_use_in_breeze
-
-sanity_checks::basic_sanity_checks
-
-start_end::script_start
-
-traps::add_trap start_end::script_end EXIT
-
-set +u
-breeze::parse_arguments "${@}"
-
-breeze::print_header_line
-
-breeze::check_and_save_all_params
-
-build_images::determine_docker_cache_strategy
-
-build_images::get_docker_cache_image_names
-
-initialization::make_constants_read_only
-
-sanity_checks::sanitize_mounted_files
-
-breeze::prepare_command_files
-
-breeze::run_build_command
-
-breeze::print_header_line
-
-breeze::print_badge
-
-breeze::print_cheatsheet
-
-breeze::print_setup_instructions
-
-set +u # Account for an empty array
-breeze::run_breeze_command "${REMAINING_ARGS[@]}"
+check_breeze_installed
-set +u # Account for an empty array
-if [[ -n ${second_command_to_run} ]]; then
- command_to_run=${second_command_to_run}
- breeze::run_breeze_command "${REMAINING_ARGS[@]}"
-fi
+${BREEZE_BINARY} "${@}"
diff --git a/breeze-complete b/breeze-complete
index b8e032aa37..2de6f54753 100644
--- a/breeze-complete
+++ b/breeze-complete
@@ -154,8 +154,9 @@ sort-spelling-wordlist
stylelint
trailing-whitespace
ui-lint
-update-breeze-config-hash
+limit-breeze-dependencies
update-breeze-file
+update-breeze-config-hash
update-extras
update-local-yml-file
update-setup-cfg-file
@@ -482,5 +483,5 @@ function breeze_complete::_comp_breeze {
breeze_complete::_build_options_breeze
# set autocompletion function for breeze
-complete -F breeze_complete::_comp_breeze breeze
-complete -F breeze_complete::_comp_breeze ./breeze
+complete -F breeze_complete::_comp_breeze breeze-legacy
+complete -F breeze_complete::_comp_breeze ./breeze-legacy
diff --git a/breeze b/breeze-legacy
similarity index 100%
copy from breeze
copy to breeze-legacy
diff --git a/dev/PROVIDER_PACKAGE_DETAILS.md b/dev/PROVIDER_PACKAGE_DETAILS.md
index ac5743b0f2..8717e86836 100644
--- a/dev/PROVIDER_PACKAGE_DETAILS.md
+++ b/dev/PROVIDER_PACKAGE_DETAILS.md
@@ -62,7 +62,7 @@ Details to be hashed out in [the related issue](https://github.com/apache/airflo
When you want to prepare release notes for a package, you need to run:
```
-./breeze prepare-provider-documentation <PACKAGE_ID> ...
+./breeze-legacy prepare-provider-documentation <PACKAGE_ID> ...
```
The version for each package is going to be updated separately for each package when we agree to the
@@ -80,7 +80,7 @@ release date without providing
the date (to update the existing release notes)
```
-./breeze prepare-provider-documentation google
+./breeze-legacy prepare-provider-documentation google
```
@@ -119,7 +119,7 @@ the folders (for example Apache Hive's PACKAGE_ID is `apache.hive` ). You can se
providers by running:
```bash
-./breeze prepare-provider-packages -- --help
+./breeze-legacy prepare-provider-packages -- --help
```
The examples below show how you can build selected packages, but you can also build all packages by
@@ -128,38 +128,38 @@ omitting the package ids altogether.
* To build the release candidate packages for SVN Apache upload run the following command:
```bash
-./breeze prepare-provider-packages package-format both --version-suffix-for-svn=rc1 [PACKAGE_ID] ...
+./breeze-legacy prepare-provider-packages package-format both --version-suffix-for-svn=rc1 [PACKAGE_ID] ...
```
for example:
```bash
-./breeze prepare-provider-packages package-format both --version-suffix-for-svn=rc1 http ...
+./breeze-legacy prepare-provider-packages package-format both --version-suffix-for-svn=rc1 http ...
```
* To build the release candidate packages for PyPI upload run the following command:
```bash
-./breeze prepare-provider-packages package-format both --version-suffix-for-pypi=rc1 [PACKAGE_ID] ...
+./breeze-legacy prepare-provider-packages package-format both --version-suffix-for-pypi=rc1 [PACKAGE_ID] ...
```
for example:
```bash
-./breeze prepare-provider-packages package-format both --version-suffix-for-pypi=rc1 http ...
+./breeze-legacy prepare-provider-packages package-format both --version-suffix-for-pypi=rc1 http ...
```
* To build the final release packages run the following command:
```bash
-./breeze prepare-provider-packages package-format both [PACKAGE_ID] ...
+./breeze-legacy prepare-provider-packages package-format both [PACKAGE_ID] ...
```
for example:
```bash
-./breeze prepare-provider-packages package-format both http ...
+./breeze-legacy prepare-provider-packages package-format both http ...
```
* For each package, this creates a wheel package and source distribution package in your `dist` folder with
diff --git a/dev/README_RELEASE_AIRFLOW.md b/dev/README_RELEASE_AIRFLOW.md
index 55fe68229a..101866391d 100644
--- a/dev/README_RELEASE_AIRFLOW.md
+++ b/dev/README_RELEASE_AIRFLOW.md
@@ -239,7 +239,7 @@ For now this is done manually, example run `git log --oneline v2-2-test..HEAD -
- Generate SHA512/ASC (If you have not generated a key yet, generate it by following instructions on http://www.apache.org/dev/openpgp.html#key-gen-generate-key)
```shell script
- ./breeze prepare-airflow-packages --package-format both
+ ./breeze-legacy prepare-airflow-packages --package-format both
pushd dist
${AIRFLOW_REPO_ROOT}/dev/sign.sh *
popd
@@ -414,7 +414,7 @@ To do this we need to
- Build the package:
```shell script
- ./breeze prepare-airflow-packages --version-suffix-for-pypi "${VERSION_SUFFIX}" --package-format both
+ ./breeze-legacy prepare-airflow-packages --version-suffix-for-pypi "${VERSION_SUFFIX}" --package-format both
```
- Verify the artifacts that would be uploaded:
@@ -761,7 +761,7 @@ There is also an easy way of installation with Breeze if you have the latest sou
Running the following command will use tmux inside breeze, create `admin` user and run Webserver & Scheduler:
```shell script
-./breeze start-airflow --use-airflow-version <VERSION>rc<X> --python 3.7 --backend postgres
+breeze start-airflow --use-airflow-version <VERSION>rc<X> --python 3.7 --backend postgres
```
Once you install and run Airflow, you should perform any verification you see as necessary to check
@@ -947,7 +947,7 @@ Documentation for providers can be found in the ``/docs/apache-airflow`` directo
```shell script
cd "${AIRFLOW_REPO_ROOT}"
- ./breeze build-docs -- --package-filter apache-airflow --package-filter docker-stack --for-production
+ breeze build-docs --package-filter apache-airflow --package-filter docker-stack --for-production
```
- Now you can preview the documentation.
diff --git a/dev/README_RELEASE_HELM_CHART.md b/dev/README_RELEASE_HELM_CHART.md
index 8738c49dc5..e2e6eabc35 100644
--- a/dev/README_RELEASE_HELM_CHART.md
+++ b/dev/README_RELEASE_HELM_CHART.md
@@ -602,7 +602,7 @@ between the two repositories to be able to build the documentation.
```shell
cd "${AIRFLOW_REPO_ROOT}"
git checkout helm-chart/${VERSION}
- ./breeze build-docs -- --package-filter helm-chart --for-production
+ breeze build-docs --package-filter helm-chart --for-production
```
- Now you can preview the documentation.
diff --git a/dev/README_RELEASE_PROVIDER_PACKAGES.md b/dev/README_RELEASE_PROVIDER_PACKAGES.md
index ab6e596435..edb1647fe2 100644
--- a/dev/README_RELEASE_PROVIDER_PACKAGES.md
+++ b/dev/README_RELEASE_PROVIDER_PACKAGES.md
@@ -86,7 +86,7 @@ Details about maintaining the SEMVER version are going to be discussed and imple
```shell script
-./breeze prepare-provider-documentation [packages]
+./breeze-legacy prepare-provider-documentation [packages]
```
This command will not only prepare documentation but will also help the release manager to review
@@ -107,7 +107,7 @@ When you want to regenerate the changes before the release and make sure all cha
are updated, run it in non-interactive mode:
```shell script
-./breeze --non-interactive prepare-provider-documentation [packages]
+./breeze-legacy --non-interactive prepare-provider-documentation [packages]
```
## Build provider packages for SVN apache upload
@@ -132,13 +132,13 @@ rm -rf ${AIRFLOW_REPO_ROOT}/dist/*
* Release candidate packages:
```shell script
-./breeze prepare-provider-packages --package-format both
+./breeze-legacy prepare-provider-packages --package-format both
```
if you only build few packages, run:
```shell script
-./breeze prepare-provider-packages --package-format both PACKAGE PACKAGE ....
+./breeze-legacy prepare-provider-packages --package-format both PACKAGE PACKAGE ....
```
* Sign all your packages
@@ -197,13 +197,13 @@ this will clean up dist folder before generating the packages, so you will only
```shell script
rm -rf ${AIRFLOW_REPO_ROOT}/dist/*
-./breeze prepare-provider-packages --version-suffix-for-pypi rc1 --package-format both
+./breeze-legacy prepare-provider-packages --version-suffix-for-pypi rc1 --package-format both
```
if you only build few packages, run:
```shell script
-./breeze prepare-provider-packages --version-suffix-for-pypi rc1 --package-format both \
+./breeze-legacy prepare-provider-packages --version-suffix-for-pypi rc1 --package-format both \
PACKAGE PACKAGE ....
```
@@ -264,10 +264,8 @@ export AIRFLOW_SITE_DIRECTORY="$(pwd)"
```shell script
cd "${AIRFLOW_REPO_ROOT}"
-./breeze build-docs -- \
- --for-production \
- --package-filter apache-airflow-providers \
- --package-filter 'apache-airflow-providers-*'
+breeze build-docs --for-production --package-filter apache-airflow-providers \
+ --package-filter 'apache-airflow-providers-*'
```
Usually when we release packages we also build documentation for the "documentation-only" packages. This
@@ -278,8 +276,7 @@ If we want to just release some providers you can release them in this way:
```shell script
cd "${AIRFLOW_REPO_ROOT}"
-./breeze build-docs -- \
- --for-production \
+breeze build-docs --for-production \
--package-filter apache-airflow-providers \
--package-filter 'apache-airflow-providers-PACKAGE1' \
--package-filter 'apache-airflow-providers-PACKAGE2' \
@@ -595,7 +592,7 @@ pip install apache-airflow-providers-<provider>==<VERSION>rc<X>
### Installing with Breeze
```shell
-./breeze start-airflow --use-airflow-version 2.2.4 --python 3.7 --backend postgres \
+breeze start-airflow --use-airflow-version 2.2.4 --python 3.7 --backend postgres \
--load-example-dags --load-default-connections
```
diff --git a/dev/REFRESHING_CI_CACHE.md b/dev/REFRESHING_CI_CACHE.md
index 740af2fbb1..b57d30130c 100644
--- a/dev/REFRESHING_CI_CACHE.md
+++ b/dev/REFRESHING_CI_CACHE.md
@@ -51,7 +51,7 @@ manual refresh might be needed.
export CURRENT_PYTHON_MAJOR_MINOR_VERSIONS_AS_STRING="3.7 3.8 3.9 3.10"
for python_version in $(echo "${CURRENT_PYTHON_MAJOR_MINOR_VERSIONS_AS_STRING}")
do
- ./breeze build-image --upgrade-to-newer-dependencies --python ${python_version}
+ breeze build-image --upgrade-to-newer-dependencies --python ${python_version}
done
GENERATE_CONSTRAINTS_MODE="pypi-providers" ./scripts/ci/constraints/ci_generate_all_constraints.sh
diff --git a/dev/TRACKING_BACKTRACKING_ISSUES.md b/dev/TRACKING_BACKTRACKING_ISSUES.md
index a8d1fd3a2c..a4c6e14efc 100644
--- a/dev/TRACKING_BACKTRACKING_ISSUES.md
+++ b/dev/TRACKING_BACKTRACKING_ISSUES.md
@@ -130,8 +130,8 @@ the version that was correctly installed before and is stored in the current con
The process of tracking down which package is the "root cause" looks as follows:
1. Checkout the latest main of Airflow
-2. Build the latest image (with constraints): `./breeze build-image --python 3.7`
-3. Enter breeze `./breeze`
+2. Build the latest image (with constraints): `breeze build-image --python 3.7`
+3. Enter breeze `breeze`
4. Attempt to run the `pip install` command that was printed in the "Candidates ..." step
5. The command should succeed (the candidates are pinned to the "working" version)
6. Attempt to run `pip install ".[devel_all]" --upgrade --upgrade-strategy eager "dill<0.3.3" "certifi<2021.0.0" "google-ads<14.0.1"`
@@ -188,10 +188,9 @@ was able to find the right resolution without backtracking.
You can also find the candidates manually. This is especially when you are not sure when the build broke,
and you need to extend the time or when you need to run it for another branch.
-You need to install the new Breeze2 in one of two ways:
+You need to install the breeze:
* `pipx install -e ./dev/breeze` if you use pipx install.
-* `pip install -e ./dev/breeze` in your own dev virtualenv.
Then you can run airflow-find-newer-dependencies with optional flags. For example if you know that the build
was likely broken on a given date and time (in your timezone) and you want to check python 3.8
diff --git a/dev/breeze/README.md b/dev/breeze/README.md
index 797bded297..4f5da05561 100644
--- a/dev/breeze/README.md
+++ b/dev/breeze/README.md
@@ -52,6 +52,6 @@ PLEASE DO NOT MODIFY THE HASH BELOW! IT IS AUTOMATICALLY UPDATED BY PRE-COMMIT.
---------------------------------------------------------------------------------------------------------
-Package config hash: 4832d19ac2e6ffaf6184adb3741d58ae1934849c49a8c28305f5efa966f4bab0b450472767cb923cebcbb1a020e7f1eee46b660e11abf8c737d61a0f12796faa
+Package config hash: 352a3150dd5b8763de98e99e6dde1271cd35c98df506e091008f50f4c7765ea68eb23005e26af8bc61116bd90777a3c31afd12041890c6df7fd8b518500c08ae
---------------------------------------------------------------------------------------------------------
diff --git a/dev/breeze/autocomplete/Breeze2-complete-bash.sh b/dev/breeze/autocomplete/breeze-complete-bash.sh
similarity index 73%
rename from dev/breeze/autocomplete/Breeze2-complete-bash.sh
rename to dev/breeze/autocomplete/breeze-complete-bash.sh
index a9ba398985..4b265b6934 100644
--- a/dev/breeze/autocomplete/Breeze2-complete-bash.sh
+++ b/dev/breeze/autocomplete/breeze-complete-bash.sh
@@ -1,8 +1,8 @@
-_Breeze2_completion() {
+_breeze_completion() {
local IFS=$'\n'
local response
- response=$(env COMP_WORDS="${COMP_WORDS[*]}" COMP_CWORD=$COMP_CWORD _BREEZE2_COMPLETE=bash_complete $1)
+ response=$(env COMP_WORDS="${COMP_WORDS[*]}" COMP_CWORD=$COMP_CWORD _BREEZE_COMPLETE=bash_complete $1)
for completion in $response; do
IFS=',' read type value <<< "$completion"
@@ -21,8 +21,8 @@ _Breeze2_completion() {
return 0
}
-_Breeze2_completion_setup() {
- complete -o nosort -F _Breeze2_completion Breeze2
+_breeze_completion_setup() {
+ complete -o nosort -F _breeze_completion breeze
}
-_Breeze2_completion_setup;
+_breeze_completion_setup;
diff --git a/dev/breeze/autocomplete/Breeze2-complete-fish.sh b/dev/breeze/autocomplete/breeze-complete-fish.sh
similarity index 66%
rename from dev/breeze/autocomplete/Breeze2-complete-fish.sh
rename to dev/breeze/autocomplete/breeze-complete-fish.sh
index 7dea0bc38d..9cad9bb1a1 100644
--- a/dev/breeze/autocomplete/Breeze2-complete-fish.sh
+++ b/dev/breeze/autocomplete/breeze-complete-fish.sh
@@ -1,7 +1,7 @@
-function _Breeze2_completion;
+function _breeze_completion;
set -l response;
- for value in (env _BREEZE2_COMPLETE=fish_complete COMP_WORDS=(commandline -cp) COMP_CWORD=(commandline -t) Breeze2);
+ for value in (env _BREEZE_COMPLETE=fish_complete COMP_WORDS=(commandline -cp) COMP_CWORD=(commandline -t) breeze);
set response $response $value;
end;
@@ -18,4 +18,4 @@ function _Breeze2_completion;
end;
end;
-complete --no-files --command Breeze2 --arguments "(_Breeze2_completion)";
+complete --no-files --command breeze --arguments "(_breeze_completion)";
diff --git a/dev/breeze/autocomplete/Breeze2-complete-zsh.sh b/dev/breeze/autocomplete/breeze-complete-zsh.sh
similarity index 81%
rename from dev/breeze/autocomplete/Breeze2-complete-zsh.sh
rename to dev/breeze/autocomplete/breeze-complete-zsh.sh
index 11092e75f3..39bb77905d 100644
--- a/dev/breeze/autocomplete/Breeze2-complete-zsh.sh
+++ b/dev/breeze/autocomplete/breeze-complete-zsh.sh
@@ -1,12 +1,12 @@
-#compdef Breeze2
+#compdef breeze
-_Breeze2_completion() {
+_breeze_completion() {
local -a completions
local -a completions_with_descriptions
local -a response
- (( ! $+commands[Breeze2] )) && return 1
+ (( ! $+commands[breeze] )) && return 1
- response=("${(@f)$(env COMP_WORDS="${words[*]}" COMP_CWORD=$((CURRENT-1)) _BREEZE2_COMPLETE=zsh_complete Breeze2)}")
+ response=("${(@f)$(env COMP_WORDS="${words[*]}" COMP_CWORD=$((CURRENT-1)) _BREEZE_COMPLETE=zsh_complete breeze)}")
for type key descr in ${response}; do
if [[ "$type" == "plain" ]]; then
@@ -31,4 +31,4 @@ _Breeze2_completion() {
fi
}
-compdef _Breeze2_completion Breeze2;
+compdef _breeze_completion breeze;
diff --git a/dev/breeze/doc/BREEZE2.md b/dev/breeze/doc/BREEZE2.md
deleted file mode 100644
index 98e5c65564..0000000000
--- a/dev/breeze/doc/BREEZE2.md
+++ /dev/null
@@ -1,141 +0,0 @@
-<!--
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
- -->
-
-Here we are, introducing the post about BREEZE (Updated BREEZE version). It's going to be great!
-But first: Created a TOC for easy reference about BREEZE commands
-
-<!-- START doctoc generated TOC please keep comment here to allow auto update -->
-<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
-
-- [Updated BREEZE](#updated-breeze)
-- [BREEZE setting up autocomplete](#breeze-setting-up-autocomplete)
-- [Breeze on Linux](#breeze-on-linux)
-- [Breeze on Windows](#breeze-on-windows)
-
-<!-- END doctoc generated TOC please keep comment here to allow auto update -->
-
-<div align="center">
- <img src="../../../images/AirflowBreeze_logo.png"
- alt="Airflow Breeze - Development and Test Environment for Apache Airflow">
-</div>
-
-# Updated BREEZE
-
-Here we'll discuss about BREEZE...
-
-# BREEZE setting up autocomplete
-
-The full potential of Breeze can be put to use by enabling autocompletion. Breeze setup-autocomplete helps you to enable autocompletion for the list of commands that BREEZE supports with tab click. The command that helps to setup autocompletion is:
-
-`Breeze2 setup-autocomplete`
-
-After setting up the autocomplete command, you can use <TAB> to see list of available options and get autocompletion. To enable this feature, under the hood we use click-completion that helps setup the autocompletion for bash, zsh, fish and powershell. It auto detects the shell, generate the respective code for the shell type and links the shell type and generated activation command for autocompletion.
-
-Using this command, it will display confirmation message asking if the command has to add the autocompletion activation script to shell. If yes, then the command will take care of associating the autocompletion activation script to shell. If no, these steps have to done manually.
-
-**For Bash users**
-Activation script is generated in the path `AIRFLOW_SOURCE/.build/autocomplete/Breeze2-complete.bash`. In the ~/.bash_completion file, append the below command and save it
-
-```
-
-source AIRFLOW_SOURCE/.build/autocomplete/Breeze2-complete.bash
-
-```
-
-After editing ~/.bash_completion file, `source ~/.bash_completion`
-
-**For Zsh users**
-Activation script is generated in the path `AIRFLOW_SOURCE/.build/autocomplete/Breeze2-complete.zsh`. In the ~/.zshrc file, append the below command and save it
-
-```
-
-source AIRFLOW_SOURCE/.build/autocomplete/Breeze2-complete.zsh
-
-```
-
-After editing ~/.zshrc, `source ~/.zshrc`
-
-**For Fish users**
-Activation script is generated in the path `AIRFLOW_SOURCE/.build/autocomplete/Breeze2-complete.fish`. Copy the generated script to `~/.config/fish/completions/Breeze.fish`
-
-```
-
-cp AIRFLOW_SOURCE/.build/autocomplete/Breeze2-complete.fish ~/.config/fish/completions/Breeze.fish
-
-```
-
-**For Powershell users**
-
-# Breeze on Linux
-
-Installation is as easy as checking out Airflow repository and running Breeze command. You enter the Breeze test environment by running the ./breeze script. You can run it with the help command to see the list of available options. See Breeze Command-Line Interface Reference for details.
-
-```bash
-./breeze
-```
-
-The First time you run Breeze, it pulls and builds a local version of Docker images. It pulls the latest Airflow CI images from the GitHub Container Registry and uses them to build your local Docker images. Note that the first run (per python) might take up to 10 minutes on a fast connection to start. Subsequent runs should be much faster.
-
-Once you enter the environment, you are dropped into bash shell of the Airflow container and you can run tests immediately.
-
-To use the full potential of Breeze you should set up autocomplete and you can add the checked-out Airflow repository to your PATH to run Breeze without the ./ and from any directory.
-
-The Breeze command comes with a built-in bash/zsh autocomplete setup command. After installing, when you start typing the command, you can use <TAB> to show all the available switches and get auto-completion on typical values of parameters that you can use.
-
-You should set up the autocomplete option automatically by running:
-
-```bash
-./breeze setup-autocomplete
-```
-
-You get the auto-completion working when you re-enter the shell.
-
-# Breeze on Windows
-
-In Windows environment, you will need to use pipx to install Breeze.
-
-Install pipx
-
-```bash
-pip install --user pipx
-```
-
-Install Breeze, this command will generate Breeze2.exe
-
-```bash
-pipx install -e dev/breeze
-```
-
-Breeze, is not globally accessible until your PATH is updated. Add <USER FOLDER>\.local\bin as a variable environments. This can be done automatically by the following command.
-
-```bash
-pipx ensurepath
-```
-
-Finally, use Breeze, --help will show you Breeze options.
-
-```bash
-Breeze2 --help
-```
-
-To update the installation use: --force
-
-```bash
-pipx install --force -e dev/breeze
-```
diff --git a/dev/breeze/doc/adr/0002-implement-standalone-python-command.md b/dev/breeze/doc/adr/0002-implement-standalone-python-command.md
index dc2ca3a492..c667512afc 100644
--- a/dev/breeze/doc/adr/0002-implement-standalone-python-command.md
+++ b/dev/breeze/doc/adr/0002-implement-standalone-python-command.md
@@ -130,10 +130,6 @@ There are likely a number of scripts that will remain in Bash, but they should c
logic, they should not haave common code in form of libraries and only used to execute simple tasks inside
Docker containers. No Bash should ever be used in the host environment.
-The "working" name of the new Breeze is "Breeze2". We might come up with a better name in the future. In
-order to distinguish from the Bash version of Breeze we will always use capitalized form of Breeze as opposed
-to lower-case often used for the Bash version.
-
There are a few properties of Breeze/CI scripts that should be maintained though
* It should be possible to start Breeze and run any of the CI scripts without having a specially prepared
@@ -160,7 +156,7 @@ There are a few properties of Breeze/CI scripts that should be maintained though
* we use `pytest` to run automated tests for our code
* until we are ready to share it with developers the new `Breeze` script resides in `dev/Breeze` folder,
without yet linking it from main directory of Airflow. Later we will link to it from the main directory
- likely as `Breeze2` script (in some environments where filesystem is case-insensitive (MacOS) you cannot
+ likely as `breeze` script (in some environments where filesystem is case-insensitive (MacOS) you cannot
really put two files differing only by case in the same folder.
* There is enough overlap between the CI and Breeze to reuse a lot of commands for building images and
other CI actions that they should be shared between Breeze and CI. Therefore `dev/Breeze` will
diff --git a/dev/breeze/doc/adr/0003-bootstrapping-virtual-environment.md b/dev/breeze/doc/adr/0003-bootstrapping-virtual-environment.md
index 79b485f52d..5ade92a9c8 100644
--- a/dev/breeze/doc/adr/0003-bootstrapping-virtual-environment.md
+++ b/dev/breeze/doc/adr/0003-bootstrapping-virtual-environment.md
@@ -87,7 +87,7 @@ Integration we run. It's been established practice of the CI is that the logic
of the CI is stored in the same repository as the source code of the
application it tests and part of the Breeze functions are shared with CI.
-In the future when Breeze2 stabilizes and it's update cadence will be
+In the future when breeze stabilizes and it's update cadence will be
much slower (which is likele as it happened with the Breeze predecessor)
there could be an option that Breeze is installed as separate package and
same released Breeze version could be ued to manage multiple Airflow
@@ -108,10 +108,10 @@ The sub-project could be used in the future to produce a PyPI package
to install Breeze in a separate virtualenv bootstrapped
automatically in editable mode.
-There are two ways you will be able to install `Breeze2` - locally in
-repository using ./Breeze2 bootstrapping script and using `pipx`.
+There are two ways you will be able to install `breeze` - locally in
+repository using ./breeze bootstrapping script and using `pipx`.
-The bootstrapping Python script (`Breeze2` in the main repository
+The bootstrapping Python script (`breeze` in the main repository
of Airflow) performs the following tasks:
* when run for the first time it creates `.build/breeze2/venv` virtual
@@ -122,20 +122,20 @@ of Airflow) performs the following tasks:
* when run subsequently, it will check if setup files changed for
Breeze (dependencies changed) and if they did it will automatically
reinstall the environment, adding missing dependencies
-* after managing the venv, the Breeze2 script will simply execute
- the actual Breeze2 script in the `.build/venv` passing the
+* after managing the venv, the breeze script will simply execute
+ the actual breeze script in the `.build/venv` passing the
parameters to the script. For the user, the effect will be same
- as activating the virtualenv and executing the ./Breeze2 from
+ as activating the virtualenv and executing the ./breeze from
there (but it will happen automatically and invisibly for the
user
* In Windows environment where you have no easy/popular mechanism
of running scripts with shebang (#!) equivalent in Posix
environments, Users will be able to locally build (using
- `pyinstaller` a `Breeze2.exe` frozen Python script that will
- essentially do the same, they could also use `python Breeze2`
+ `pyinstaller` a `breeze.exe` frozen Python script that will
+ essentially do the same, they could also use `python breeze`
command or switch to Git Bash to utilize the shebang feature
(Git Bash comes together with Git when installed on Windows)
-* The second option is to use `pipx` to install Breeze2.
+* The second option is to use `pipx` to install breeze.
The `pipx` is almost equivalent to what the Bootstrapping does
and many users might actually choose to install Breeze this
way - and we will add it as an option to install Breeze
@@ -152,7 +152,7 @@ of Airflow) performs the following tasks:
environment, and it might create some confusions for the
users who would have to learn `pipx` and it's commands.
Another drawback of `pipx` is that installs one global
- version of Breeze2 for all projects, where it is quite
+ version of breeze for all projects, where it is quite
possible that someone has two different versions of
Airflow repository checked out and the bootstraping
script provides this capability.
@@ -184,13 +184,13 @@ The alternatives considered were:
purpose quite well. However the problem is that if you
also use `pyenv` to manage your `airflow` virtualenv this might
be source of confusion. Should I activate airflow virtualenv
- or Breeze2 venv to run tests? Part of Breeze experience is
+ or breeze venv to run tests? Part of Breeze experience is
to activate local Airflow virtualenv for IDE integration and
since this is different than simple Breeze virtualenv, using
pytest and autoactivation in this case might lead to a lot
of confusion. Keeping the Breeze virtualenv "hidden" and
mostly "used" but not deliberately activated is a better
- choice - especially that most users will simply "use" Breeze2
+ choice - especially that most users will simply "use" breeze
as an app rather than activate the environment deliberately.
Also choosing `pyenv` and it's virtualenv plugin would
add extra, unnecessary steps and prerequisites for Breeze.
@@ -200,7 +200,7 @@ The alternatives considered were:
Using Breeze for new users will be much simpler, without
having to install any prerequisites. The virtualenv used by
-Breeze2 will be hidden from the user, and used behind the
+breeze will be hidden from the user, and used behind the
scenes - and the dependencies used will be automatically
installed when needed. This will allow to seamlessly
integrate Breeze tool in the develiopment experience without
diff --git a/dev/breeze/setup.cfg b/dev/breeze/setup.cfg
index a5eaf6af30..abdc8b9c41 100644
--- a/dev/breeze/setup.cfg
+++ b/dev/breeze/setup.cfg
@@ -70,7 +70,7 @@ where=src
[options.entry_points]
console_scripts=
- Breeze2=airflow_breeze.breeze:main
+ breeze=airflow_breeze.breeze:main
airflow-freespace=airflow_ci.freespace:main
airflow-find-newer-dependencies=airflow_ci.find_newer_dependencies:main
diff --git a/dev/breeze/src/airflow_breeze/__init__.py b/dev/breeze/src/airflow_breeze/__init__.py
index b83b4e303f..3adc51a7b0 100644
--- a/dev/breeze/src/airflow_breeze/__init__.py
+++ b/dev/breeze/src/airflow_breeze/__init__.py
@@ -14,5 +14,5 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-NAME = "Breeze2"
-VERSION = "0.0.1"
+NAME = "breeze"
+VERSION = "0.9.0"
diff --git a/dev/breeze/src/airflow_breeze/breeze.py b/dev/breeze/src/airflow_breeze/breeze.py
index 8795aa617a..499626b20c 100755
--- a/dev/breeze/src/airflow_breeze/breeze.py
+++ b/dev/breeze/src/airflow_breeze/breeze.py
@@ -15,6 +15,7 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
+import atexit
import os
import shutil
import subprocess
@@ -23,8 +24,11 @@ from dataclasses import dataclass
from pathlib import Path
from typing import List, Optional, Tuple
+import rich
+
from airflow_breeze import NAME, VERSION
from airflow_breeze.shell.shell_params import ShellParams
+from airflow_breeze.utils.confirm import Answer, set_forced_answer, user_confirm
from airflow_breeze.utils.reinstall import ask_to_reinstall_breeze, reinstall_breeze, warn_non_editable
try:
@@ -39,7 +43,7 @@ try:
"\nTo find out more, visit [bright_blue]https://github.com/apache/airflow/blob/main/BREEZE.rst[/]\n"
)
click.rich_click.OPTION_GROUPS = {
- "Breeze2": [
+ "breeze": [
{
"name": "Basic flags for the default (shell) command",
"options": [
@@ -62,7 +66,7 @@ try:
],
},
],
- "Breeze2 shell": [
+ "breeze shell": [
{
"name": "Basic flags",
"options": [
@@ -85,7 +89,7 @@ try:
],
},
],
- "Breeze2 start-airflow": [
+ "breeze start-airflow": [
{
"name": "Basic flags",
"options": [
@@ -110,7 +114,7 @@ try:
],
},
],
- "Breeze2 build-image": [
+ "breeze build-image": [
{
"name": "Basic usage",
"options": [
@@ -148,7 +152,7 @@ try:
],
},
],
- "Breeze2 build-prod-image": [
+ "breeze build-prod-image": [
{
"name": "Basic usage",
"options": [
@@ -202,7 +206,7 @@ try:
],
},
],
- "Breeze2 static-check": [
+ "breeze static-checks": [
{
"name": "Pre-commit flags",
"options": [
@@ -214,7 +218,7 @@ try:
],
},
],
- "Breeze2 build-docs": [
+ "breeze build-docs": [
{
"name": "Doc flags",
"options": [
@@ -224,7 +228,7 @@ try:
],
},
],
- "Breeze2 stop": [
+ "breeze stop": [
{
"name": "Stop flags",
"options": [
@@ -232,7 +236,7 @@ try:
],
},
],
- "Breeze2 setup-autocomplete": [
+ "breeze setup-autocomplete": [
{
"name": "Setup autocomplete flags",
"options": [
@@ -240,7 +244,7 @@ try:
],
},
],
- "Breeze2 config": [
+ "breeze config": [
{
"name": "Config flags",
"options": [
@@ -254,7 +258,7 @@ try:
}
click.rich_click.COMMAND_GROUPS = {
- "Breeze2": [
+ "breeze": [
{
"name": "Developer tools",
"commands": [
@@ -264,7 +268,7 @@ try:
"build-image",
"build-prod-image",
"build-docs",
- "static-check",
+ "static-checks",
],
},
{
@@ -301,7 +305,13 @@ from airflow_breeze.global_constants import (
)
from airflow_breeze.pre_commit_ids import PRE_COMMIT_LIST
from airflow_breeze.shell.enter_shell import enter_shell
-from airflow_breeze.utils.cache import delete_cache, touch_cache_file, write_to_cache_file
+from airflow_breeze.utils.cache import (
+ check_if_cache_exists,
+ delete_cache,
+ read_from_cache_file,
+ touch_cache_file,
+ write_to_cache_file,
+)
from airflow_breeze.utils.console import console
from airflow_breeze.utils.docker_command_utils import (
check_docker_resources,
@@ -318,12 +328,15 @@ from airflow_breeze.utils.path_utils import (
get_package_setup_metadata_hash,
get_used_airflow_sources,
get_used_sources_setup_metadata_hash,
+ in_autocomplete,
)
from airflow_breeze.utils.run_utils import check_pre_commit_installed, run_command
from airflow_breeze.utils.visuals import ASCIIART, ASCIIART_STYLE
find_airflow_sources_root_to_operate_on()
+output_file_for_recording = os.environ.get('RECORD_BREEZE_OUTPUT_FILE')
+
option_verbose = click.option(
"-v", "--verbose", is_flag=True, help="Print verbose information about performed steps.", envvar='VERBOSE'
)
@@ -336,6 +349,13 @@ option_dry_run = click.option(
envvar='DRY_RUN',
)
+option_answer = click.option(
+ "-a",
+ "--answer",
+ type=click.Choice(['y', 'n', 'q', 'yes', 'no', 'quit']),
+ help="Force answer to questions.",
+ envvar='FORCE_ANSWER_TO_QUESTIONS',
+)
option_python = click.option(
'-p',
@@ -373,7 +393,7 @@ option_mssql_version = click.option(
option_executor = click.option(
'--executor',
- help='Executor to use in a kubernetes cluster. Default is KubernetesExecutor.',
+ help='Executor to use for a kubernetes cluster. Default is KubernetesExecutor.',
type=click.Choice(ALLOWED_EXECUTORS),
)
@@ -382,9 +402,9 @@ option_forward_credentials = click.option(
)
option_use_airflow_version = click.option(
- '-a',
+ '-V',
'--use-airflow-version',
- help="Use (reinstall) specified Airflow version after entering the container.",
+ help="Use (reinstall at entry) Airflow version from PyPI.",
envvar='USE_AIRFLOW_VERSION',
)
@@ -392,7 +412,7 @@ option_mount_sources = click.option(
'--mount-sources',
type=click.Choice(ALLOWED_MOUNT_OPTIONS),
default=ALLOWED_MOUNT_OPTIONS[0],
- help="Choose which local sources should be mounted (default = selected)",
+ help="Choose scope of local sources should be mounted (default = selected).",
)
option_force_build = click.option('--force-build', help="Force image build before running.", is_flag=True)
@@ -400,7 +420,7 @@ option_force_build = click.option('--force-build', help="Force image build befor
option_db_reset = click.option(
'-d',
'--db-reset',
- help="Resets DB when entering the container.",
+ help="Reset DB when entering the container.",
is_flag=True,
envvar='DB_RESET',
)
@@ -420,6 +440,7 @@ option_db_reset = click.option(
@option_mount_sources
@option_integration
@option_db_reset
+@option_answer
@click.pass_context
def main(ctx: Context, **kwargs):
create_directories()
@@ -454,7 +475,7 @@ option_image_tag = click.option(
option_platform = click.option(
'--platform',
- help='Builds image for the platform specified.',
+ help='Platform for Airflow image.',
envvar='PLATFORM',
type=click.Choice(ALLOWED_PLATFORMS),
)
@@ -469,12 +490,13 @@ option_debian_version = click.option(
option_upgrade_to_newer_dependencies = click.option(
"-u",
'--upgrade-to-newer-dependencies',
- help='If set to anything else than false, upgrades PIP packages to latest versions available.',
+ default="false",
+ help='When other than "false", upgrade all PIP packages to latest.',
envvar='UPGRADE_TO_NEWER_DEPENDENCIES',
)
option_additional_extras = click.option(
'--additional-extras',
- help='This installs additional extra package while installing airflow in the image.',
+ help='Additional extra package while installing Airflow in the image.',
envvar='ADDITIONAL_AIRFLOW_EXTRAS',
)
option_additional_dev_apt_deps = click.option(
@@ -514,22 +536,22 @@ option_additional_runtime_apt_env = click.option(
)
option_dev_apt_command = click.option(
'--dev-apt-command',
- help='The basic command executed before dev apt deps are installed.',
+ help='Command executed before dev apt deps are installed.',
envvar='DEV_APT_COMMAND',
)
option_dev_apt_deps = click.option(
'--dev-apt-deps',
- help='The basic apt dev dependencies to use when building the images.',
+ help='Apt dev dependencies to use when building the images.',
envvar='DEV_APT_DEPS',
)
option_runtime_apt_command = click.option(
'--runtime-apt-command',
- help='The basic command executed before runtime apt deps are installed.',
+ help='Command executed before runtime apt deps are installed.',
envvar='RUNTIME_APT_COMMAND',
)
option_runtime_apt_deps = click.option(
'--runtime-apt-deps',
- help='The basic apt runtime dependencies to use when building the images.',
+ help='Apt runtime dependencies to use when building the images.',
envvar='RUNTIME_APT_DEPS',
)
@@ -613,6 +635,7 @@ def version(verbose: bool):
@option_mount_sources
@option_integration
@option_db_reset
+@option_answer
@click.argument('extra-args', nargs=-1, type=click.UNPROCESSED)
def shell(
verbose: bool,
@@ -628,10 +651,11 @@ def shell(
use_airflow_version: str,
force_build: bool,
db_reset: bool,
+ answer: Optional[str],
extra_args: Tuple,
):
- """Enters breeze.py environment. this is the default command use when no other is selected."""
-
+ """Enter breeze.py environment. this is the default command use when no other is selected."""
+ set_forced_answer(answer)
if verbose:
console.print("\n[green]Welcome to breeze.py[/]\n")
console.print(f"\n[green]Root of Airflow Sources = {AIRFLOW_SOURCES_ROOT}[/]\n")
@@ -669,6 +693,7 @@ def shell(
@option_mount_sources
@option_integration
@option_db_reset
+@option_answer
@click.argument('extra-args', nargs=-1, type=click.UNPROCESSED)
def start_airflow(
verbose: bool,
@@ -686,9 +711,11 @@ def start_airflow(
use_airflow_version: str,
force_build: bool,
db_reset: bool,
+ answer: Optional[str],
extra_args: Tuple,
):
- """Enters breeze.py environment and starts all Airflow components in the tmux session."""
+ """Enter breeze.py environment and starts all Airflow components in the tmux session."""
+ set_forced_answer(answer)
enter_shell(
verbose=verbose,
dry_run=dry_run,
@@ -735,6 +762,7 @@ def start_airflow(
@option_dev_apt_deps
@option_runtime_apt_command
@option_runtime_apt_deps
+@option_answer
def build_ci_image(
verbose: bool,
dry_run: bool,
@@ -758,10 +786,11 @@ def build_ci_image(
platform: Optional[str],
debian_version: Optional[str],
prepare_buildx_cache: bool,
+ answer: Optional[str],
upgrade_to_newer_dependencies: str = "false",
):
- """Builds docker CI image."""
-
+ """Build CI image."""
+ set_forced_answer(answer)
if verbose:
console.print(
f"\n[bright_blue]Building image of airflow from {AIRFLOW_SOURCES_ROOT} "
@@ -807,33 +836,35 @@ def build_ci_image(
@option_prepare_buildx_cache
@click.option(
'--installation-method',
- help="Whether to install airflow from sources ('.') or PyPI ('apache-airflow')",
+ help="Install Airflow from: sources or PyPI.",
type=click.Choice(ALLOWED_INSTALLATION_METHODS),
)
@option_install_providers_from_sources
@click.option(
'--install-from-docker-context-files',
- help='Install wheels from local docker-context-files when building image',
+ help='Install wheels from local docker-context-files when building image.',
is_flag=True,
)
@click.option(
'--cleanup-docker-context-files',
- help='Cleans up docker context files before running build.',
+ help='Clean up docker context files before running build.',
is_flag=True,
)
-@click.option('--extras', help="Extras to install by default")
-@click.option('--disable-mysql-client-installation', help="Do not install MySQL client", is_flag=True)
-@click.option('--disable-mssql-client-installation', help="Do not install MsSQl client", is_flag=True)
-@click.option('--disable-postgres-client-installation', help="Do not install Postgres client", is_flag=True)
+@click.option('--extras', help="Extras to install by default.")
+@click.option('--disable-mysql-client-installation', help="Do not install MySQL client.", is_flag=True)
+@click.option('--disable-mssql-client-installation', help="Do not install MsSQl client.", is_flag=True)
+@click.option('--disable-postgres-client-installation', help="Do not install Postgres client.", is_flag=True)
@click.option(
- '--disable-airflow-repo-cache', help="Disable cache from Airflow repository during building", is_flag=True
+ '--disable-airflow-repo-cache',
+ help="Disable cache from Airflow repository during building.",
+ is_flag=True,
)
-@click.option('--disable-pypi', help="Disable pypi during building", is_flag=True)
+@click.option('--disable-pypi', help="Disable PyPI during building.", is_flag=True)
@click.option(
'--install-airflow-reference',
- help="Install airflow using specified reference (tag/branch) from GitHub",
+ help="Install Airflow using GitHub tag or branch.",
)
-@click.option('-a', '--install-airflow-version', help="Install specified version of airflow")
+@click.option('-V', '--install-airflow-version', help="Install version of Airflow from PyPI.")
@option_additional_extras
@option_additional_dev_apt_deps
@option_additional_runtime_apt_deps
@@ -846,6 +877,7 @@ def build_ci_image(
@option_dev_apt_deps
@option_runtime_apt_command
@option_runtime_apt_deps
+@option_answer
def build_prod_image(
verbose: bool,
dry_run: bool,
@@ -880,9 +912,11 @@ def build_prod_image(
extras: Optional[str],
installation_method: Optional[str],
install_from_docker_context_files: bool,
+ answer: Optional[str],
upgrade_to_newer_dependencies: str = "false",
):
- """Builds docker Production image."""
+ """Build Production image."""
+ set_forced_answer(answer)
if verbose:
console.print("\n[bright_blue]Building image[/]\n")
if prepare_buildx_cache:
@@ -1002,12 +1036,13 @@ def write_to_shell(command_to_execute: str, dry_run: bool, script_path: str, for
is_flag=True,
help='Force autocomplete setup even if already setup before (overrides the setup).',
)
+@option_answer
@main.command(name='setup-autocomplete')
-def setup_autocomplete(verbose: bool, dry_run: bool, force: bool):
+def setup_autocomplete(verbose: bool, dry_run: bool, force: bool, answer: Optional[str]):
"""
- Enables autocompletion of Breeze2 commands.
+ Enables autocompletion of breeze commands.
"""
-
+ set_forced_answer(answer)
# Determine if the shell is bash/zsh/powershell. It helps to build the autocomplete path
detected_shell = os.environ.get('SHELL')
detected_shell = None if detected_shell is None else detected_shell.split(os.sep)[-1]
@@ -1019,12 +1054,9 @@ def setup_autocomplete(verbose: bool, dry_run: bool, force: bool):
AIRFLOW_SOURCES_ROOT / "dev" / "breeze" / "autocomplete" / f"{NAME}-complete-{detected_shell}.sh"
)
console.print(f"[bright_blue]Activation command script is available here: {autocomplete_path}[/]\n")
- console.print(
- f"[bright_yellow]We need to add above script to your {detected_shell} profile and "
- "install 'click' package in your default python installation destination.[/]\n"
- )
- if click.confirm("Should we proceed ?"):
- run_command(['pip', 'install', '--upgrade', 'click'], verbose=True, dry_run=dry_run, check=False)
+ console.print(f"[bright_yellow]We need to add above script to your {detected_shell} profile.[/]\n")
+ answer = user_confirm("Should we proceed ?", default_answer=Answer.NO, timeout=3)
+ if answer == Answer.YES:
if detected_shell == 'bash':
script_path = str(Path('~').expanduser() / '.bash_completion')
command_to_execute = f"source {autocomplete_path}"
@@ -1053,42 +1085,60 @@ def setup_autocomplete(verbose: bool, dry_run: bool, force: bool):
)
command_to_execute = f". {autocomplete_path}"
write_to_shell(command_to_execute, dry_run, script_path, force)
- else:
+ elif answer == Answer.NO:
console.print(
"\nPlease follow the https://click.palletsprojects.com/en/8.1.x/shell-completion/ "
"to setup autocompletion for breeze manually if you want to use it.\n"
)
+ else:
+ sys.exit(0)
@main.command(name='config')
@option_python
@option_backend
-@click.option('-C/-c', '--cheatsheet/--no-cheatsheet', help="Enable/disable cheatsheet", default=None)
-@click.option('-A/-a', '--asciiart/--no-asciiart', help="Enable/disable ASCIIart", default=None)
+@click.option('-C/-c', '--cheatsheet/--no-cheatsheet', help="Enable/disable cheatsheet.", default=None)
+@click.option('-A/-a', '--asciiart/--no-asciiart', help="Enable/disable ASCIIart.", default=None)
def change_config(python, backend, cheatsheet, asciiart):
"""
- Toggles on/off cheatsheet, asciiart. Sets default Python and backend.
+ Show/update configuration (Python, Backend, Cheatsheet, ASCIIART).
"""
- if asciiart:
- console.print('[bright_blue] ASCIIART enabled')
- delete_cache('suppress_asciiart')
- elif asciiart is not None:
- touch_cache_file('suppress_asciiart')
- else:
- pass
- if cheatsheet:
- console.print('[bright_blue] Cheatsheet enabled')
- delete_cache('suppress_cheatsheet')
- elif cheatsheet is not None:
- touch_cache_file('suppress_cheatsheet')
- else:
- pass
+ asciiart_file = "suppress_asciiart"
+ cheatsheet_file = "suppress_cheatsheet"
+ python_file = 'PYTHON_MAJOR_MINOR_VERSION'
+ backend_file = 'BACKEND'
+ if asciiart is not None:
+ if asciiart:
+ delete_cache(asciiart_file)
+ console.print('[bright_blue]Enable ASCIIART![/]')
+ else:
+ touch_cache_file(asciiart_file)
+ console.print('[bright_blue]Disable ASCIIART![/]')
+ if cheatsheet is not None:
+ if cheatsheet:
+ delete_cache(cheatsheet_file)
+ console.print('[bright_blue]Enable Cheatsheet[/]')
+ elif cheatsheet is not None:
+ touch_cache_file(cheatsheet_file)
+ console.print('[bright_blue]Disable Cheatsheet[/]')
if python is not None:
- write_to_cache_file('PYTHON_MAJOR_MINOR_VERSION', python)
- console.print(f'[bright_blue]Python cached_value {python}')
+ write_to_cache_file(python_file, python)
+ console.print(f'[bright_blue]Python default value set to: {python}[/]')
if backend is not None:
- write_to_cache_file('BACKEND', backend)
- console.print(f'[bright_blue]Backend cached_value {backend}')
+ write_to_cache_file(backend_file, backend)
+ console.print(f'[bright_blue]Backend default value set to: {backend}[/]')
+
+ def get_status(file: str):
+ return "disabled" if check_if_cache_exists(file) else "enabled"
+
+ console.print()
+ console.print("[bright_blue]Current configuration:[/]")
+ console.print()
+ console.print(f"[bright_blue]* Python: {read_from_cache_file(python_file)}[/]")
+ console.print(f"[bright_blue]* Backend: {read_from_cache_file(backend_file)}[/]")
+ console.print(f"[bright_blue]* ASCIIART: {get_status(asciiart_file)}[/]")
+ console.print(f"[bright_blue]* Cheatsheet: {get_status(cheatsheet_file)}[/]")
+ console.print()
@dataclass
@@ -1113,25 +1163,19 @@ class DocParams:
@main.command(name='build-docs')
@option_verbose
@option_dry_run
-@click.option('-d', '--docs-only', help="Only build documentation", is_flag=True)
-@click.option('-s', '--spellcheck-only', help="Only run spell checking", is_flag=True)
+@click.option('-d', '--docs-only', help="Only build documentation.", is_flag=True)
+@click.option('-s', '--spellcheck-only', help="Only run spell checking.", is_flag=True)
@click.option(
'-p',
'--package-filter',
- help="List of packages to consider",
+ help="List of packages to consider.",
type=click.Choice(get_available_packages()),
multiple=True,
)
def build_docs(
verbose: bool, dry_run: bool, docs_only: bool, spellcheck_only: bool, package_filter: Tuple[str]
):
- """
- Builds documentation in the container.
-
- * figures out CI image name
- * checks if there are enough resources
- * converts parameters into a DocParams class
- """
+ """Build documentation in the container."""
params = BuildCiParams()
ci_image_name = params.airflow_image_name
check_docker_resources(verbose, ci_image_name)
@@ -1152,7 +1196,7 @@ def build_docs(
@main.command(
- name="static-check",
+ name="static-checks",
help="Run static checks.",
context_settings=dict(
ignore_unknown_options=True,
@@ -1162,18 +1206,20 @@ def build_docs(
@click.option(
'-t',
'--type',
- help="Type(s) of the static checks to run",
+ help="Type(s) of the static checks to run (multiple can be added).",
type=click.Choice(PRE_COMMIT_LIST),
multiple=True,
)
-@click.option('-a', '--all-files', help="Run checks on all files", is_flag=True)
-@click.option('-f', '--files', help="List of files to run the checks on", multiple=True)
-@click.option('-s', '--show-diff-on-failure', help="Show diff for files modified by the checks", is_flag=True)
-@click.option('-c', '--last-commit', help="Run check for all files in all commits", is_flag=True)
+@click.option('-a', '--all-files', help="Run checks on all files.", is_flag=True)
+@click.option('-f', '--files', help="List of files to run the checks on.", multiple=True)
+@click.option(
+ '-s', '--show-diff-on-failure', help="Show diff for files modified by the checks.", is_flag=True
+)
+@click.option('-c', '--last-commit', help="Run checks for all files in last commit.", is_flag=True)
@option_verbose
@option_dry_run
@click.argument('precommit_args', nargs=-1, type=click.UNPROCESSED)
-def static_check(
+def static_checks(
verbose: bool,
dry_run: bool,
all_files: bool,
@@ -1209,13 +1255,13 @@ def static_check(
)
-@main.command(name="stop", help="Stops running breeze environment.")
+@main.command(name="stop", help="Stop running breeze environment.")
@option_verbose
@option_dry_run
@click.option(
"-p",
"--preserve-volumes",
- help="By default the stop command removes volumes with data. " "Specifying the flag will preserve them.",
+ help="Skip removing volumes when stopping Breeze.",
is_flag=True,
)
def stop(verbose: bool, dry_run: bool, preserve_volumes: bool):
@@ -1236,12 +1282,15 @@ def stop(verbose: bool, dry_run: bool, preserve_volumes: bool):
@click.option(
'--use-current-airflow-sources',
is_flag=True,
- help=f'Use current Airflow sources for upgrade rather than from {get_installation_airflow_sources()}.',
+ help='Use current workdir Airflow sources for upgrade'
+ + (f" rather than from {get_installation_airflow_sources()}." if not output_file_for_recording else "."),
)
@main.command(
name='self-upgrade',
help="Self upgrade Breeze. By default it re-installs Breeze "
- f"from {get_installation_airflow_sources()}.",
+ f"from {get_installation_airflow_sources()}."
+ if not output_file_for_recording
+ else "Self upgrade Breeze.",
)
def self_upgrade(force: bool, use_current_airflow_sources: bool):
if use_current_airflow_sources:
@@ -1259,12 +1308,28 @@ def self_upgrade(force: bool, use_current_airflow_sources: bool):
sys.exit(1)
-@main.command(name="cleanup", help="Removes the cache of parameters, images and cleans up docker cache.")
+@main.command(
+ name="cleanup",
+ help=" the cache of parameters, docker cache and optionally - currently downloaded images.",
+)
+@click.option(
+ '--also-remove-current-images',
+ is_flag=True,
+ help='Also remove currently downloaded Breeze images.',
+)
@option_verbose
+@option_answer
@option_dry_run
-def cleanup(verbose: bool, dry_run: bool):
- console.print("\n[bright_yellow]Removing cache of parameters, images, and cleans up docker cache[/]")
- if click.confirm("Are you sure?"):
+def cleanup(verbose: bool, dry_run: bool, also_remove_current_images: bool, answer: Optional[str]):
+ set_forced_answer(answer)
+ if also_remove_current_images:
+ console.print(
+ "\n[bright_yellow]Removing cache of parameters, clean up docker cache "
+ "and remove locally downloaded images[/]"
+ )
+ else:
+ console.print("\n[bright_yellow]Removing cache of parameters, and cleans up docker cache[/]")
+ if also_remove_current_images:
docker_images_command_to_execute = [
'docker',
'images',
@@ -1288,16 +1353,64 @@ def cleanup(verbose: bool, dry_run: bool):
'--force',
]
docker_rmi_command_to_execute.extend(images)
- run_command(docker_rmi_command_to_execute, verbose=verbose, dry_run=dry_run, check=False)
+ answer = user_confirm("Are you sure?", timeout=None)
+ if answer == Answer.YES:
+ run_command(docker_rmi_command_to_execute, verbose=verbose, dry_run=dry_run, check=False)
+ elif answer == Answer.QUIT:
+ sys.exit(0)
else:
- console.print("[light_blue]No images to remote[/]\n")
+ console.print("[light_blue]No locally downloaded images to remove[/]\n")
+ console.print("Pruning docker images")
+ answer = user_confirm("Are you sure?", timeout=None)
+ if answer == Answer.YES:
system_prune_command_to_execute = ['docker', 'system', 'prune']
- console.print("Pruning docker images")
run_command(system_prune_command_to_execute, verbose=verbose, dry_run=dry_run, check=False)
- console.print(f"Removing build cache dir ${BUILD_CACHE_DIR}")
+ elif answer == Answer.QUIT:
+ sys.exit(0)
+ console.print(f"Removing build cache dir ${BUILD_CACHE_DIR}")
+ answer = user_confirm("Are you sure?", timeout=None)
+ if answer == Answer.YES:
if not dry_run:
shutil.rmtree(BUILD_CACHE_DIR, ignore_errors=True)
+ elif answer == Answer.QUIT:
+ sys.exit(0)
+
+help_console = None
+
+
+def enable_recording_of_help_output(path: str, title: Optional[str], width: Optional[str]):
+ if not title:
+ title = "Breeze screenshot"
+ if not width:
+ width_int = 120
+ else:
+ width_int = int(width)
+
+ def save_ouput_as_svg():
+ if help_console:
+ help_console.save_svg(path=path, title=title)
+
+ class RecordingConsole(rich.console.Console):
+ def __init__(self, **kwargs):
+ super().__init__(record=True, width=width_int, force_terminal=True, **kwargs)
+ global help_console
+ help_console = self
+
+ atexit.register(save_ouput_as_svg)
+ click.rich_click.MAX_WIDTH = width_int
+ click.formatting.FORCED_WIDTH = width_int
+ click.rich_click.COLOR_SYSTEM = "standard"
+ # monkeypatch rich_click console to record help (rich_click does not allow passing extra args to console)
+ click.rich_click.Console = RecordingConsole
+
+
+if output_file_for_recording and not in_autocomplete():
+ enable_recording_of_help_output(
+ path=output_file_for_recording,
+ title=os.environ.get('RECORD_BREEZE_TITLE'),
+ width=os.environ.get('RECORD_BREEZE_WIDTH'),
+ )
if __name__ == '__main__':
main()
diff --git a/dev/breeze/src/airflow_breeze/global_constants.py b/dev/breeze/src/airflow_breeze/global_constants.py
index 013a73a21d..e177df0d44 100644
--- a/dev/breeze/src/airflow_breeze/global_constants.py
+++ b/dev/breeze/src/airflow_breeze/global_constants.py
@@ -122,7 +122,9 @@ EXCLUDE_DOCS_PACKAGE_FOLDER = [
def get_available_packages() -> List[str]:
docs_path_content = (AIRFLOW_SOURCES_ROOT / 'docs').glob('*/')
available_packages = [x.name for x in docs_path_content if x.is_dir()]
- return list(set(available_packages) - set(EXCLUDE_DOCS_PACKAGE_FOLDER))
+ package_list = list(set(available_packages) - set(EXCLUDE_DOCS_PACKAGE_FOLDER))
+ package_list.sort()
+ return package_list
# Initialise base variables
diff --git a/dev/breeze/src/airflow_breeze/pre_commit_ids.py b/dev/breeze/src/airflow_breeze/pre_commit_ids.py
index f9a9aaadbf..92e841eda4 100644
--- a/dev/breeze/src/airflow_breeze/pre_commit_ids.py
+++ b/dev/breeze/src/airflow_breeze/pre_commit_ids.py
@@ -64,6 +64,7 @@ PRE_COMMIT_LIST = [
'isort',
'json-schema',
'language-matters',
+ 'limit-breeze-dependencies',
'lint-dockerfile',
'lint-openapi',
'markdownlint',
diff --git a/dev/breeze/src/airflow_breeze/shell/enter_shell.py b/dev/breeze/src/airflow_breeze/shell/enter_shell.py
index f5a1cf73b7..3cb03b054e 100644
--- a/dev/breeze/src/airflow_breeze/shell/enter_shell.py
+++ b/dev/breeze/src/airflow_breeze/shell/enter_shell.py
@@ -19,8 +19,6 @@ import sys
from pathlib import Path
from typing import Dict
-import click
-
from airflow_breeze import global_constants
from airflow_breeze.build_image.ci.build_ci_image import build_image
from airflow_breeze.shell.shell_params import ShellParams
@@ -78,7 +76,10 @@ def build_image_if_needed_steps(verbose: bool, dry_run: bool, shell_params: Shel
"\n[bright_yellow]This might take a lot of time, w"
"e think you should rebase first.[/]\n"
)
- if click.confirm("But if you really, really want - you can do it"):
+ answer = user_confirm(
+ "But if you really, really want - you can do it", timeout=5, default_answer=Answer.NO
+ )
+ if answer == Answer.YES:
build_image(
verbose=verbose,
dry_run=dry_run,
diff --git a/dev/breeze/src/airflow_breeze/utils/cache.py b/dev/breeze/src/airflow_breeze/utils/cache.py
index bc22516c41..d2e1117ab3 100644
--- a/dev/breeze/src/airflow_breeze/utils/cache.py
+++ b/dev/breeze/src/airflow_breeze/utils/cache.py
@@ -60,7 +60,6 @@ def write_to_cache_file(param_name: str, param_value: str, check_allowed_values:
if check_allowed_values:
allowed, allowed_values = check_if_values_allowed(param_name, param_value)
if allowed or not check_allowed_values:
- print('BUILD CACHE DIR:', BUILD_CACHE_DIR)
cache_path = Path(BUILD_CACHE_DIR, f".{param_name}")
cache_path.parent.mkdir(parents=True, exist_ok=True)
cache_path.write_text(param_value)
diff --git a/dev/breeze/src/airflow_breeze/utils/confirm.py b/dev/breeze/src/airflow_breeze/utils/confirm.py
index c81494c84b..620608d295 100644
--- a/dev/breeze/src/airflow_breeze/utils/confirm.py
+++ b/dev/breeze/src/airflow_breeze/utils/confirm.py
@@ -14,6 +14,7 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
+import os
import sys
from enum import Enum
from typing import Optional
@@ -25,12 +26,24 @@ class Answer(Enum):
QUIT = 2
+forced_answer: Optional[str] = None
+
+
+def set_forced_answer(answer: Optional[str]):
+ global forced_answer
+ forced_answer = answer
+
+
def user_confirm(
- message: str, timeout: float, default_answer: Optional[Answer] = None, quit_allowed: bool = True
+ message: str,
+ timeout: Optional[float],
+ default_answer: Optional[Answer] = Answer.NO,
+ quit_allowed: bool = True,
) -> Answer:
"""
Ask the user for confirmation.
+ :rtype: object
:param message: message to display to the user (should end with the question mark)
:param timeout: time given user to answer
:param default_answer: default value returned on timeout. If no default - the question is
@@ -43,15 +56,19 @@ def user_confirm(
allowed_answers = "y/n/q" if quit_allowed else "y/n"
while True:
try:
- user_status = inputimeout(
- prompt=f'\n{message} \nPress {allowed_answers} in {timeout} seconds: ',
- timeout=timeout,
- )
+ force = forced_answer or os.environ.get('FORCE_ANSWER_TO_QUESTIONS')
+ if force:
+ user_status = force
+ else:
+ user_status = inputimeout(
+ prompt=f'\n{message} \nPress {allowed_answers} in {timeout} seconds: ',
+ timeout=timeout,
+ )
if user_status.upper() in ['Y', 'YES']:
return Answer.YES
elif user_status.upper() in ['N', 'NO']:
return Answer.NO
- elif user_status.upper() in ['Q', 'QUIT']:
+ elif user_status.upper() in ['Q', 'QUIT'] and quit_allowed:
return Answer.QUIT
else:
print(f"Wrong answer given {user_status}. Should be one of {allowed_answers}. Try again.")
diff --git a/dev/breeze/src/airflow_breeze/utils/console.py b/dev/breeze/src/airflow_breeze/utils/console.py
index ab7679f0f0..ca5a4a3096 100644
--- a/dev/breeze/src/airflow_breeze/utils/console.py
+++ b/dev/breeze/src/airflow_breeze/utils/console.py
@@ -18,12 +18,23 @@
Console used by all processes. We are forcing colors and terminal output as Breeze is supposed
to be only run in CI or real development terminal - in both cases we want to have colors on.
"""
+import os
+
try:
from rich.console import Console
from rich.theme import Theme
+ recording_width = os.environ.get("RECORD_BREEZE_WIDTH")
+ recording_file = os.environ.get("RECORD_BREEZE_OUTPUT_FILE")
+
custom_theme = Theme({"info": "blue", "warning": "magenta", "error": "red"})
- console = Console(force_terminal=True, color_system="standard", width=180, theme=custom_theme)
+ console = Console(
+ force_terminal=True,
+ color_system="standard",
+ width=180 if not recording_width else int(recording_width),
+ theme=custom_theme,
+ record=True if recording_file else False,
+ )
except ImportError:
# We handle the ImportError so that autocomplete works with just click installed
diff --git a/dev/breeze/src/airflow_breeze/utils/path_utils.py b/dev/breeze/src/airflow_breeze/utils/path_utils.py
index 7a71d00136..dc40e99ac8 100644
--- a/dev/breeze/src/airflow_breeze/utils/path_utils.py
+++ b/dev/breeze/src/airflow_breeze/utils/path_utils.py
@@ -211,7 +211,6 @@ def find_airflow_sources_root_to_operate_on() -> Path:
# only print warning and sleep if not producing complete results
print_warning_if_different_sources(airflow_sources)
print_warning_if_setup_changed()
- console.print(f"[bright_blue]Airflow sources: {airflow_sources}[/]")
os.chdir(str(airflow_sources))
return airflow_sources
diff --git a/dev/breeze/src/airflow_breeze/utils/run_utils.py b/dev/breeze/src/airflow_breeze/utils/run_utils.py
index da0ceeeb7e..f4d988122d 100644
--- a/dev/breeze/src/airflow_breeze/utils/run_utils.py
+++ b/dev/breeze/src/airflow_breeze/utils/run_utils.py
@@ -162,7 +162,7 @@ def instruct_build_image(python: str):
"""Print instructions to the user that they should build the image"""
console.print(f'[bright_yellow]\nThe CI image for ' f'python version {python} may be outdated[/]\n')
console.print('Please run this command at earliest convenience:\n')
- console.print(f' `./Breeze2 build-image --python {python}`\n')
+ console.print(f' `./breeze build-image --python {python}`\n')
@contextlib.contextmanager
diff --git a/dev/breeze/src/airflow_breeze/utils/visuals.py b/dev/breeze/src/airflow_breeze/utils/visuals.py
index b5148a2c74..a42c250444 100644
--- a/dev/breeze/src/airflow_breeze/utils/visuals.py
+++ b/dev/breeze/src/airflow_breeze/utils/visuals.py
@@ -78,16 +78,16 @@ CHEATSHEET = f"""
[bright_blue]* Installation[/]
- When you have multiple copies of Airflow, it's better if you use `./Breeze2` from those
- repository as it will have the latest version of Breeze2 and it's dependencies.
+ When you have multiple copies of Airflow, it's better if you use `./breeze` from those
+ repository as it will have the latest version of breeze and it's dependencies.
However if you only have one Airflow repository and you have `pipx` installed, you can use
- `pipx` to install `Breeze2` command in your path (`Breeze2` command is run from this repository then)
+ `pipx` to install `breeze` command in your path (`breeze` command is run from this repository then)
pipx install -e ./dev/breeze --force
- In case you use `pipx`, you might need to occasionally reinstall `Breeze2` with the `--force` flag
- when dependencies change for it. You do not have to do it when you use it via `./Breeze2`
+ In case you use `pipx`, you might need to occasionally reinstall `breeze` with the `--force` flag
+ when dependencies change for it. You do not have to do it when you use it via `./breeze`
[bright_blue]* Port forwarding:[/]
@@ -128,7 +128,7 @@ CHEATSHEET = f"""
[bright_blue]* Other options[/]
- Check out `--help` for ./Breeze2 commands. It will show you other options, such as running
+ Check out `--help` for ./breeze commands. It will show you other options, such as running
integration or starting complete Airflow using `start-airflow` command as well as ways
of cleaning up the installation.
@@ -137,7 +137,7 @@ CHEATSHEET = f"""
You can disable this cheatsheet by running:
- ./Breeze2 config --no-cheatsheet
+ ./breeze config --no-cheatsheet
"""
CHEATSHEET_STYLE = "white"
diff --git a/dev/provider_packages/README.md b/dev/provider_packages/README.md
index 838e76af57..e46c1dfe94 100644
--- a/dev/provider_packages/README.md
+++ b/dev/provider_packages/README.md
@@ -97,7 +97,7 @@ The `index.rst` stored in the `docs\apache-airflow-providers-<PROVIDER>` folder
When you want to prepare release notes for a package, you need to run:
```
-./breeze prepare-provider-documentation <PACKAGE_ID> ...
+./breeze-legacy prepare-provider-documentation <PACKAGE_ID> ...
```
* <PACKAGE_ID> is usually directory in the `airflow/providers` folder (for example `google` but in several
diff --git a/dev/refresh_images.sh b/dev/refresh_images.sh
index c883a28f95..0ae90e9247 100755
--- a/dev/refresh_images.sh
+++ b/dev/refresh_images.sh
@@ -31,5 +31,5 @@ fi
python_version=$1
-./breeze prepare-build-cache --python "${python_version}" --platform linux/amd64,linux/arm64 --verbose
-./breeze prepare-build-cache --python "${python_version}" --platform linux/amd64,linux/arm64 --production-image --verbose
+breeze build-image --prepare-buildx-cache --python "${python_version}" --platform linux/amd64,linux/arm64 --verbose
+breeze build-prod-image --prepare-buildx-cache --python "${python_version}" --platform linux/amd64,linux/arm64 --verbose
diff --git a/docs/build_docs.py b/docs/build_docs.py
index 978e788838..f2361f00ab 100755
--- a/docs/build_docs.py
+++ b/docs/build_docs.py
@@ -69,12 +69,12 @@ def _promote_new_flags():
console.print()
if ON_GITHUB_ACTIONS:
console.print("You can quickly build documentation locally with just one command.")
- console.print(" [blue]./breeze build-docs[/]")
+ console.print(" [blue]breeze build-docs[/]")
console.print()
console.print("[yellow]Still too slow?[/]")
console.print()
console.print("You can only build one documentation package:")
- console.print(" [blue]./breeze build-docs -- --package-filter <PACKAGE-NAME>[/]")
+ console.print(" [blue]breeze build-docs --package-filter <PACKAGE-NAME>[/]")
console.print()
console.print("This usually takes from [yellow]20 seconds[/] to [yellow]2 minutes[/].")
console.print()
@@ -83,7 +83,7 @@ def _promote_new_flags():
console.print(" [blue]--spellcheck-only - Only perform spellchecking[/]")
console.print()
console.print("For more info:")
- console.print(" [blue]./breeze build-docs --help[/]")
+ console.print(" [blue]breeze build-docs --help[/]")
console.print()
diff --git a/images/breeze/output-breeze-commands.svg b/images/breeze/output-breeze-commands.svg
new file mode 100644
index 0000000000..5f8cceec28
--- /dev/null
+++ b/images/breeze/output-breeze-commands.svg
@@ -0,0 +1,136 @@
+<svg width="1484.0" height="626" viewBox="0 0 1484.0 626"
+ xmlns="http://www.w3.org/2000/svg">
+ <style>
+ @font-face {
+ font-family: "Fira Code";
+ src: local("FiraCode-Regular"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff2/FiraCode-Regular.woff2") format("woff2"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff/FiraCode-Regular.woff") format("woff");
+ font-style: normal;
+ font-weight: 400;
+ }
+ @font-face {
+ font-family: "Fira Code";
+ src: local("FiraCode-Bold"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff2/FiraCode-Bold.woff2") format("woff2"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff/FiraCode-Bold.woff") format("woff");
+ font-style: bold;
+ font-weight: 700;
+ }
+ span {
+ display: inline-block;
+ white-space: pre;
+ vertical-align: top;
+ font-size: 18px;
+ font-family:'Fira Code','Cascadia Code',Monaco,Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace;
+ }
+ a {
+ text-decoration: none;
+ color: inherit;
+ }
+ .blink {
+ animation: blinker 1s infinite;
+ }
+ @keyframes blinker {
+ from { opacity: 1.0; }
+ 50% { opacity: 0.3; }
+ to { opacity: 1.0; }
+ }
+ #wrapper {
+ padding: 140px;
+ padding-top: 100px;
+ }
+ #terminal {
+ position: relative;
+ display: flex;
+ flex-direction: column;
+ align-items: center;
+ background-color: #0c0c0c;
+ border-radius: 14px;
+ outline: 1px solid #484848;
+ }
+ #terminal:after {
+ position: absolute;
+ width: 100%;
+ height: 100%;
+ content: '';
+ border-radius: 14px;
+ background: rgb(71,77,102);
+ background: linear-gradient(90deg, #804D69 0%, #4E4B89 100%);
+ transform: rotate(-4.5deg);
+ z-index: -1;
+ }
+ #terminal-header {
+ position: relative;
+ width: 100%;
+ background-color: #2e2e2e;
+ margin-bottom: 12px;
+ font-weight: bold;
+ border-radius: 14px 14px 0 0;
+ color: #f2f2f2;
+ font-size: 18px;
+ box-shadow: inset 0px -1px 0px 0px #4e4e4e,
+ inset 0px -4px 8px 0px #1a1a1a;
+ }
+ #terminal-title-tab {
+ display: inline-block;
+ margin-top: 14px;
+ margin-left: 124px;
+ font-family: sans-serif;
+ padding: 14px 28px;
+ border-radius: 6px 6px 0 0;
+ background-color: #0c0c0c;
+ box-shadow: inset 0px 1px 0px 0px #4e4e4e,
+ 0px -4px 4px 0px #1e1e1e,
+ inset 1px 0px 0px 0px #4e4e4e,
+ inset -1px 0px 0px 0px #4e4e4e;
+ }
+ #terminal-traffic-lights {
+ position: absolute;
+ top: 24px;
+ left: 20px;
+ }
+ #terminal-body {
+ line-height: 22px;
+ padding: 14px;
+ }
+ .r1 {color: #f2f2f2; text-decoration-color: #f2f2f2;background-color: #0c0c0c;}
+.r2 {font-weight: bold;color: #f2f2f2; text-decoration-color: #f2f2f2;;background-color: #0c0c0c;}
+.r3 {color: #e5e510; text-decoration-color: #e5e510; font-weight: bold;background-color: #0c0c0c;}
+.r4 {color: #7f7f7f; text-decoration-color: #7f7f7f;color: #f2f2f2; text-decoration-color: #f2f2f2;;background-color: #0c0c0c;}
+.r5 {color: #11a8cd; text-decoration-color: #11a8cd; font-weight: bold;background-color: #0c0c0c;}
+.r6 {color: #0dbc79; text-decoration-color: #0dbc79; font-weight: bold;background-color: #0c0c0c;}
+.r7 {color: #78780e; text-decoration-color: #78780e;background-color: #0c0c0c;}
+ </style>
+ <foreignObject x="0" y="0" width="100%" height="100%">
+ <body xmlns="http://www.w3.org/1999/xhtml">
+ <div id="wrapper">
+ <div id="terminal">
+ <div id='terminal-header'>
+ <svg id="terminal-traffic-lights" width="90" height="21" viewBox="0 0 90 21" xmlns="http://www.w3.org/2000/svg">
+ <circle cx="14" cy="8" r="8" fill="#ff6159"/>
+ <circle cx="38" cy="8" r="8" fill="#ffbd2e"/>
+ <circle cx="62" cy="8" r="8" fill="#28c941"/>
+ </svg>
+ <div id="terminal-title-tab">Breeze screenshot</div>
+ </div>
+ <div id='terminal-body'>
+ <div><span class="r2"> </span></div>
+<div><span class="r2"> </span><span class="r3">Usage: </span><span class="r2">breeze cleanup [OPTIONS] </span></div>
+<div><span class="r2"> </span></div>
+<div><span class="r1"> </span><span class="r1">Removes the cache of parameters, docker cache and optionally - currently downloaded images. </span><span class="r1"> </span></div>
+<div><span class="r1"> </span></div>
+<div><span class="r4">╭─ Options ────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--also-remove-current-images</span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r1">Also removes currently downloaded Breeze images. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y|n|q)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
+<div><span class="r1"></span><span class="r1"> </span></div>
+ </div>
+ </div>
+ </div>
+ </body>
+ </foreignObject>
+</svg>
diff --git a/images/breeze/output-build-docs.svg b/images/breeze/output-build-docs.svg
new file mode 100644
index 0000000000..085615ef40
--- /dev/null
+++ b/images/breeze/output-build-docs.svg
@@ -0,0 +1,168 @@
+<svg width="1720.0" height="1330" viewBox="0 0 1720.0 1330"
+ xmlns="http://www.w3.org/2000/svg">
+ <style>
+ @font-face {
+ font-family: "Fira Code";
+ src: local("FiraCode-Regular"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff2/FiraCode-Regular.woff2") format("woff2"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff/FiraCode-Regular.woff") format("woff");
+ font-style: normal;
+ font-weight: 400;
+ }
+ @font-face {
+ font-family: "Fira Code";
+ src: local("FiraCode-Bold"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff2/FiraCode-Bold.woff2") format("woff2"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff/FiraCode-Bold.woff") format("woff");
+ font-style: bold;
+ font-weight: 700;
+ }
+ span {
+ display: inline-block;
+ white-space: pre;
+ vertical-align: top;
+ font-size: 18px;
+ font-family:'Fira Code','Cascadia Code',Monaco,Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace;
+ }
+ a {
+ text-decoration: none;
+ color: inherit;
+ }
+ .blink {
+ animation: blinker 1s infinite;
+ }
+ @keyframes blinker {
+ from { opacity: 1.0; }
+ 50% { opacity: 0.3; }
+ to { opacity: 1.0; }
+ }
+ #wrapper {
+ padding: 140px;
+ padding-top: 100px;
+ }
+ #terminal {
+ position: relative;
+ display: flex;
+ flex-direction: column;
+ align-items: center;
+ background-color: #0c0c0c;
+ border-radius: 14px;
+ outline: 1px solid #484848;
+ }
+ #terminal:after {
+ position: absolute;
+ width: 100%;
+ height: 100%;
+ content: '';
+ border-radius: 14px;
+ background: rgb(71,77,102);
+ background: linear-gradient(90deg, #804D69 0%, #4E4B89 100%);
+ transform: rotate(-4.5deg);
+ z-index: -1;
+ }
+ #terminal-header {
+ position: relative;
+ width: 100%;
+ background-color: #2e2e2e;
+ margin-bottom: 12px;
+ font-weight: bold;
+ border-radius: 14px 14px 0 0;
+ color: #f2f2f2;
+ font-size: 18px;
+ box-shadow: inset 0px -1px 0px 0px #4e4e4e,
+ inset 0px -4px 8px 0px #1a1a1a;
+ }
+ #terminal-title-tab {
+ display: inline-block;
+ margin-top: 14px;
+ margin-left: 124px;
+ font-family: sans-serif;
+ padding: 14px 28px;
+ border-radius: 6px 6px 0 0;
+ background-color: #0c0c0c;
+ box-shadow: inset 0px 1px 0px 0px #4e4e4e,
+ 0px -4px 4px 0px #1e1e1e,
+ inset 1px 0px 0px 0px #4e4e4e,
+ inset -1px 0px 0px 0px #4e4e4e;
+ }
+ #terminal-traffic-lights {
+ position: absolute;
+ top: 24px;
+ left: 20px;
+ }
+ #terminal-body {
+ line-height: 22px;
+ padding: 14px;
+ }
+ .r1 {color: #f2f2f2; text-decoration-color: #f2f2f2;background-color: #0c0c0c;}
+.r2 {font-weight: bold;color: #f2f2f2; text-decoration-color: #f2f2f2;;background-color: #0c0c0c;}
+.r3 {color: #e5e510; text-decoration-color: #e5e510; font-weight: bold;background-color: #0c0c0c;}
+.r4 {color: #7f7f7f; text-decoration-color: #7f7f7f;color: #f2f2f2; text-decoration-color: #f2f2f2;;background-color: #0c0c0c;}
+.r5 {color: #11a8cd; text-decoration-color: #11a8cd; font-weight: bold;background-color: #0c0c0c;}
+.r6 {color: #0dbc79; text-decoration-color: #0dbc79; font-weight: bold;background-color: #0c0c0c;}
+.r7 {color: #78780e; text-decoration-color: #78780e;background-color: #0c0c0c;}
+ </style>
+ <foreignObject x="0" y="0" width="100%" height="100%">
+ <body xmlns="http://www.w3.org/1999/xhtml">
+ <div id="wrapper">
+ <div id="terminal">
+ <div id='terminal-header'>
+ <svg id="terminal-traffic-lights" width="90" height="21" viewBox="0 0 90 21" xmlns="http://www.w3.org/2000/svg">
+ <circle cx="14" cy="8" r="8" fill="#ff6159"/>
+ <circle cx="38" cy="8" r="8" fill="#ffbd2e"/>
+ <circle cx="62" cy="8" r="8" fill="#28c941"/>
+ </svg>
+ <div id="terminal-title-tab">Command: build-docs</div>
+ </div>
+ <div id='terminal-body'>
+ <div><span class="r2"> </span></div>
+<div><span class="r2"> </span><span class="r3">Usage: </span><span class="r2">breeze build-docs [OPTIONS] </span></div>
+<div><span class="r2"> </span></div>
+<div><span class="r1"> </span><span class="r1">Build documentation in the container. </span><span class="r1"> </span></div>
+<div><span class="r1"> </span></div>
+<div><span class="r4">╭─ Doc flags ──────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--docs-only</span><span class="r1"> </span><span class="r6">-d</span><span class="r1"> </span><span class="r1">Only build documentation. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--spellcheck-only</span><span class="r1"> </span><span class="r6">-s</span><span class="r1"> </span><span class="r1">Only run spell checking. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--package-filter</span><span class="r1"> </span><span class="r6">-p</span><span class="r1"> </span><span class="r1">List of packages to consider. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">(apache-airflow|apache-airflow-providers|apache-airflow-providers-airbyte|apache-airflow-pr</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">oviders-alibaba|apache-airflow-providers-amazon|apache-airflow-providers-apache-beam|apache</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">-airflow-providers-apache-cassandra|apache-airflow-providers-apache-drill|apache-airflow-pr</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">oviders-apache-druid|apache-airflow-providers-apache-hdfs|apache-airflow-providers-apache-h</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">ive|apache-airflow-providers-apache-kylin|apache-airflow-providers-apache-livy|apache-airfl</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">ow-providers-apache-pig|apache-airflow-providers-apache-pinot|apache-airflow-providers-apac</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">he-spark|apache-airflow-providers-apache-sqoop|apache-airflow-providers-arangodb|apache-air</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">flow-providers-asana|apache-airflow-providers-celery|apache-airflow-providers-cloudant|apac</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">he-airflow-providers-cncf-kubernetes|apache-airflow-providers-databricks|apache-airflow-pro</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">viders-datadog|apache-airflow-providers-dbt-cloud|apache-airflow-providers-dingding|apache-</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">airflow-providers-discord|apache-airflow-providers-docker|apache-airflow-providers-elastics</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">earch|apache-airflow-providers-exasol|apache-airflow-providers-facebook|apache-airflow-prov</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">iders-ftp|apache-airflow-providers-github|apache-airflow-providers-google|apache-airflow-pr</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">oviders-grpc|apache-airflow-providers-hashicorp|apache-airflow-providers-http|apache-airflo</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">w-providers-imap|apache-airflow-providers-influxdb|apache-airflow-providers-jdbc|apache-air</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">flow-providers-jenkins|apache-airflow-providers-jira|apache-airflow-providers-microsoft-azu</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">re|apache-airflow-providers-microsoft-mssql|apache-airflow-providers-microsoft-psrp|apache-</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">airflow-providers-microsoft-winrm|apache-airflow-providers-mongo|apache-airflow-providers-m</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">ysql|apache-airflow-providers-neo4j|apache-airflow-providers-odbc|apache-airflow-providers-</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">openfaas|apache-airflow-providers-opsgenie|apache-airflow-providers-oracle|apache-airflow-p</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">roviders-pagerduty|apache-airflow-providers-papermill|apache-airflow-providers-plexus|apach</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">e-airflow-providers-postgres|apache-airflow-providers-presto|apache-airflow-providers-qubol</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">e|apache-airflow-providers-redis|apache-airflow-providers-salesforce|apache-airflow-provide</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">rs-samba|apache-airflow-providers-segment|apache-airflow-providers-sendgrid|apache-airflow-</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">providers-sftp|apache-airflow-providers-singularity|apache-airflow-providers-slack|apache-a</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">irflow-providers-snowflake|apache-airflow-providers-sqlite|apache-airflow-providers-ssh|apa</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">che-airflow-providers-tableau|apache-airflow-providers-telegram|apache-airflow-providers-tr</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">ino|apache-airflow-providers-vertica|apache-airflow-providers-yandex|apache-airflow-provide</span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">rs-zendesk|docker-stack|helm-chart) </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
+<div><span class="r4">╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
+<div><span class="r1"></span><span class="r1"> </span></div>
+ </div>
+ </div>
+ </div>
+ </body>
+ </foreignObject>
+</svg>
diff --git a/images/breeze/output-build-image.svg b/images/breeze/output-build-image.svg
new file mode 100644
index 0000000000..5cb1a030c7
--- /dev/null
+++ b/images/breeze/output-build-image.svg
@@ -0,0 +1,162 @@
+<svg width="1720.0" height="1198" viewBox="0 0 1720.0 1198"
+ xmlns="http://www.w3.org/2000/svg">
+ <style>
+ @font-face {
+ font-family: "Fira Code";
+ src: local("FiraCode-Regular"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff2/FiraCode-Regular.woff2") format("woff2"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff/FiraCode-Regular.woff") format("woff");
+ font-style: normal;
+ font-weight: 400;
+ }
+ @font-face {
+ font-family: "Fira Code";
+ src: local("FiraCode-Bold"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff2/FiraCode-Bold.woff2") format("woff2"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff/FiraCode-Bold.woff") format("woff");
+ font-style: bold;
+ font-weight: 700;
+ }
+ span {
+ display: inline-block;
+ white-space: pre;
+ vertical-align: top;
+ font-size: 18px;
+ font-family:'Fira Code','Cascadia Code',Monaco,Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace;
+ }
+ a {
+ text-decoration: none;
+ color: inherit;
+ }
+ .blink {
+ animation: blinker 1s infinite;
+ }
+ @keyframes blinker {
+ from { opacity: 1.0; }
+ 50% { opacity: 0.3; }
+ to { opacity: 1.0; }
+ }
+ #wrapper {
+ padding: 140px;
+ padding-top: 100px;
+ }
+ #terminal {
+ position: relative;
+ display: flex;
+ flex-direction: column;
+ align-items: center;
+ background-color: #0c0c0c;
+ border-radius: 14px;
+ outline: 1px solid #484848;
+ }
+ #terminal:after {
+ position: absolute;
+ width: 100%;
+ height: 100%;
+ content: '';
+ border-radius: 14px;
+ background: rgb(71,77,102);
+ background: linear-gradient(90deg, #804D69 0%, #4E4B89 100%);
+ transform: rotate(-4.5deg);
+ z-index: -1;
+ }
+ #terminal-header {
+ position: relative;
+ width: 100%;
+ background-color: #2e2e2e;
+ margin-bottom: 12px;
+ font-weight: bold;
+ border-radius: 14px 14px 0 0;
+ color: #f2f2f2;
+ font-size: 18px;
+ box-shadow: inset 0px -1px 0px 0px #4e4e4e,
+ inset 0px -4px 8px 0px #1a1a1a;
+ }
+ #terminal-title-tab {
+ display: inline-block;
+ margin-top: 14px;
+ margin-left: 124px;
+ font-family: sans-serif;
+ padding: 14px 28px;
+ border-radius: 6px 6px 0 0;
+ background-color: #0c0c0c;
+ box-shadow: inset 0px 1px 0px 0px #4e4e4e,
+ 0px -4px 4px 0px #1e1e1e,
+ inset 1px 0px 0px 0px #4e4e4e,
+ inset -1px 0px 0px 0px #4e4e4e;
+ }
+ #terminal-traffic-lights {
+ position: absolute;
+ top: 24px;
+ left: 20px;
+ }
+ #terminal-body {
+ line-height: 22px;
+ padding: 14px;
+ }
+ .r1 {color: #f2f2f2; text-decoration-color: #f2f2f2;background-color: #0c0c0c;}
+.r2 {font-weight: bold;color: #f2f2f2; text-decoration-color: #f2f2f2;;background-color: #0c0c0c;}
+.r3 {color: #e5e510; text-decoration-color: #e5e510; font-weight: bold;background-color: #0c0c0c;}
+.r4 {color: #7f7f7f; text-decoration-color: #7f7f7f;color: #f2f2f2; text-decoration-color: #f2f2f2;;background-color: #0c0c0c;}
+.r5 {color: #11a8cd; text-decoration-color: #11a8cd; font-weight: bold;background-color: #0c0c0c;}
+.r6 {color: #0dbc79; text-decoration-color: #0dbc79; font-weight: bold;background-color: #0c0c0c;}
+.r7 {color: #78780e; text-decoration-color: #78780e;background-color: #0c0c0c;}
+ </style>
+ <foreignObject x="0" y="0" width="100%" height="100%">
+ <body xmlns="http://www.w3.org/1999/xhtml">
+ <div id="wrapper">
+ <div id="terminal">
+ <div id='terminal-header'>
+ <svg id="terminal-traffic-lights" width="90" height="21" viewBox="0 0 90 21" xmlns="http://www.w3.org/2000/svg">
+ <circle cx="14" cy="8" r="8" fill="#ff6159"/>
+ <circle cx="38" cy="8" r="8" fill="#ffbd2e"/>
+ <circle cx="62" cy="8" r="8" fill="#28c941"/>
+ </svg>
+ <div id="terminal-title-tab">Command: build-image</div>
+ </div>
+ <div id='terminal-body'>
+ <div><span class="r2"> </span></div>
+<div><span class="r2"> </span><span class="r3">Usage: </span><span class="r2">breeze build-image [OPTIONS] </span></div>
+<div><span class="r2"> </span></div>
+<div><span class="r1"> </span><span class="r1">Build CI image. </span><span class="r1"> </span></div>
+<div><span class="r1"> </span></div>
+<div><span class="r4">╭─ Basic usage ────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--python</span><span class="r1"> </span><span class="r6">-p</span><span class="r1"> </span><span class="r1">Python version to use.</span><span class="r1"> </span><span class="r7">(3.7|3.8|3.9|3.10)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--upgrade-to-newer-dependencies</span><span class="r1"> </span><span class="r6">-u</span><span class="r1"> </span><span class="r1">When other than "false", upgrade all PIP packages to latest.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--debian-version</span><span class="r1"> </span><span class="r6">-d</span><span class="r1"> </span><span class="r1">Debian version used for the image.</span><span class="r1"> </span><span class="r7">(buster|bullseye)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--image-tag</span><span class="r1"> </span><span class="r6">-t</span><span class="r1"> </span><span class="r1">Set tag for the image (additionally to default Airflow convention).</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--docker-cache</span><span class="r1"> </span><span class="r6">-c</span><span class="r1"> </span><span class="r1">Cache option for image used during the build.</span><span class="r1"> </span><span class="r7">(pulled|local|disabled)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--github-repository</span><span class="r1"> </span><span class="r6">-g</span><span class="r1"> </span><span class="r1">GitHub repository used to pull, push images. Default: apache/airflow.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
+<div><span class="r4">╭─ Advanced options (for power users) ─────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--install-providers-from-sources</span><span class="r1"> </span><span class="r1">Install providers from sources when installing. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-extras</span><span class="r1"> </span><span class="r1">Additional extra package while installing Airflow in the image.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-dev-apt-deps</span><span class="r1"> </span><span class="r1">Additional apt dev dependencies to use when building the images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-runtime-apt-deps</span><span class="r1"> </span><span class="r1">Additional apt runtime dependencies to use when building the images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-python-deps</span><span class="r1"> </span><span class="r1">Additional python dependencies to use when building the images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-dev-apt-command</span><span class="r1"> </span><span class="r1">Additional command executed before dev apt deps are installed.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--runtime-apt-command</span><span class="r1"> </span><span class="r1">Command executed before runtime apt deps are installed.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-dev-apt-env</span><span class="r1"> </span><span class="r1">Additional environment variables set when adding dev dependencies.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-runtime-apt-env</span><span class="r1"> </span><span class="r1">Additional environment variables set when adding runtime dependencies.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-runtime-apt-command</span><span class="r1"> </span><span class="r1">Additional command executed before runtime apt deps are installed.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dev-apt-command</span><span class="r1"> </span><span class="r1">Command executed before dev apt deps are installed.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dev-apt-deps</span><span class="r1"> </span><span class="r1">Apt dev dependencies to use when building the images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--runtime-apt-deps</span><span class="r1"> </span><span class="r1">Apt runtime dependencies to use when building the images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
+<div><span class="r4">╭─ Preparing cache (for maintainers) ──────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--platform</span><span class="r1"> </span><span class="r1">Platform for Airflow image.</span><span class="r1"> </span><span class="r7">(linux/amd64|linux/arm64|linux/amd64,linux/arm64)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--prepare-buildx-cache</span><span class="r1"> </span><span class="r1">Prepares build cache rather than build images locally. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
+<div><span class="r4">╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y|n|q|yes|no|quit)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
+<div><span class="r1"></span><span class="r1"> </span></div>
+ </div>
+ </div>
+ </div>
+ </body>
+ </foreignObject>
+</svg>
diff --git a/images/breeze/output-build-prod-image.svg b/images/breeze/output-build-prod-image.svg
new file mode 100644
index 0000000000..625ae140f8
--- /dev/null
+++ b/images/breeze/output-build-prod-image.svg
@@ -0,0 +1,175 @@
+<svg width="1720.0" height="1484" viewBox="0 0 1720.0 1484"
+ xmlns="http://www.w3.org/2000/svg">
+ <style>
+ @font-face {
+ font-family: "Fira Code";
+ src: local("FiraCode-Regular"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff2/FiraCode-Regular.woff2") format("woff2"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff/FiraCode-Regular.woff") format("woff");
+ font-style: normal;
+ font-weight: 400;
+ }
+ @font-face {
+ font-family: "Fira Code";
+ src: local("FiraCode-Bold"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff2/FiraCode-Bold.woff2") format("woff2"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff/FiraCode-Bold.woff") format("woff");
+ font-style: bold;
+ font-weight: 700;
+ }
+ span {
+ display: inline-block;
+ white-space: pre;
+ vertical-align: top;
+ font-size: 18px;
+ font-family:'Fira Code','Cascadia Code',Monaco,Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace;
+ }
+ a {
+ text-decoration: none;
+ color: inherit;
+ }
+ .blink {
+ animation: blinker 1s infinite;
+ }
+ @keyframes blinker {
+ from { opacity: 1.0; }
+ 50% { opacity: 0.3; }
+ to { opacity: 1.0; }
+ }
+ #wrapper {
+ padding: 140px;
+ padding-top: 100px;
+ }
+ #terminal {
+ position: relative;
+ display: flex;
+ flex-direction: column;
+ align-items: center;
+ background-color: #0c0c0c;
+ border-radius: 14px;
+ outline: 1px solid #484848;
+ }
+ #terminal:after {
+ position: absolute;
+ width: 100%;
+ height: 100%;
+ content: '';
+ border-radius: 14px;
+ background: rgb(71,77,102);
+ background: linear-gradient(90deg, #804D69 0%, #4E4B89 100%);
+ transform: rotate(-4.5deg);
+ z-index: -1;
+ }
+ #terminal-header {
+ position: relative;
+ width: 100%;
+ background-color: #2e2e2e;
+ margin-bottom: 12px;
+ font-weight: bold;
+ border-radius: 14px 14px 0 0;
+ color: #f2f2f2;
+ font-size: 18px;
+ box-shadow: inset 0px -1px 0px 0px #4e4e4e,
+ inset 0px -4px 8px 0px #1a1a1a;
+ }
+ #terminal-title-tab {
+ display: inline-block;
+ margin-top: 14px;
+ margin-left: 124px;
+ font-family: sans-serif;
+ padding: 14px 28px;
+ border-radius: 6px 6px 0 0;
+ background-color: #0c0c0c;
+ box-shadow: inset 0px 1px 0px 0px #4e4e4e,
+ 0px -4px 4px 0px #1e1e1e,
+ inset 1px 0px 0px 0px #4e4e4e,
+ inset -1px 0px 0px 0px #4e4e4e;
+ }
+ #terminal-traffic-lights {
+ position: absolute;
+ top: 24px;
+ left: 20px;
+ }
+ #terminal-body {
+ line-height: 22px;
+ padding: 14px;
+ }
+ .r1 {color: #f2f2f2; text-decoration-color: #f2f2f2;background-color: #0c0c0c;}
+.r2 {font-weight: bold;color: #f2f2f2; text-decoration-color: #f2f2f2;;background-color: #0c0c0c;}
+.r3 {color: #e5e510; text-decoration-color: #e5e510; font-weight: bold;background-color: #0c0c0c;}
+.r4 {color: #7f7f7f; text-decoration-color: #7f7f7f;color: #f2f2f2; text-decoration-color: #f2f2f2;;background-color: #0c0c0c;}
+.r5 {color: #11a8cd; text-decoration-color: #11a8cd; font-weight: bold;background-color: #0c0c0c;}
+.r6 {color: #0dbc79; text-decoration-color: #0dbc79; font-weight: bold;background-color: #0c0c0c;}
+.r7 {color: #78780e; text-decoration-color: #78780e;background-color: #0c0c0c;}
+ </style>
+ <foreignObject x="0" y="0" width="100%" height="100%">
+ <body xmlns="http://www.w3.org/1999/xhtml">
+ <div id="wrapper">
+ <div id="terminal">
+ <div id='terminal-header'>
+ <svg id="terminal-traffic-lights" width="90" height="21" viewBox="0 0 90 21" xmlns="http://www.w3.org/2000/svg">
+ <circle cx="14" cy="8" r="8" fill="#ff6159"/>
+ <circle cx="38" cy="8" r="8" fill="#ffbd2e"/>
+ <circle cx="62" cy="8" r="8" fill="#28c941"/>
+ </svg>
+ <div id="terminal-title-tab">Command: build-prod-image</div>
+ </div>
+ <div id='terminal-body'>
+ <div><span class="r2"> </span></div>
+<div><span class="r2"> </span><span class="r3">Usage: </span><span class="r2">breeze build-prod-image [OPTIONS] </span></div>
+<div><span class="r2"> </span></div>
+<div><span class="r1"> </span><span class="r1">Build Production image. </span><span class="r1"> </span></div>
+<div><span class="r1"> </span></div>
+<div><span class="r4">╭─ Basic usage ────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--python</span><span class="r1"> </span><span class="r6">-p</span><span class="r1"> </span><span class="r1">Python version to use.</span><span class="r1"> </span><span class="r7">(3.7|3.8|3.9|3.10)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--install-airflow-version</span><span class="r1"> </span><span class="r6">-V</span><span class="r1"> </span><span class="r1">Install version of Airflow from PyPI.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--upgrade-to-newer-dependencies</span><span class="r1"> </span><span class="r6">-u</span><span class="r1"> </span><span class="r1">When other than "false", upgrade all PIP packages to latest.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--debian-version</span><span class="r1"> </span><span class="r6">-d</span><span class="r1"> </span><span class="r1">Debian version used for the image.</span><span class="r1"> </span><span class="r7">(buster|bullseye)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--image-tag</span><span class="r1"> </span><span class="r6">-t</span><span class="r1"> </span><span class="r1">Set tag for the image (additionally to default Airflow convention).</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--docker-cache</span><span class="r1"> </span><span class="r6">-c</span><span class="r1"> </span><span class="r1">Cache option for image used during the build.</span><span class="r1"> </span><span class="r7">(pulled|local|disabled)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--github-repository</span><span class="r1"> </span><span class="r6">-g</span><span class="r1"> </span><span class="r1">GitHub repository used to pull, push images. Default: apache/airflow.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
+<div><span class="r4">╭─ Options for customizing images ─────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--install-providers-from-sources</span><span class="r1"> </span><span class="r1">Install providers from sources when installing. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--extras</span><span class="r1"> </span><span class="r1">Extras to install by default.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-extras</span><span class="r1"> </span><span class="r1">Additional extra package while installing Airflow in the image.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-dev-apt-deps</span><span class="r1"> </span><span class="r1">Additional apt dev dependencies to use when building the images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-runtime-apt-deps</span><span class="r1"> </span><span class="r1">Additional apt runtime dependencies to use when building the images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-python-deps</span><span class="r1"> </span><span class="r1">Additional python dependencies to use when building the images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-dev-apt-command</span><span class="r1"> </span><span class="r1">Additional command executed before dev apt deps are installed.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--runtime-apt-command</span><span class="r1"> </span><span class="r1">Command executed before runtime apt deps are installed.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-dev-apt-env</span><span class="r1"> </span><span class="r1">Additional environment variables set when adding dev dependencies.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-runtime-apt-env</span><span class="r1"> </span><span class="r1">Additional environment variables set when adding runtime dependencies.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--additional-runtime-apt-command</span><span class="r1"> </span><span class="r1">Additional command executed before runtime apt deps are installed.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dev-apt-command</span><span class="r1"> </span><span class="r1">Command executed before dev apt deps are installed.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dev-apt-deps</span><span class="r1"> </span><span class="r1">Apt dev dependencies to use when building the images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--runtime-apt-deps</span><span class="r1"> </span><span class="r1">Apt runtime dependencies to use when building the images.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
+<div><span class="r4">╭─ Customization options (for specific customization needs) ───────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--install-from-docker-context-files</span><span class="r1"> </span><span class="r1">Install wheels from local docker-context-files when building image. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--cleanup-docker-context-files</span><span class="r1"> </span><span class="r1">Clean up docker context files before running build. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--disable-mysql-client-installation</span><span class="r1"> </span><span class="r1">Do not install MySQL client. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--disable-mssql-client-installation</span><span class="r1"> </span><span class="r1">Do not install MsSQl client. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--disable-postgres-client-installation</span><span class="r1"> </span><span class="r1">Do not install Postgres client. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--disable-airflow-repo-cache</span><span class="r1"> </span><span class="r1">Disable cache from Airflow repository during building. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--disable-pypi</span><span class="r1"> </span><span class="r1">Disable PyPI during building. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--install-airflow-reference</span><span class="r1"> </span><span class="r1">Install Airflow using GitHub tag or branch.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--installation-method</span><span class="r1"> </span><span class="r1">Install Airflow from: sources or PyPI.</span><span class="r1"> </span><span class="r7">(.|apache-airflow)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
+<div><span class="r4">╭─ Preparing cache (for maintainers) ──────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--platform</span><span class="r1"> </span><span class="r1">Platform for Airflow image.</span><span class="r1"> </span><span class="r7">(linux/amd64|linux/arm64|linux/amd64,linux/arm64)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--prepare-buildx-cache</span><span class="r1"> </span><span class="r1">Prepares build cache rather than build images locally. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
+<div><span class="r4">╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y|n|q|yes|no|quit)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
+<div><span class="r1"></span><span class="r1"> </span></div>
+ </div>
+ </div>
+ </div>
+ </body>
+ </foreignObject>
+</svg>
diff --git a/images/breeze/output-cleanup.svg b/images/breeze/output-cleanup.svg
new file mode 100644
index 0000000000..4da7923766
--- /dev/null
+++ b/images/breeze/output-cleanup.svg
@@ -0,0 +1,136 @@
+<svg width="1720.0" height="626" viewBox="0 0 1720.0 626"
+ xmlns="http://www.w3.org/2000/svg">
+ <style>
+ @font-face {
+ font-family: "Fira Code";
+ src: local("FiraCode-Regular"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff2/FiraCode-Regular.woff2") format("woff2"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff/FiraCode-Regular.woff") format("woff");
+ font-style: normal;
+ font-weight: 400;
+ }
+ @font-face {
+ font-family: "Fira Code";
+ src: local("FiraCode-Bold"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff2/FiraCode-Bold.woff2") format("woff2"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff/FiraCode-Bold.woff") format("woff");
+ font-style: bold;
+ font-weight: 700;
+ }
+ span {
+ display: inline-block;
+ white-space: pre;
+ vertical-align: top;
+ font-size: 18px;
+ font-family:'Fira Code','Cascadia Code',Monaco,Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace;
+ }
+ a {
+ text-decoration: none;
+ color: inherit;
+ }
+ .blink {
+ animation: blinker 1s infinite;
+ }
+ @keyframes blinker {
+ from { opacity: 1.0; }
+ 50% { opacity: 0.3; }
+ to { opacity: 1.0; }
+ }
+ #wrapper {
+ padding: 140px;
+ padding-top: 100px;
+ }
+ #terminal {
+ position: relative;
+ display: flex;
+ flex-direction: column;
+ align-items: center;
+ background-color: #0c0c0c;
+ border-radius: 14px;
+ outline: 1px solid #484848;
+ }
+ #terminal:after {
+ position: absolute;
+ width: 100%;
+ height: 100%;
+ content: '';
+ border-radius: 14px;
+ background: rgb(71,77,102);
+ background: linear-gradient(90deg, #804D69 0%, #4E4B89 100%);
+ transform: rotate(-4.5deg);
+ z-index: -1;
+ }
+ #terminal-header {
+ position: relative;
+ width: 100%;
+ background-color: #2e2e2e;
+ margin-bottom: 12px;
+ font-weight: bold;
+ border-radius: 14px 14px 0 0;
+ color: #f2f2f2;
+ font-size: 18px;
+ box-shadow: inset 0px -1px 0px 0px #4e4e4e,
+ inset 0px -4px 8px 0px #1a1a1a;
+ }
+ #terminal-title-tab {
+ display: inline-block;
+ margin-top: 14px;
+ margin-left: 124px;
+ font-family: sans-serif;
+ padding: 14px 28px;
+ border-radius: 6px 6px 0 0;
+ background-color: #0c0c0c;
+ box-shadow: inset 0px 1px 0px 0px #4e4e4e,
+ 0px -4px 4px 0px #1e1e1e,
+ inset 1px 0px 0px 0px #4e4e4e,
+ inset -1px 0px 0px 0px #4e4e4e;
+ }
+ #terminal-traffic-lights {
+ position: absolute;
+ top: 24px;
+ left: 20px;
+ }
+ #terminal-body {
+ line-height: 22px;
+ padding: 14px;
+ }
+ .r1 {color: #f2f2f2; text-decoration-color: #f2f2f2;background-color: #0c0c0c;}
+.r2 {font-weight: bold;color: #f2f2f2; text-decoration-color: #f2f2f2;;background-color: #0c0c0c;}
+.r3 {color: #e5e510; text-decoration-color: #e5e510; font-weight: bold;background-color: #0c0c0c;}
+.r4 {color: #7f7f7f; text-decoration-color: #7f7f7f;color: #f2f2f2; text-decoration-color: #f2f2f2;;background-color: #0c0c0c;}
+.r5 {color: #11a8cd; text-decoration-color: #11a8cd; font-weight: bold;background-color: #0c0c0c;}
+.r6 {color: #0dbc79; text-decoration-color: #0dbc79; font-weight: bold;background-color: #0c0c0c;}
+.r7 {color: #78780e; text-decoration-color: #78780e;background-color: #0c0c0c;}
+ </style>
+ <foreignObject x="0" y="0" width="100%" height="100%">
+ <body xmlns="http://www.w3.org/1999/xhtml">
+ <div id="wrapper">
+ <div id="terminal">
+ <div id='terminal-header'>
+ <svg id="terminal-traffic-lights" width="90" height="21" viewBox="0 0 90 21" xmlns="http://www.w3.org/2000/svg">
+ <circle cx="14" cy="8" r="8" fill="#ff6159"/>
+ <circle cx="38" cy="8" r="8" fill="#ffbd2e"/>
+ <circle cx="62" cy="8" r="8" fill="#28c941"/>
+ </svg>
+ <div id="terminal-title-tab">Command: cleanup</div>
+ </div>
+ <div id='terminal-body'>
+ <div><span class="r2"> </span></div>
+<div><span class="r2"> </span><span class="r3">Usage: </span><span class="r2">breeze cleanup [OPTIONS] </span></div>
+<div><span class="r2"> </span></div>
+<div><span class="r1"> </span><span class="r1">the cache of parameters, docker cache and optionally - currently downloaded images. </span><span class="r1"> </span></div>
+<div><span class="r1"> </span></div>
+<div><span class="r4">╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--also-remove-current-images</span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r1">Also remove currently downloaded Breeze images. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y|n|q|yes|no|quit)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
+<div><span class="r1"></span><span class="r1"> </span></div>
+ </div>
+ </div>
+ </div>
+ </body>
+ </foreignObject>
+</svg>
diff --git a/images/breeze/output-commands.svg b/images/breeze/output-commands.svg
new file mode 100644
index 0000000000..cd83f79e46
--- /dev/null
+++ b/images/breeze/output-commands.svg
@@ -0,0 +1,165 @@
+<svg width="1720.0" height="1264" viewBox="0 0 1720.0 1264"
+ xmlns="http://www.w3.org/2000/svg">
+ <style>
+ @font-face {
+ font-family: "Fira Code";
+ src: local("FiraCode-Regular"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff2/FiraCode-Regular.woff2") format("woff2"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff/FiraCode-Regular.woff") format("woff");
+ font-style: normal;
+ font-weight: 400;
+ }
+ @font-face {
+ font-family: "Fira Code";
+ src: local("FiraCode-Bold"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff2/FiraCode-Bold.woff2") format("woff2"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff/FiraCode-Bold.woff") format("woff");
+ font-style: bold;
+ font-weight: 700;
+ }
+ span {
+ display: inline-block;
+ white-space: pre;
+ vertical-align: top;
+ font-size: 18px;
+ font-family:'Fira Code','Cascadia Code',Monaco,Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace;
+ }
+ a {
+ text-decoration: none;
+ color: inherit;
+ }
+ .blink {
+ animation: blinker 1s infinite;
+ }
+ @keyframes blinker {
+ from { opacity: 1.0; }
+ 50% { opacity: 0.3; }
+ to { opacity: 1.0; }
+ }
+ #wrapper {
+ padding: 140px;
+ padding-top: 100px;
+ }
+ #terminal {
+ position: relative;
+ display: flex;
+ flex-direction: column;
+ align-items: center;
+ background-color: #0c0c0c;
+ border-radius: 14px;
+ outline: 1px solid #484848;
+ }
+ #terminal:after {
+ position: absolute;
+ width: 100%;
+ height: 100%;
+ content: '';
+ border-radius: 14px;
+ background: rgb(71,77,102);
+ background: linear-gradient(90deg, #804D69 0%, #4E4B89 100%);
+ transform: rotate(-4.5deg);
+ z-index: -1;
+ }
+ #terminal-header {
+ position: relative;
+ width: 100%;
+ background-color: #2e2e2e;
+ margin-bottom: 12px;
+ font-weight: bold;
+ border-radius: 14px 14px 0 0;
+ color: #f2f2f2;
+ font-size: 18px;
+ box-shadow: inset 0px -1px 0px 0px #4e4e4e,
+ inset 0px -4px 8px 0px #1a1a1a;
+ }
+ #terminal-title-tab {
+ display: inline-block;
+ margin-top: 14px;
+ margin-left: 124px;
+ font-family: sans-serif;
+ padding: 14px 28px;
+ border-radius: 6px 6px 0 0;
+ background-color: #0c0c0c;
+ box-shadow: inset 0px 1px 0px 0px #4e4e4e,
+ 0px -4px 4px 0px #1e1e1e,
+ inset 1px 0px 0px 0px #4e4e4e,
+ inset -1px 0px 0px 0px #4e4e4e;
+ }
+ #terminal-traffic-lights {
+ position: absolute;
+ top: 24px;
+ left: 20px;
+ }
+ #terminal-body {
+ line-height: 22px;
+ padding: 14px;
+ }
+ .r1 {color: #f2f2f2; text-decoration-color: #f2f2f2;background-color: #0c0c0c;}
+.r2 {font-weight: bold;color: #f2f2f2; text-decoration-color: #f2f2f2;;background-color: #0c0c0c;}
+.r3 {color: #e5e510; text-decoration-color: #e5e510; font-weight: bold;background-color: #0c0c0c;}
+.r4 {color: #7f7f7f; text-decoration-color: #7f7f7f;color: #f2f2f2; text-decoration-color: #f2f2f2;;background-color: #0c0c0c;}
+.r5 {color: #11a8cd; text-decoration-color: #11a8cd; font-weight: bold;background-color: #0c0c0c;}
+.r6 {color: #0dbc79; text-decoration-color: #0dbc79; font-weight: bold;background-color: #0c0c0c;}
+.r7 {color: #78780e; text-decoration-color: #78780e;background-color: #0c0c0c;}
+ </style>
+ <foreignObject x="0" y="0" width="100%" height="100%">
+ <body xmlns="http://www.w3.org/1999/xhtml">
+ <div id="wrapper">
+ <div id="terminal">
+ <div id='terminal-header'>
+ <svg id="terminal-traffic-lights" width="90" height="21" viewBox="0 0 90 21" xmlns="http://www.w3.org/2000/svg">
+ <circle cx="14" cy="8" r="8" fill="#ff6159"/>
+ <circle cx="38" cy="8" r="8" fill="#ffbd2e"/>
+ <circle cx="62" cy="8" r="8" fill="#28c941"/>
+ </svg>
+ <div id="terminal-title-tab">Breeze commands</div>
+ </div>
+ <div id='terminal-body'>
+ <div><span class="r2"> </span></div>
+<div><span class="r2"> </span><span class="r3">Usage: </span><span class="r2">breeze [OPTIONS] COMMAND [ARGS]... </span></div>
+<div><span class="r2"> </span></div>
+<div><span class="r4">╭─ Basic flags for the default (shell) command ────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--python</span><span class="r1"> </span><span class="r6">-p</span><span class="r1"> </span><span class="r1">Python version to use.</span><span class="r1"> </span><span class="r7">(3.7|3.8|3.9|3.10)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--backend</span><span class="r1"> </span><span class="r6">-b</span><span class="r1"> </span><span class="r1">Database backend to use.</span><span class="r1"> </span><span class="r7">(sqlite|mysql|postgres|mssql)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--use-airflow-version</span><span class="r1"> </span><span class="r6">-V</span><span class="r1"> </span><span class="r1">Use (reinstall at entry) Airflow version from PyPI.</span><span class="r1"> </span><span class="r7">(TEXT)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--postgres-version</span><span class="r1"> </span><span class="r6">-P</span><span class="r1"> </span><span class="r1">Version of Postgres.</span><span class="r1"> </span><span class="r7">(10|11|12|13)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--mysql-version</span><span class="r1"> </span><span class="r6">-M</span><span class="r1"> </span><span class="r1">Version of MySQL.</span><span class="r1"> </span><span class="r7">(5.7|8)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--mssql-version</span><span class="r1"> </span><span class="r6">-S</span><span class="r1"> </span><span class="r1">Version of MsSQL.</span><span class="r1"> </span><span class="r7">(2017-latest|2019-latest)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--forward-credentials</span><span class="r1"> </span><span class="r6">-f</span><span class="r1"> </span><span class="r1">Forward local credentials to container when running. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--db-reset</span><span class="r1"> </span><span class="r6">-d</span><span class="r1"> </span><span class="r1">Reset DB when entering the container. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
+<div><span class="r4">╭─ Advanced flags for the default (shell) command ─────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--force-build</span><span class="r1"> </span><span class="r1">Force image build before running. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--mount-sources</span><span class="r1"> </span><span class="r1">Choose scope of local sources should be mounted (default = selected).</span><span class="r1"> </span><span class="r7">(selected|all|none)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--integration</span><span class="r1"> </span><span class="r1">Integration(s) to enable when running (can be more than one). </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r7">(cassandra|kerberos|mongo|openldap|pinot|rabbitmq|redis|statsd|trino|all)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
+<div><span class="r4">╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--verbose</span><span class="r1"> </span><span class="r6">-v</span><span class="r1"> </span><span class="r1">Print verbose information about performed steps. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--dry-run</span><span class="r1"> </span><span class="r6">-D</span><span class="r1"> </span><span class="r1">If dry-run is set, commands are only printed, not executed. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--answer</span><span class="r1"> </span><span class="r6">-a</span><span class="r1"> </span><span class="r1">Force answer to questions.</span><span class="r1"> </span><span class="r7">(y|n|q|yes|no|quit)</span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5">--help</span><span class="r1"> </span><span class="r6">-h</span><span class="r1"> </span><span class="r1">Show this message and exit. </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
+<div><span class="r4">╭─ Developer tools ────────────────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> shell </span><span class="r1"> </span><span class="r1">Enter breeze.py environment. this is the default command use when no other is selected.</span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> start-airflow </span><span class="r1"> </span><span class="r1">Enter breeze.py environment and starts all Airflow components in the tmux session. </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> stop </span><span class="r1"> </span><span class="r1">Stop running breeze environment. </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> build-image </span><span class="r1"> </span><span class="r1">Build CI image. </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> build-prod-image </span><span class="r1"> </span><span class="r1">Build Production image. </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> build-docs </span><span class="r1"> </span><span class="r1">Build documentation in the container. </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> static-checks </span><span class="r1"> </span><span class="r1">Run static checks. </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
+<div><span class="r4">╭─ Configuration & maintenance ────────────────────────────────────────────────────────────────────────────────────────╮</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> cleanup </span><span class="r1"> </span><span class="r1">the cache of parameters, docker cache and optionally - currently downloaded images.</span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> self-upgrade </span><span class="r1"> </span><span class="r1">Self upgrade Breeze. </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> setup-autocomplete </span><span class="r1"> </span><span class="r1">Enables autocompletion of breeze commands. </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> config </span><span class="r1"> </span><span class="r1">Show/update configuration (Python, Backend, Cheatsheet, ASCIIART). </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">│</span><span class="r1"> </span><span class="r5"> version </span><span class="r1"> </span><span class="r1">Prints version of breeze.py. </span><span class="r1"> </span><span class="r1"> </span><span class="r1"> </span><span class="r4">│</span></div>
+<div><span class="r4">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span></div>
+<div><span class="r1"></span><span class="r1"> </span></div>
+ </div>
+ </div>
+ </div>
+ </body>
+ </foreignObject>
+</svg>
diff --git a/images/breeze/output-config.svg b/images/breeze/output-config.svg
new file mode 100644
index 0000000000..d44a208826
--- /dev/null
+++ b/images/breeze/output-config.svg
@@ -0,0 +1,138 @@
+<svg width="1720.0" height="670" viewBox="0 0 1720.0 670"
+ xmlns="http://www.w3.org/2000/svg">
+ <style>
+ @font-face {
+ font-family: "Fira Code";
+ src: local("FiraCode-Regular"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff2/FiraCode-Regular.woff2") format("woff2"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff/FiraCode-Regular.woff") format("woff");
+ font-style: normal;
+ font-weight: 400;
+ }
+ @font-face {
+ font-family: "Fira Code";
+ src: local("FiraCode-Bold"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff2/FiraCode-Bold.woff2") format("woff2"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff/FiraCode-Bold.woff") format("woff");
+ font-style: bold;
+ font-weight: 700;
+ }
+ span {
+ display: inline-block;
+ white-space: pre;
+ vertical-align: top;
+ font-size: 18px;
+ font-family:'Fira Code','Cascadia Code',Monaco,Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace;
+ }
+ a {
+ text-decoration: none;
+ color: inherit;
+ }
+ .blink {
+ animation: blinker 1s infinite;
+ }
+ @keyframes blinker {
+ from { opacity: 1.0; }
+ 50% { opacity: 0.3; }
... 1411 lines suppressed ...