You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2024/01/13 05:45:11 UTC
(airflow) branch v2-8-test updated (2527fb0e09 -> bf7fb4b019)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
discard 2527fb0e09 Update Airflow release process to include reproducible tarballs (#36744)
discard 18cbdf6364 Add support of Pendulum 3 (#36281)
discard 9a5f702e8d Add support for reproducible build date epoch for Airflow releases (#36726)
discard dfa98fc200 Standardize airflow build process and switch to Hatchling build backend (#36537)
discard c682bc4182 Update providers metadata 2024-01-10 (#36718)
discard 64834807dc Explicitly passing `raise_on_deleted_version=True` to `read_secret_version` in Hashicorp operator (#36532)
discard f3eeeca20d Get rid of pyarrow-hotfix for CVE-2023-47248 (#36697)
discard 05ebe7ad49 Add version check for k8s setup venv command (#36673)
discard bd7c6c8879 Bump min telegram-bot version (#36670)
discard def648156b Bump min provider version for papermill (#36669)
discard 93018f4c2f Bump min version of mysql-connector-python (#36668)
discard b803e3c36a Bump Cohere min version to limit backtracking on upgrade (#36666)
discard b2f3ade873 Bump min version for grpcio-status in spark provider (#36662)
discard 188d121d95 Upgrade to latest ruff and remove ISC001 warning from output (#36649)
discard e0fe4bf7bc Make `graphviz` dependency optional (#36647)
discard 8617a63261 Add Airflow logo to the README (#36645)
discard b4205c00b3 Run mypy checks for full packages in CI (#36638)
discard 4dee09153d Upgrade to latest versions of `pip` and `pipx` in CI runners (#36646)
discard 587b0a2e37 Tracking airflow-github changelog activities using progress bar (#36610)
discard b8e4dcde20 Optimise getting changelogs in airflow-github (#36548)
discard 950b08f425 Add support for openlineage to AFS and common.io (#36410)
new 8dde227637 Optimise getting changelogs in airflow-github (#36548)
new eee1cd7fcc Tracking airflow-github changelog activities using progress bar (#36610)
new a608981227 Upgrade to latest versions of `pip` and `pipx` in CI runners (#36646)
new b7485d7670 Run mypy checks for full packages in CI (#36638)
new 5fc90253f5 Add Airflow logo to the README (#36645)
new e6de1ac90c Make `graphviz` dependency optional (#36647)
new e447cae1c5 Upgrade to latest ruff and remove ISC001 warning from output (#36649)
new 3b080e0e84 Bump min version for grpcio-status in spark provider (#36662)
new ac9c4cfbac Bump Cohere min version to limit backtracking on upgrade (#36666)
new 8344febc98 Bump min version of mysql-connector-python (#36668)
new 55de7176fc Bump min provider version for papermill (#36669)
new 2d04f8a83d Bump min telegram-bot version (#36670)
new 8ab07ce525 Add version check for k8s setup venv command (#36673)
new 5690c0d17e Get rid of pyarrow-hotfix for CVE-2023-47248 (#36697)
new 2ca6d8edeb Explicitly passing `raise_on_deleted_version=True` to `read_secret_version` in Hashicorp operator (#36532)
new 4206c97ba4 Update providers metadata 2024-01-10 (#36718)
new f3d8126812 Standardize airflow build process and switch to Hatchling build backend (#36537)
new 405f8712e5 Add support for reproducible build date epoch for Airflow releases (#36726)
new 0c93669556 Add support of Pendulum 3 (#36281)
new bf7fb4b019 Update Airflow release process to include reproducible tarballs (#36744)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (2527fb0e09)
\
N -- N -- N refs/heads/v2-8-test (bf7fb4b019)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 20 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
airflow/io/path.py | 11 +------
.../providers/common/io/operators/file_transfer.py | 37 +++++++---------------
dev/breeze/tests/test_selective_checks.py | 4 +--
.../apache-airflow/core-concepts/objectstorage.rst | 8 +----
generated/provider_dependencies.json | 4 +--
tests/io/test_path.py | 11 +++----
.../common/io/operators/test_file_transfer.py | 24 --------------
7 files changed, 21 insertions(+), 78 deletions(-)
(airflow) 04/20: Run mypy checks for full packages in CI (#36638)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit b7485d7670d6fb147e8122e2fd3358f51937e916
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sun Jan 7 13:53:23 2024 +0100
Run mypy checks for full packages in CI (#36638)
MyPy as used in our static checks has slightly different heuristics
when running on on individual files and whole packages. This sometimes
causes semi-random failures when different set of files is produced
when pre-commits split the files between parallel processes.
The regular `mypy-*` pre-commits work by passing filenames to mypy
checks, and when `--all-files` flag is passed to mypy, this means
that 2700 files are passed. In this case pre-commit will split such
long list of files to several sequential muypy executions. This
is not very good because depending on the list of files passed,
mypy can split the list diferently and results will be different
when just list of files changes - so mypy might start detecting
problems that were not present before.
This PR introduces new `mypy` check that runs mypy for packages
rather than individual files. We cannot run them for local
pre-commit runs, because in many cases, such package based
mypy check will run for minutes when a single file changes,
due to cache invalidation rules - and we do not want to penalise
commits that are changing common airflow code (because such PRs
would invalidate a lot of mypy cache every time such common file
changes). So we still want to run file-based mypy for local
commits. But we do not want to pass 2700 files in CI, rather than
that on CI we want to run mypy checks "per package".
This PR introduces a new "manual" stage mypy pre-commit check that
will run "package" based mypy checks and adds selective check rules
that will decide properly when to run such tests and separate,
matrix-based CI job that will run such mypy checks - separately
for each of the packages: "airflow", "providers", "docs", "dev".
Also this job will skip providers checks in non-main branch and
will run all tests when "full tests needed" are requested.
This PR ignores some errors resulted from 3rd-party libraries used
that are randomply appearing when some files are modified (and fixes
the current main failures)
(cherry picked from commit f7b663d9aff472d0a419e16c262fbae2a8a69ce1)
---
.github/workflows/ci.yml | 56 ++++-
.pre-commit-config.yaml | 13 +-
STATIC_CODE_CHECKS.rst | 32 ++-
airflow/decorators/__init__.pyi | 18 +-
airflow/models/taskreschedule.py | 3 +-
airflow/operators/latest_only.py | 2 +-
airflow/providers/apache/druid/hooks/druid.py | 4 +-
.../providers/databricks/hooks/databricks_sql.py | 4 +-
airflow/providers/exasol/hooks/exasol.py | 6 +-
airflow/providers/google/cloud/hooks/cloud_run.py | 2 +-
.../providers/google/cloud/triggers/cloud_run.py | 2 +-
.../google/cloud/utils/credentials_provider.py | 2 +-
.../providers/google/common/hooks/base_google.py | 2 +-
.../google/common/utils/id_token_credentials.py | 4 +-
airflow/providers/grpc/hooks/grpc.py | 2 +-
airflow/providers/openlineage/plugins/facets.py | 4 +-
airflow/providers/postgres/hooks/postgres.py | 4 +-
airflow/providers/snowflake/hooks/snowflake.py | 6 +-
airflow/providers/trino/operators/trino.py | 8 +-
airflow/providers/vertica/hooks/vertica.py | 2 +-
airflow/providers_manager.py | 10 +-
airflow/sensors/base.py | 4 +-
airflow/utils/operator_helpers.py | 2 +-
dev/breeze/src/airflow_breeze/pre_commit_ids.py | 1 +
.../src/airflow_breeze/utils/selective_checks.py | 63 ++++--
dev/breeze/tests/test_selective_checks.py | 234 +++++++++++++++++----
images/breeze/output_static-checks.svg | 24 +--
images/breeze/output_static-checks.txt | 2 +-
scripts/ci/pre_commit/common_precommit_utils.py | 26 ++-
scripts/ci/pre_commit/pre_commit_mypy.py | 31 ++-
30 files changed, 443 insertions(+), 130 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 610132b81a..f321ec2619 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -107,6 +107,8 @@ jobs:
ci-image-build: ${{ steps.selective-checks.outputs.ci-image-build }}
prod-image-build: ${{ steps.selective-checks.outputs.prod-image-build }}
docs-build: ${{ steps.selective-checks.outputs.docs-build }}
+ mypy-packages: ${{ steps.selective-checks.outputs.mypy-packages }}
+ needs-mypy: ${{ steps.selective-checks.outputs.needs-mypy }}
needs-helm-tests: ${{ steps.selective-checks.outputs.needs-helm-tests }}
needs-api-tests: ${{ steps.selective-checks.outputs.needs-api-tests }}
needs-api-codegen: ${{ steps.selective-checks.outputs.needs-api-codegen }}
@@ -513,7 +515,6 @@ jobs:
retention-days: 7
if-no-files-found: error
-
static-checks:
timeout-minutes: 45
name: "Static checks"
@@ -553,6 +554,57 @@ jobs:
DEFAULT_BRANCH: ${{ needs.build-info.outputs.default-branch }}
RUFF_FORMAT: "github"
+ # Runs static checks for groups of files in the repository in a single process without passing .
+ # List of files
+ mypy:
+ timeout-minutes: 45
+ name: "MyPy checks"
+ runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
+ needs: [build-info, wait-for-ci-images]
+ strategy:
+ fail-fast: false
+ matrix:
+ mypy-package: ${{fromJson(needs.build-info.outputs.mypy-packages)}}
+ env:
+ RUNS_ON: "${{needs.build-info.outputs.runs-on}}"
+ PYTHON_MAJOR_MINOR_VERSION: "${{needs.build-info.outputs.default-python-version}}"
+ UPGRADE_TO_NEWER_DEPENDENCIES: "${{ needs.build-info.outputs.upgrade-to-newer-dependencies }}"
+ steps:
+ - name: Cleanup repo
+ run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
+ if: needs.build-info.outputs.needs-mypy == 'true'
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
+ uses: actions/checkout@v4
+ with:
+ persist-credentials: false
+ if: needs.build-info.outputs.needs-mypy == 'true'
+ - name: >
+ Prepare breeze & CI image: ${{needs.build-info.outputs.default-python-version}}:${{env.IMAGE_TAG}}
+ uses: ./.github/actions/prepare_breeze_and_image
+ id: breeze
+ if: needs.build-info.outputs.needs-mypy == 'true'
+ - name: Cache pre-commit envs
+ uses: actions/cache@v3
+ with:
+ path: ~/.cache/pre-commit
+ # yamllint disable-line rule:line-length
+ key: "pre-commit-${{steps.breeze.outputs.host-python-version}}-${{ hashFiles('.pre-commit-config.yaml') }}"
+ restore-keys: |
+ pre-commit-${{steps.breeze.outputs.host-python-version}}-
+ if: needs.build-info.outputs.needs-mypy == 'true'
+ - name: "MyPy checks for ${{ matrix.mypy-package }}"
+ run: |
+ pip install pre-commit
+ pre-commit run --color always --verbose --hook-stage manual mypy --all-files
+ env:
+ VERBOSE: "false"
+ COLUMNS: "250"
+ SKIP_GROUP_OUTPUT: "true"
+ DEFAULT_BRANCH: ${{ needs.build-info.outputs.default-branch }}
+ RUFF_FORMAT: "github"
+ MYPY_PACKAGES: ${{ matrix.mypy-package }}
+ if: needs.build-info.outputs.needs-mypy == 'true'
+
# Those checks are run if no image needs to be built for checks. This is for simple changes that
# Do not touch any of the python code or any of the important files that might require building
# The CI Docker image and they can be run entirely using the pre-commit virtual environments on host
@@ -2115,6 +2167,7 @@ jobs:
- wait-for-ci-images
- wait-for-prod-images
- static-checks
+ - mypy
- tests-sqlite
- tests-mysql
- tests-postgres
@@ -2269,6 +2322,7 @@ jobs:
- build-docs
- spellcheck-docs
- static-checks
+ - mypy
- tests-sqlite
- tests-mysql
- tests-postgres
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index d7f44643b7..8676195a42 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -736,7 +736,7 @@ repos:
# Keep dependency versions in sync w/ airflow/www/package.json
additional_dependencies: ['stylelint@13.3.1', 'stylelint-config-standard@20.0.0', 'stylelint-config-prettier@9.0.5']
- id: compile-www-assets
- name: Compile www assets
+ name: Compile www assets (manual)
language: node
stages: ['manual']
'types_or': [javascript, ts, tsx]
@@ -745,7 +745,7 @@ repos:
pass_filenames: false
additional_dependencies: ['yarn@1.22.19']
- id: compile-www-assets-dev
- name: Compile www assets in dev mode
+ name: Compile www assets in dev mode (manual)
language: node
stages: ['manual']
'types_or': [javascript, ts, tsx]
@@ -1105,6 +1105,15 @@ repos:
exclude: ^docs/rtd-deprecation
require_serial: true
additional_dependencies: ['rich>=12.4.4']
+ - id: mypy
+ stages: ['manual']
+ name: Run mypy for specified packages (manual)
+ language: python
+ entry: ./scripts/ci/pre_commit/pre_commit_mypy.py
+ pass_filenames: false
+ files: ^.*\.py$
+ require_serial: true
+ additional_dependencies: ['rich>=12.4.4']
- id: check-provider-yaml-valid
name: Validate provider.yaml files
entry: ./scripts/ci/pre_commit/pre_commit_check_provider_yaml_files.py
diff --git a/STATIC_CODE_CHECKS.rst b/STATIC_CODE_CHECKS.rst
index 29a70e533f..57d11e42c2 100644
--- a/STATIC_CODE_CHECKS.rst
+++ b/STATIC_CODE_CHECKS.rst
@@ -114,6 +114,13 @@ Available pre-commit checks
This table lists pre-commit hooks used by Airflow. The ``Image`` column indicates which hooks
require Breeze Docker image to be built locally.
+.. note:: Manual pre-commits
+
+ Most of the checks we run are configured to run automatically when you commit the code. However,
+ there are some checks that are not run automatically and you need to run them manually. Those
+ checks are marked with ``manual`` in the ``Description`` column in the table below. You can run
+ them manually by running ``pre-commit run --hook-stage manual <hook-id>``.
+
.. note:: Disabling particular checks
In case you have a problem with running particular ``pre-commit`` check you can still continue using the
@@ -127,6 +134,24 @@ require Breeze Docker image to be built locally.
the image by setting ``SKIP_BREEZE_PRE_COMMITS`` to "true". This will mark the tests as "green" automatically
when run locally (note that those checks will anyway run in CI).
+.. note:: Mypy checks
+
+ When we run mypy checks locally when committing a change, one of the ``mypy-*`` checks is run, ``mypy-core``,
+ ``mypy-dev``, ``mypy-providers``, ``mypy-docs``, depending on the files you are changing. The mypy checks
+ are run by passing those changed files to mypy. This is way faster than running checks for all files (even
+ if mypy cache is used - especially when you change a file in airflow core that is imported and used by many
+ files). However, in some cases, it produces different results than when running checks for the whole set
+ of files, because ``mypy`` does not even know that some types are defined in other files and it might not
+ be able to follow imports properly if they are dynamic. Therefore in CI we run ``mypy`` check for whole
+ directories (``airflow`` - excluding providers, ``airflow/providers``, ``dev`` and ``docs``) to make sure
+ that we catch all ``mypy`` errors - so you can experience different results when running mypy locally and
+ in CI. If you want to run mypy checks for all files locally, you can do it by running the following
+ command (example for ``airflow`` files):
+
+ .. code-block:: bash
+
+ MYPY_PACKAGES="airflow" pre-commit run --hook-stage manual mypy --all-files
+
.. note:: Mypy volume cache
MyPy uses a separate docker-volume (called ``mypy-cache-volume``) that keeps the cache of last MyPy
@@ -135,6 +160,7 @@ require Breeze Docker image to be built locally.
is the hard problem in computer science). This might happen for example when we upgrade MyPY. In such
cases you might need to manually remove the cache volume by running ``breeze down --cleanup-mypy-cache``.
+
.. BEGIN AUTO-GENERATED STATIC CHECK LIST
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
@@ -258,9 +284,9 @@ require Breeze Docker image to be built locally.
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
| codespell | Run codespell to check for common misspellings in files | |
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| compile-www-assets | Compile www assets | |
+| compile-www-assets | Compile www assets (manual) | |
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| compile-www-assets-dev | Compile www assets in dev mode | |
+| compile-www-assets-dev | Compile www assets in dev mode (manual) | |
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
| create-missing-init-py-files-tests | Create missing init.py files in tests | |
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
@@ -316,6 +342,8 @@ require Breeze Docker image to be built locally.
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
| mixed-line-ending | Detect if mixed line ending is used (\r vs. \r\n) | |
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
+| mypy | Run mypy for specified packages (manual) | * |
++-----------------------------------------------------------+--------------------------------------------------------------+---------+
| mypy-core | Run mypy for core | * |
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
| mypy-dev | Run mypy for dev | * |
diff --git a/airflow/decorators/__init__.pyi b/airflow/decorators/__init__.pyi
index 21bddcbd0a..2fe8c704e5 100644
--- a/airflow/decorators/__init__.pyi
+++ b/airflow/decorators/__init__.pyi
@@ -60,7 +60,7 @@ __all__ = [
class TaskDecoratorCollection:
@overload
- def python(
+ def python( # type: ignore[misc]
self,
*,
multiple_outputs: bool | None = None,
@@ -88,7 +88,7 @@ class TaskDecoratorCollection:
def python(self, python_callable: Callable[FParams, FReturn]) -> Task[FParams, FReturn]: ...
# [END mixin_for_typing]
@overload
- def __call__(
+ def __call__( # type: ignore[misc]
self,
*,
multiple_outputs: bool | None = None,
@@ -101,7 +101,7 @@ class TaskDecoratorCollection:
def __call__(self, python_callable: Callable[FParams, FReturn]) -> Task[FParams, FReturn]:
"""Aliasing ``python``; signature should match exactly."""
@overload
- def virtualenv(
+ def virtualenv( # type: ignore[misc]
self,
*,
multiple_outputs: bool | None = None,
@@ -187,7 +187,9 @@ class TaskDecoratorCollection:
such as transmission a large amount of XCom to TaskAPI.
"""
@overload
- def branch(self, *, multiple_outputs: bool | None = None, **kwargs) -> TaskDecorator:
+ def branch( # type: ignore[misc]
+ self, *, multiple_outputs: bool | None = None, **kwargs
+ ) -> TaskDecorator:
"""Create a decorator to wrap the decorated callable into a BranchPythonOperator.
For more information on how to use this decorator, see :ref:`concepts:branching`.
@@ -199,7 +201,7 @@ class TaskDecoratorCollection:
@overload
def branch(self, python_callable: Callable[FParams, FReturn]) -> Task[FParams, FReturn]: ...
@overload
- def branch_virtualenv(
+ def branch_virtualenv( # type: ignore[misc]
self,
*,
multiple_outputs: bool | None = None,
@@ -292,7 +294,7 @@ class TaskDecoratorCollection:
self, python_callable: Callable[FParams, FReturn]
) -> Task[FParams, FReturn]: ...
@overload
- def short_circuit(
+ def short_circuit( # type: ignore[misc]
self,
*,
multiple_outputs: bool | None = None,
@@ -627,7 +629,7 @@ class TaskDecoratorCollection:
:param progress_callback: Callback function for receiving k8s container logs.
"""
@overload
- def sensor(
+ def sensor( # type: ignore[misc]
self,
*,
poke_interval: float = ...,
@@ -664,7 +666,7 @@ class TaskDecoratorCollection:
@overload
def sensor(self, python_callable: Callable[FParams, FReturn] | None = None) -> Task[FParams, FReturn]: ...
@overload
- def pyspark(
+ def pyspark( # type: ignore[misc]
self,
*,
multiple_outputs: bool | None = None,
diff --git a/airflow/models/taskreschedule.py b/airflow/models/taskreschedule.py
index 1305932d3b..145dfae848 100644
--- a/airflow/models/taskreschedule.py
+++ b/airflow/models/taskreschedule.py
@@ -38,6 +38,7 @@ if TYPE_CHECKING:
from airflow.models.operator import Operator
from airflow.models.taskinstance import TaskInstance
+ from airflow.serialization.pydantic.taskinstance import TaskInstancePydantic
class TaskReschedule(Base):
@@ -103,7 +104,7 @@ class TaskReschedule(Base):
@classmethod
def stmt_for_task_instance(
cls,
- ti: TaskInstance,
+ ti: TaskInstance | TaskInstancePydantic,
*,
try_number: int | None = None,
descending: bool = False,
diff --git a/airflow/operators/latest_only.py b/airflow/operators/latest_only.py
index bca66dc2d4..7b4341b101 100644
--- a/airflow/operators/latest_only.py
+++ b/airflow/operators/latest_only.py
@@ -45,7 +45,7 @@ class LatestOnlyOperator(BaseBranchOperator):
def choose_branch(self, context: Context) -> str | Iterable[str]:
# If the DAG Run is externally triggered, then return without
# skipping downstream tasks
- dag_run: DagRun = context["dag_run"]
+ dag_run: DagRun = context["dag_run"] # type: ignore[assignment]
if dag_run.external_trigger:
self.log.info("Externally triggered DAG_Run: allowing execution to proceed.")
return list(context["task"].get_direct_relative_ids(upstream=False))
diff --git a/airflow/providers/apache/druid/hooks/druid.py b/airflow/providers/apache/druid/hooks/druid.py
index 455c780f76..92d4aa6408 100644
--- a/airflow/providers/apache/druid/hooks/druid.py
+++ b/airflow/providers/apache/druid/hooks/druid.py
@@ -200,7 +200,7 @@ class DruidDbApiHook(DbApiHook):
endpoint = conn.extra_dejson.get("endpoint", "druid/v2/sql")
return f"{conn_type}://{host}/{endpoint}"
- def set_autocommit(self, conn: connect, autocommit: bool) -> NotImplementedError:
+ def set_autocommit(self, conn: connect, autocommit: bool) -> None:
raise NotImplementedError()
def insert_rows(
@@ -211,5 +211,5 @@ class DruidDbApiHook(DbApiHook):
commit_every: int = 1000,
replace: bool = False,
**kwargs: Any,
- ) -> NotImplementedError:
+ ) -> None:
raise NotImplementedError()
diff --git a/airflow/providers/databricks/hooks/databricks_sql.py b/airflow/providers/databricks/hooks/databricks_sql.py
index d61d9f1bd7..80771c1aab 100644
--- a/airflow/providers/databricks/hooks/databricks_sql.py
+++ b/airflow/providers/databricks/hooks/databricks_sql.py
@@ -145,7 +145,7 @@ class DatabricksSqlHook(BaseDatabricksHook, DbApiHook):
)
return self._sql_conn
- @overload
+ @overload # type: ignore[override]
def run(
self,
sql: str | Iterable[str],
@@ -220,7 +220,7 @@ class DatabricksSqlHook(BaseDatabricksHook, DbApiHook):
self.set_autocommit(conn, autocommit)
with closing(conn.cursor()) as cur:
- self._run_command(cur, sql_statement, parameters)
+ self._run_command(cur, sql_statement, parameters) # type: ignore[attr-defined]
if handler is not None:
result = self._make_serializable(handler(cur))
if return_single_query_results(sql, return_last, split_statements):
diff --git a/airflow/providers/exasol/hooks/exasol.py b/airflow/providers/exasol/hooks/exasol.py
index 98955a2579..ccb1c2e0a4 100644
--- a/airflow/providers/exasol/hooks/exasol.py
+++ b/airflow/providers/exasol/hooks/exasol.py
@@ -162,7 +162,7 @@ class ExasolHook(DbApiHook):
)
return cols
- @overload
+ @overload # type: ignore[override]
def run(
self,
sql: str | Iterable[str],
@@ -232,7 +232,9 @@ class ExasolHook(DbApiHook):
with closing(conn.execute(sql_statement, parameters)) as exa_statement:
self.log.info("Running statement: %s, parameters: %s", sql_statement, parameters)
if handler is not None:
- result = handler(exa_statement)
+ result = self._make_common_data_structure( # type: ignore[attr-defined]
+ handler(exa_statement)
+ )
if return_single_query_results(sql, return_last, split_statements):
_last_result = result
_last_columns = self.get_description(exa_statement)
diff --git a/airflow/providers/google/cloud/hooks/cloud_run.py b/airflow/providers/google/cloud/hooks/cloud_run.py
index 8741aa1d4d..61a6fadb76 100644
--- a/airflow/providers/google/cloud/hooks/cloud_run.py
+++ b/airflow/providers/google/cloud/hooks/cloud_run.py
@@ -31,7 +31,7 @@ from google.cloud.run_v2 import (
RunJobRequest,
UpdateJobRequest,
)
-from google.longrunning import operations_pb2
+from google.longrunning import operations_pb2 # type: ignore[attr-defined]
from airflow.exceptions import AirflowException
from airflow.providers.google.common.consts import CLIENT_INFO
diff --git a/airflow/providers/google/cloud/triggers/cloud_run.py b/airflow/providers/google/cloud/triggers/cloud_run.py
index 9506245d20..772650723a 100644
--- a/airflow/providers/google/cloud/triggers/cloud_run.py
+++ b/airflow/providers/google/cloud/triggers/cloud_run.py
@@ -25,7 +25,7 @@ from airflow.providers.google.cloud.hooks.cloud_run import CloudRunAsyncHook
from airflow.triggers.base import BaseTrigger, TriggerEvent
if TYPE_CHECKING:
- from google.longrunning import operations_pb2
+ from google.longrunning import operations_pb2 # type: ignore[attr-defined]
DEFAULT_BATCH_LOCATION = "us-central1"
diff --git a/airflow/providers/google/cloud/utils/credentials_provider.py b/airflow/providers/google/cloud/utils/credentials_provider.py
index 18c00b0844..df83358e4b 100644
--- a/airflow/providers/google/cloud/utils/credentials_provider.py
+++ b/airflow/providers/google/cloud/utils/credentials_provider.py
@@ -30,7 +30,7 @@ from urllib.parse import urlencode
import google.auth
import google.auth.credentials
import google.oauth2.service_account
-from google.auth import impersonated_credentials
+from google.auth import impersonated_credentials # type: ignore[attr-defined]
from google.auth.environment_vars import CREDENTIALS, LEGACY_PROJECT, PROJECT
from airflow.exceptions import AirflowException
diff --git a/airflow/providers/google/common/hooks/base_google.py b/airflow/providers/google/common/hooks/base_google.py
index 3bde53571a..a50887b3a1 100644
--- a/airflow/providers/google/common/hooks/base_google.py
+++ b/airflow/providers/google/common/hooks/base_google.py
@@ -36,7 +36,7 @@ import requests
import tenacity
from asgiref.sync import sync_to_async
from google.api_core.exceptions import Forbidden, ResourceExhausted, TooManyRequests
-from google.auth import _cloud_sdk, compute_engine
+from google.auth import _cloud_sdk, compute_engine # type: ignore[attr-defined]
from google.auth.environment_vars import CLOUD_SDK_CONFIG_DIR, CREDENTIALS
from google.auth.exceptions import RefreshError
from google.auth.transport import _http_client
diff --git a/airflow/providers/google/common/utils/id_token_credentials.py b/airflow/providers/google/common/utils/id_token_credentials.py
index 6a41438a22..c2b78bb2e1 100644
--- a/airflow/providers/google/common/utils/id_token_credentials.py
+++ b/airflow/providers/google/common/utils/id_token_credentials.py
@@ -36,7 +36,7 @@ from typing import TYPE_CHECKING
import google.auth.transport
from google.auth import credentials as google_auth_credentials, environment_vars, exceptions
-from google.oauth2 import credentials as oauth2_credentials, service_account
+from google.oauth2 import credentials as oauth2_credentials, service_account # type: ignore[attr-defined]
if TYPE_CHECKING:
import google.oauth2
@@ -146,7 +146,7 @@ def _get_gcloud_sdk_credentials(
target_audience: str | None,
) -> google_auth_credentials.Credentials | None:
"""Gets the credentials and project ID from the Cloud SDK."""
- from google.auth import _cloud_sdk
+ from google.auth import _cloud_sdk # type: ignore[attr-defined]
# Check if application default credentials exist.
credentials_filename = _cloud_sdk.get_application_default_credentials_path()
diff --git a/airflow/providers/grpc/hooks/grpc.py b/airflow/providers/grpc/hooks/grpc.py
index a3ea7530cf..3d1623ea6c 100644
--- a/airflow/providers/grpc/hooks/grpc.py
+++ b/airflow/providers/grpc/hooks/grpc.py
@@ -21,7 +21,7 @@ from typing import Any, Callable, Generator
import grpc
from google import auth as google_auth
-from google.auth import jwt as google_auth_jwt
+from google.auth import jwt as google_auth_jwt # type: ignore[attr-defined]
from google.auth.transport import (
grpc as google_auth_transport_grpc,
requests as google_auth_transport_requests,
diff --git a/airflow/providers/openlineage/plugins/facets.py b/airflow/providers/openlineage/plugins/facets.py
index b50db6c267..4037b9026a 100644
--- a/airflow/providers/openlineage/plugins/facets.py
+++ b/airflow/providers/openlineage/plugins/facets.py
@@ -28,7 +28,7 @@ class AirflowMappedTaskRunFacet(BaseFacet):
mapIndex: int
operatorClass: str
- _additional_skip_redact: list[str] = ["operatorClass"]
+ _additional_skip_redact = ["operatorClass"]
@classmethod
def from_task_instance(cls, task_instance):
@@ -63,7 +63,7 @@ class UnknownOperatorInstance(RedactMixin):
properties: dict[str, object]
type: str = "operator"
- _skip_redact: list[str] = ["name", "type"]
+ _skip_redact = ["name", "type"]
@define(slots=False)
diff --git a/airflow/providers/postgres/hooks/postgres.py b/airflow/providers/postgres/hooks/postgres.py
index 033fff0921..e33e45b6c8 100644
--- a/airflow/providers/postgres/hooks/postgres.py
+++ b/airflow/providers/postgres/hooks/postgres.py
@@ -332,7 +332,9 @@ class PostgresHook(DbApiHook):
if is_redshift:
authority = self._get_openlineage_redshift_authority_part(connection)
else:
- authority = DbApiHook.get_openlineage_authority_part(connection, default_port=5432)
+ authority = DbApiHook.get_openlineage_authority_part( # type: ignore[attr-defined]
+ connection, default_port=5432
+ )
return DatabaseInfo(
scheme="postgres" if not is_redshift else "redshift",
diff --git a/airflow/providers/snowflake/hooks/snowflake.py b/airflow/providers/snowflake/hooks/snowflake.py
index 081d734ac3..80aeea9646 100644
--- a/airflow/providers/snowflake/hooks/snowflake.py
+++ b/airflow/providers/snowflake/hooks/snowflake.py
@@ -300,7 +300,7 @@ class SnowflakeHook(DbApiHook):
def get_autocommit(self, conn):
return getattr(conn, "autocommit_mode", False)
- @overload
+ @overload # type: ignore[override]
def run(
self,
sql: str | Iterable[str],
@@ -385,10 +385,10 @@ class SnowflakeHook(DbApiHook):
with self._get_cursor(conn, return_dictionaries) as cur:
results = []
for sql_statement in sql_list:
- self._run_command(cur, sql_statement, parameters)
+ self._run_command(cur, sql_statement, parameters) # type: ignore[attr-defined]
if handler is not None:
- result = handler(cur)
+ result = self._make_common_data_structure(handler(cur)) # type: ignore[attr-defined]
if return_single_query_results(sql, return_last, split_statements):
_last_result = result
_last_description = cur.description
diff --git a/airflow/providers/trino/operators/trino.py b/airflow/providers/trino/operators/trino.py
index 7f90bf9947..20798977b6 100644
--- a/airflow/providers/trino/operators/trino.py
+++ b/airflow/providers/trino/operators/trino.py
@@ -65,11 +65,11 @@ class TrinoOperator(SQLExecuteQueryOperator):
)
def on_kill(self) -> None:
- if self._hook is not None and isinstance(self._hook, TrinoHook):
- query_id = "'" + self._hook.query_id + "'"
+ if self._hook is not None and isinstance(self._hook, TrinoHook): # type: ignore[attr-defined]
+ query_id = "'" + self._hook.query_id + "'" # type: ignore[attr-defined]
try:
- self.log.info("Stopping query run with queryId - %s", self._hook.query_id)
- self._hook.run(
+ self.log.info("Stopping query run with queryId - %s", self._hook.query_id) # type: ignore[attr-defined]
+ self._hook.run( # type: ignore[attr-defined]
sql=f"CALL system.runtime.kill_query(query_id => {query_id},message => 'Job "
f"killed by "
f"user');",
diff --git a/airflow/providers/vertica/hooks/vertica.py b/airflow/providers/vertica/hooks/vertica.py
index 91672e2aec..191fa10a8b 100644
--- a/airflow/providers/vertica/hooks/vertica.py
+++ b/airflow/providers/vertica/hooks/vertica.py
@@ -132,7 +132,7 @@ class VerticaHook(DbApiHook):
conn = connect(**conn_config)
return conn
- @overload
+ @overload # type: ignore[override]
def run(
self,
sql: str | Iterable[str],
diff --git a/airflow/providers_manager.py b/airflow/providers_manager.py
index b510c47861..5ca1b6740e 100644
--- a/airflow/providers_manager.py
+++ b/airflow/providers_manager.py
@@ -404,7 +404,7 @@ class ProvidersManager(LoggingMixin, metaclass=Singleton):
return ProvidersManager._initialized
@staticmethod
- def initialization_stack_trace() -> str:
+ def initialization_stack_trace() -> str | None:
return ProvidersManager._initialization_stack_trace
def __init__(self):
@@ -418,7 +418,7 @@ class ProvidersManager(LoggingMixin, metaclass=Singleton):
# Keeps dict of hooks keyed by connection type
self._hooks_dict: dict[str, HookInfo] = {}
self._fs_set: set[str] = set()
- self._taskflow_decorators: dict[str, Callable] = LazyDictWithCache()
+ self._taskflow_decorators: dict[str, Callable] = LazyDictWithCache() # type: ignore[assignment]
# keeps mapping between connection_types and hook class, package they come from
self._hook_provider_dict: dict[str, HookClassProvider] = {}
# Keeps dict of hooks keyed by connection type. They are lazy evaluated at access time
@@ -1105,7 +1105,9 @@ class ProvidersManager(LoggingMixin, metaclass=Singleton):
"""Retrieve all configs defined in the providers."""
for provider_package, provider in self._provider_dict.items():
if provider.data.get("config"):
- self._provider_configs[provider_package] = provider.data.get("config")
+ self._provider_configs[provider_package] = (
+ provider.data.get("config") # type: ignore[assignment]
+ )
def _discover_plugins(self) -> None:
"""Retrieve all plugins defined in the providers."""
@@ -1175,7 +1177,7 @@ class ProvidersManager(LoggingMixin, metaclass=Singleton):
@property
def taskflow_decorators(self) -> dict[str, TaskDecorator]:
self.initialize_providers_taskflow_decorator()
- return self._taskflow_decorators
+ return self._taskflow_decorators # type: ignore[return-value]
@property
def extra_links_class_names(self) -> list[str]:
diff --git a/airflow/sensors/base.py b/airflow/sensors/base.py
index 8bf40c913b..d1a01ac87e 100644
--- a/airflow/sensors/base.py
+++ b/airflow/sensors/base.py
@@ -212,7 +212,9 @@ class BaseSensorOperator(BaseOperator, SkipMixin):
if self.reschedule:
# If reschedule, use the start date of the first try (first try can be either the very
# first execution of the task, or the first execution after the task was cleared.)
- first_try_number = context["ti"].max_tries - self.retries + 1
+ max_tries: int = context["ti"].max_tries or 0
+ retries: int = self.retries or 0
+ first_try_number = max_tries - retries + 1
with create_session() as session:
start_date = session.scalar(
TaskReschedule.stmt_for_task_instance(
diff --git a/airflow/utils/operator_helpers.py b/airflow/utils/operator_helpers.py
index ef1f05e304..5f92467147 100644
--- a/airflow/utils/operator_helpers.py
+++ b/airflow/utils/operator_helpers.py
@@ -172,7 +172,7 @@ class KeywordParameters:
def unpacking(self) -> Mapping[str, Any]:
"""Dump the kwargs mapping to unpack with ``**`` in a function call."""
- if self._wildcard and isinstance(self._kwargs, Context):
+ if self._wildcard and isinstance(self._kwargs, Context): # type: ignore[misc]
return lazy_mapping_from_context(self._kwargs)
return self._kwargs
diff --git a/dev/breeze/src/airflow_breeze/pre_commit_ids.py b/dev/breeze/src/airflow_breeze/pre_commit_ids.py
index dd9f639596..1b35b2c5bc 100644
--- a/dev/breeze/src/airflow_breeze/pre_commit_ids.py
+++ b/dev/breeze/src/airflow_breeze/pre_commit_ids.py
@@ -103,6 +103,7 @@ PRE_COMMIT_LIST = [
"lint-markdown",
"lint-openapi",
"mixed-line-ending",
+ "mypy",
"mypy-core",
"mypy-dev",
"mypy-docs",
diff --git a/dev/breeze/src/airflow_breeze/utils/selective_checks.py b/dev/breeze/src/airflow_breeze/utils/selective_checks.py
index 2cf57200a4..4db2972356 100644
--- a/dev/breeze/src/airflow_breeze/utils/selective_checks.py
+++ b/dev/breeze/src/airflow_breeze/utils/selective_checks.py
@@ -584,6 +584,43 @@ class SelectiveChecks:
)
return False
+ @cached_property
+ def mypy_packages(self) -> list[str]:
+ packages_to_run: list[str] = []
+ if (
+ self._matching_files(
+ FileGroupForCi.ALL_AIRFLOW_PYTHON_FILES, CI_FILE_GROUP_MATCHES, CI_FILE_GROUP_EXCLUDES
+ )
+ or self.full_tests_needed
+ ):
+ packages_to_run.append("airflow")
+ if (
+ self._matching_files(
+ FileGroupForCi.ALL_PROVIDERS_PYTHON_FILES, CI_FILE_GROUP_MATCHES, CI_FILE_GROUP_EXCLUDES
+ )
+ or self._are_all_providers_affected()
+ ) and self._default_branch == "main":
+ packages_to_run.append("airflow/providers")
+ if (
+ self._matching_files(
+ FileGroupForCi.ALL_DOCS_PYTHON_FILES, CI_FILE_GROUP_MATCHES, CI_FILE_GROUP_EXCLUDES
+ )
+ or self.full_tests_needed
+ ):
+ packages_to_run.append("docs")
+ if (
+ self._matching_files(
+ FileGroupForCi.ALL_DEV_PYTHON_FILES, CI_FILE_GROUP_MATCHES, CI_FILE_GROUP_EXCLUDES
+ )
+ or self.full_tests_needed
+ ):
+ packages_to_run.append("dev")
+ return packages_to_run
+
+ @cached_property
+ def needs_mypy(self) -> bool:
+ return self.mypy_packages != []
+
@cached_property
def needs_python_scans(self) -> bool:
return self._should_be_run(FileGroupForCi.PYTHON_PRODUCTION_FILES)
@@ -823,6 +860,14 @@ class SelectiveChecks:
def skip_pre_commits(self) -> str:
pre_commits_to_skip = set()
pre_commits_to_skip.add("identity")
+ # Skip all mypy "individual" file checks if we are running mypy checks in CI
+ # In the CI we always run mypy for the whole "package" rather than for `--all-files` because
+ # The pre-commit will semi-randomly skip such list of files into several groups and we want
+ # to make sure that such checks are always run in CI for whole "group" of files - i.e.
+ # whole package rather than for individual files. That's why we skip those checks in CI
+ # and run them via `mypy-all` command instead and dedicated CI job in matrix
+ # This will also speed up static-checks job usually as the jobs will be running in parallel
+ pre_commits_to_skip.update({"mypy-providers", "mypy-core", "mypy-docs", "mypy-dev"})
if self._default_branch != "main":
# Skip those tests on all "release" branches
pre_commits_to_skip.update(
@@ -831,29 +876,13 @@ class SelectiveChecks:
"check-extra-packages-references",
"check-provider-yaml-valid",
"lint-helm-chart",
- "mypy-providers",
)
)
+
if self.full_tests_needed:
# when full tests are needed, we do not want to skip any checks and we should
# run all the pre-commits just to be sure everything is ok when some structural changes occurred
return ",".join(sorted(pre_commits_to_skip))
- if not self._matching_files(
- FileGroupForCi.ALL_PROVIDERS_PYTHON_FILES, CI_FILE_GROUP_MATCHES, CI_FILE_GROUP_EXCLUDES
- ):
- pre_commits_to_skip.add("mypy-providers")
- if not self._matching_files(
- FileGroupForCi.ALL_AIRFLOW_PYTHON_FILES, CI_FILE_GROUP_MATCHES, CI_FILE_GROUP_EXCLUDES
- ):
- pre_commits_to_skip.add("mypy-core")
- if not self._matching_files(
- FileGroupForCi.ALL_DOCS_PYTHON_FILES, CI_FILE_GROUP_MATCHES, CI_FILE_GROUP_EXCLUDES
- ):
- pre_commits_to_skip.add("mypy-docs")
- if not self._matching_files(
- FileGroupForCi.ALL_DEV_PYTHON_FILES, CI_FILE_GROUP_MATCHES, CI_FILE_GROUP_EXCLUDES
- ):
- pre_commits_to_skip.add("mypy-dev")
if not self._matching_files(FileGroupForCi.WWW_FILES, CI_FILE_GROUP_MATCHES, CI_FILE_GROUP_EXCLUDES):
pre_commits_to_skip.add("ts-compile-format-lint-www")
if not self._matching_files(
diff --git a/dev/breeze/tests/test_selective_checks.py b/dev/breeze/tests/test_selective_checks.py
index 495c98fd87..56e2190628 100644
--- a/dev/breeze/tests/test_selective_checks.py
+++ b/dev/breeze/tests/test_selective_checks.py
@@ -75,7 +75,7 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
print_in_color("\nOutput received:")
print_in_color(received_output_as_dict)
print_in_color()
- assert expected_value == received_value
+ assert received_value == expected_value
else:
print(
f"\n[red]ERROR: The key '{expected_key}' missing but "
@@ -111,6 +111,8 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"mypy-docs,mypy-providers,ts-compile-format-lint-www",
"upgrade-to-newer-dependencies": "false",
"parallel-test-types-list-as-string": None,
+ "needs-mypy": "false",
+ "mypy-packages": "[]",
},
id="No tests on simple change",
)
@@ -130,10 +132,12 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"run-tests": "true",
"run-amazon-tests": "false",
"docs-build": "true",
- "skip-pre-commits": "check-provider-yaml-valid,identity,lint-helm-chart,mypy-dev,"
+ "skip-pre-commits": "check-provider-yaml-valid,identity,lint-helm-chart,mypy-core,mypy-dev,"
"mypy-docs,mypy-providers,ts-compile-format-lint-www",
"upgrade-to-newer-dependencies": "false",
"parallel-test-types-list-as-string": "API Always",
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow']",
},
id="Only API tests and DOCS should run",
)
@@ -153,10 +157,12 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"run-tests": "true",
"run-amazon-tests": "false",
"docs-build": "true",
- "skip-pre-commits": "check-provider-yaml-valid,identity,lint-helm-chart,mypy-dev,"
+ "skip-pre-commits": "check-provider-yaml-valid,identity,lint-helm-chart,mypy-core,mypy-dev,"
"mypy-docs,mypy-providers,ts-compile-format-lint-www",
"upgrade-to-newer-dependencies": "false",
"parallel-test-types-list-as-string": "Always Operators",
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow']",
},
id="Only Operator tests and DOCS should run",
)
@@ -176,11 +182,13 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"run-tests": "true",
"run-amazon-tests": "false",
"docs-build": "true",
- "skip-pre-commits": "check-provider-yaml-valid,identity,lint-helm-chart,mypy-dev,"
+ "skip-pre-commits": "check-provider-yaml-valid,identity,lint-helm-chart,mypy-core,mypy-dev,"
"mypy-docs,mypy-providers,ts-compile-format-lint-www",
"upgrade-to-newer-dependencies": "false",
"parallel-test-types-list-as-string": "Always BranchExternalPython BranchPythonVenv "
"ExternalPython Operators PythonVenv",
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow']",
},
id="Only Python tests",
)
@@ -200,10 +208,12 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"run-tests": "true",
"run-amazon-tests": "false",
"docs-build": "true",
- "skip-pre-commits": "check-provider-yaml-valid,identity,lint-helm-chart,mypy-dev,"
+ "skip-pre-commits": "check-provider-yaml-valid,identity,lint-helm-chart,mypy-core,mypy-dev,"
"mypy-docs,mypy-providers,ts-compile-format-lint-www",
"upgrade-to-newer-dependencies": "false",
"parallel-test-types-list-as-string": "Always Serialization",
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow']",
},
id="Only Serialization tests",
)
@@ -227,11 +237,13 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"run-tests": "true",
"run-amazon-tests": "true",
"docs-build": "true",
- "skip-pre-commits": "identity,lint-helm-chart,mypy-dev,mypy-docs,"
+ "skip-pre-commits": "identity,lint-helm-chart,mypy-core,mypy-dev,mypy-docs,mypy-providers,"
"ts-compile-format-lint-www",
"upgrade-to-newer-dependencies": "false",
"parallel-test-types-list-as-string": "API Always Providers[amazon] "
"Providers[common.sql,openlineage,pgvector,postgres] Providers[google]",
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow', 'airflow/providers']",
},
id="API and providers tests and docs should run",
)
@@ -251,11 +263,13 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"run-tests": "true",
"run-amazon-tests": "false",
"docs-build": "false",
- "skip-pre-commits": "identity,lint-helm-chart,mypy-core,mypy-dev,mypy-docs,"
+ "skip-pre-commits": "identity,lint-helm-chart,mypy-core,mypy-dev,mypy-docs,mypy-providers,"
"ts-compile-format-lint-www",
"run-kubernetes-tests": "false",
"upgrade-to-newer-dependencies": "false",
"parallel-test-types-list-as-string": "Always Providers[apache.beam] Providers[google]",
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow/providers']",
},
id="Selected Providers and docs should run",
)
@@ -280,6 +294,8 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"run-kubernetes-tests": "false",
"upgrade-to-newer-dependencies": "false",
"parallel-test-types-list-as-string": None,
+ "needs-mypy": "false",
+ "mypy-packages": "[]",
},
id="Only docs builds should run - no tests needed",
)
@@ -303,11 +319,13 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"run-tests": "true",
"run-amazon-tests": "true",
"docs-build": "true",
- "skip-pre-commits": "identity,mypy-core,mypy-dev,mypy-docs,ts-compile-format-lint-www",
+ "skip-pre-commits": "identity,mypy-core,mypy-dev,mypy-docs,mypy-providers,ts-compile-format-lint-www",
"run-kubernetes-tests": "true",
"upgrade-to-newer-dependencies": "false",
"parallel-test-types-list-as-string": "Always Providers[amazon] "
"Providers[common.sql,openlineage,pgvector,postgres] Providers[google]",
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow/providers']",
},
id="Helm tests, providers (both upstream and downstream),"
"kubernetes tests and docs should run",
@@ -333,11 +351,13 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"run-tests": "true",
"run-amazon-tests": "true",
"docs-build": "true",
- "skip-pre-commits": "identity,mypy-core,mypy-dev,mypy-docs,ts-compile-format-lint-www",
+ "skip-pre-commits": "identity,mypy-core,mypy-dev,mypy-docs,mypy-providers,ts-compile-format-lint-www",
"run-kubernetes-tests": "true",
"upgrade-to-newer-dependencies": "false",
"parallel-test-types-list-as-string": "Always "
"Providers[airbyte,apache.livy,dbt.cloud,dingding,discord,http] Providers[amazon]",
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow/providers']",
},
id="Helm tests, http and all relevant providers, kubernetes tests and "
"docs should run even if unimportant files were added",
@@ -362,10 +382,12 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"run-tests": "true",
"run-amazon-tests": "false",
"docs-build": "true",
- "skip-pre-commits": "identity,mypy-core,mypy-dev,mypy-docs,ts-compile-format-lint-www",
+ "skip-pre-commits": "identity,mypy-core,mypy-dev,mypy-docs,mypy-providers,ts-compile-format-lint-www",
"run-kubernetes-tests": "true",
"upgrade-to-newer-dependencies": "false",
"parallel-test-types-list-as-string": "Always Providers[airbyte,http]",
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow/providers']",
},
id="Helm tests, airbyte/http providers, kubernetes tests and "
"docs should run even if unimportant files were added",
@@ -389,12 +411,14 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"needs-helm-tests": "true",
"run-tests": "true",
"docs-build": "true",
- "skip-pre-commits": "check-provider-yaml-valid,identity,mypy-dev,"
+ "skip-pre-commits": "check-provider-yaml-valid,identity,mypy-core,mypy-dev,"
"mypy-docs,mypy-providers,ts-compile-format-lint-www",
"run-amazon-tests": "false",
"run-kubernetes-tests": "true",
"upgrade-to-newer-dependencies": "false",
"parallel-test-types-list-as-string": "Always",
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow']",
},
id="Docs should run even if unimportant files were added and prod image "
"should be build for chart changes",
@@ -416,9 +440,11 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"run-amazon-tests": "true",
"docs-build": "true",
"full-tests-needed": "true",
- "skip-pre-commits": "identity",
+ "skip-pre-commits": "identity,mypy-core,mypy-dev,mypy-docs,mypy-providers",
"upgrade-to-newer-dependencies": "true",
"parallel-test-types-list-as-string": ALL_CI_SELECTIVE_TEST_TYPES,
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow', 'airflow/providers', 'docs', 'dev']",
},
id="Everything should run - including all providers and upgrading to "
"newer requirements as setup.py changed and all Python versions",
@@ -440,9 +466,11 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"run-amazon-tests": "true",
"docs-build": "true",
"full-tests-needed": "true",
- "skip-pre-commits": "identity",
+ "skip-pre-commits": "identity,mypy-core,mypy-dev,mypy-docs,mypy-providers",
"upgrade-to-newer-dependencies": "true",
"parallel-test-types-list-as-string": ALL_CI_SELECTIVE_TEST_TYPES,
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow', 'airflow/providers', 'docs', 'dev']",
},
id="Everything should run and upgrading to newer requirements as dependencies change",
)
@@ -462,13 +490,15 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"needs-helm-tests": "false",
"run-tests": "true",
"docs-build": "true",
- "skip-pre-commits": "identity,lint-helm-chart,mypy-core,mypy-dev,mypy-docs,ts-compile-format-lint-www",
+ "skip-pre-commits": "identity,lint-helm-chart,mypy-core,mypy-dev,mypy-docs,mypy-providers,ts-compile-format-lint-www",
"run-kubernetes-tests": "false",
"upgrade-to-newer-dependencies": "false",
"run-amazon-tests": "true",
"parallel-test-types-list-as-string": "Always Providers[amazon] "
"Providers[apache.hive,cncf.kubernetes,common.sql,exasol,ftp,http,"
"imap,microsoft.azure,mongo,mysql,openlineage,postgres,salesforce,ssh] Providers[google]",
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow/providers']",
},
id="Providers tests run including amazon tests if amazon provider files changed",
),
@@ -486,10 +516,12 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"run-tests": "true",
"run-amazon-tests": "false",
"docs-build": "false",
- "skip-pre-commits": "identity,lint-helm-chart,mypy-core,mypy-dev,mypy-docs,ts-compile-format-lint-www",
+ "skip-pre-commits": "identity,lint-helm-chart,mypy-core,mypy-dev,mypy-docs,mypy-providers,ts-compile-format-lint-www",
"run-kubernetes-tests": "false",
"upgrade-to-newer-dependencies": "false",
"parallel-test-types-list-as-string": "Always Providers[airbyte,http]",
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow/providers']",
},
id="Providers tests run without amazon tests if no amazon file changed",
),
@@ -509,12 +541,14 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"run-tests": "true",
"run-amazon-tests": "true",
"docs-build": "true",
- "skip-pre-commits": "identity,lint-helm-chart,mypy-core,mypy-dev,mypy-docs,ts-compile-format-lint-www",
+ "skip-pre-commits": "identity,lint-helm-chart,mypy-core,mypy-dev,mypy-docs,mypy-providers,ts-compile-format-lint-www",
"run-kubernetes-tests": "false",
"upgrade-to-newer-dependencies": "false",
"parallel-test-types-list-as-string": "Always Providers[amazon] "
"Providers[apache.hive,cncf.kubernetes,common.sql,exasol,ftp,http,"
"imap,microsoft.azure,mongo,mysql,openlineage,postgres,salesforce,ssh] Providers[google]",
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow/providers']",
},
id="Providers tests run including amazon tests if amazon provider files changed",
),
@@ -537,8 +571,11 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"run-amazon-tests": "false",
"docs-build": "false",
"run-kubernetes-tests": "false",
+ "skip-pre-commits": "identity,lint-helm-chart,mypy-core,mypy-dev,mypy-docs,mypy-providers,ts-compile-format-lint-www",
"upgrade-to-newer-dependencies": "false",
"parallel-test-types-list-as-string": "Always Providers[common.io]",
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow', 'airflow/providers']",
},
id="Only Always and Common.IO tests should run when only common.io and tests/always changed",
),
@@ -578,9 +615,11 @@ def test_expected_output_pull_request_main(
"docs-build": "true",
"docs-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
"full-tests-needed": "true",
- "skip-pre-commits": "identity",
+ "skip-pre-commits": "identity,mypy-core,mypy-dev,mypy-docs,mypy-providers",
"upgrade-to-newer-dependencies": "false",
"parallel-test-types-list-as-string": ALL_CI_SELECTIVE_TEST_TYPES,
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow', 'airflow/providers', 'docs', 'dev']",
},
id="Everything should run including all providers when full tests are needed",
)
@@ -605,9 +644,11 @@ def test_expected_output_pull_request_main(
"docs-build": "true",
"docs-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
"full-tests-needed": "true",
- "skip-pre-commits": "identity",
+ "skip-pre-commits": "identity,mypy-core,mypy-dev,mypy-docs,mypy-providers",
"upgrade-to-newer-dependencies": "false",
"parallel-test-types-list-as-string": ALL_CI_SELECTIVE_TEST_TYPES,
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow', 'airflow/providers', 'docs', 'dev']",
},
id="Everything should run including full providers when full "
"tests are needed even with different label set as well",
@@ -630,9 +671,11 @@ def test_expected_output_pull_request_main(
"docs-build": "true",
"docs-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
"full-tests-needed": "true",
- "skip-pre-commits": "identity",
+ "skip-pre-commits": "identity,mypy-core,mypy-dev,mypy-docs,mypy-providers",
"upgrade-to-newer-dependencies": "false",
"parallel-test-types-list-as-string": ALL_CI_SELECTIVE_TEST_TYPES,
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow', 'airflow/providers', 'docs', 'dev']",
},
id="Everything should run including full providers when"
"full tests are needed even if no files are changed",
@@ -655,14 +698,14 @@ def test_expected_output_pull_request_main(
"docs-build": "true",
"docs-list-as-string": "apache-airflow docker-stack",
"full-tests-needed": "true",
- "skip-pre-commits": "check-airflow-provider-compatibility,"
- "check-extra-packages-references,check-provider-yaml-valid,identity,"
- "lint-helm-chart,mypy-providers",
+ "skip-pre-commits": "check-airflow-provider-compatibility,check-extra-packages-references,check-provider-yaml-valid,identity,lint-helm-chart,mypy-core,mypy-dev,mypy-docs,mypy-providers",
"skip-provider-tests": "true",
"upgrade-to-newer-dependencies": "false",
"parallel-test-types-list-as-string": "API Always BranchExternalPython "
"BranchPythonVenv CLI Core ExternalPython Operators Other PlainAsserts "
"PythonVenv Serialization WWW",
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow', 'docs', 'dev']",
},
id="Everything should run except Providers and lint pre-commit "
"when full tests are needed for non-main branch",
@@ -707,6 +750,8 @@ def test_expected_output_full_tests_needed(
"upgrade-to-newer-dependencies": "false",
"skip-provider-tests": "true",
"parallel-test-types-list-as-string": None,
+ "needs-mypy": "false",
+ "mypy-packages": "[]",
},
id="Nothing should run if only non-important files changed",
),
@@ -735,6 +780,8 @@ def test_expected_output_full_tests_needed(
"upgrade-to-newer-dependencies": "false",
"skip-provider-tests": "true",
"parallel-test-types-list-as-string": "Always",
+ "needs-mypy": "false",
+ "mypy-packages": "[]",
},
id="No Helm tests, No providers no lint charts, should run if "
"only chart/providers changed in non-main but PROD image should be built",
@@ -759,13 +806,13 @@ def test_expected_output_full_tests_needed(
"docs-build": "true",
"docs-list-as-string": "apache-airflow docker-stack",
"full-tests-needed": "false",
- "skip-pre-commits": "check-airflow-provider-compatibility,check-extra-packages-references,"
- "check-provider-yaml-valid,identity,lint-helm-chart,"
- "mypy-dev,mypy-docs,mypy-providers,ts-compile-format-lint-www",
+ "skip-pre-commits": "check-airflow-provider-compatibility,check-extra-packages-references,check-provider-yaml-valid,identity,lint-helm-chart,mypy-core,mypy-dev,mypy-docs,mypy-providers,ts-compile-format-lint-www",
"run-kubernetes-tests": "true",
"upgrade-to-newer-dependencies": "false",
"skip-provider-tests": "true",
"parallel-test-types-list-as-string": "Always CLI",
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow']",
},
id="Only CLI tests and Kubernetes tests should run if cli/chart files changed in non-main branch",
),
@@ -788,11 +835,11 @@ def test_expected_output_full_tests_needed(
"run-kubernetes-tests": "false",
"upgrade-to-newer-dependencies": "false",
"skip-provider-tests": "true",
- "skip-pre-commits": "check-airflow-provider-compatibility,check-extra-packages-references,"
- "check-provider-yaml-valid,identity,lint-helm-chart,"
- "mypy-dev,mypy-docs,mypy-providers,ts-compile-format-lint-www",
+ "skip-pre-commits": "check-airflow-provider-compatibility,check-extra-packages-references,check-provider-yaml-valid,identity,lint-helm-chart,mypy-core,mypy-dev,mypy-docs,mypy-providers,ts-compile-format-lint-www",
"parallel-test-types-list-as-string": "API Always BranchExternalPython BranchPythonVenv "
"CLI Core ExternalPython Operators Other PlainAsserts PythonVenv Serialization WWW",
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow']",
},
id="All tests except Providers and helm lint pre-commit "
"should run if core file changed in non-main branch",
@@ -832,6 +879,8 @@ def test_expected_output_pull_request_v2_7(
"mypy-core,mypy-dev,mypy-docs,mypy-providers,ts-compile-format-lint-www",
"skip-provider-tests": "true",
"parallel-test-types-list-as-string": None,
+ "needs-mypy": "false",
+ "mypy-packages": "[]",
},
id="Nothing should run if only non-important files changed",
),
@@ -847,11 +896,12 @@ def test_expected_output_pull_request_v2_7(
"run-tests": "true",
"docs-build": "true",
"docs-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
- "skip-pre-commits": "check-provider-yaml-valid,identity,lint-helm-chart,"
- "mypy-dev,mypy-docs,mypy-providers,ts-compile-format-lint-www",
+ "skip-pre-commits": "check-provider-yaml-valid,identity,lint-helm-chart,mypy-core,mypy-dev,mypy-docs,mypy-providers,ts-compile-format-lint-www",
"upgrade-to-newer-dependencies": "false",
"skip-provider-tests": "true",
"parallel-test-types-list-as-string": "Always",
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow']",
},
id="Only Always and docs build should run if only system tests changed",
),
@@ -877,7 +927,7 @@ def test_expected_output_pull_request_v2_7(
"cncf.kubernetes common.sql facebook google hashicorp microsoft.azure "
"microsoft.mssql mysql openlineage oracle postgres "
"presto salesforce samba sftp ssh trino",
- "skip-pre-commits": "identity,mypy-dev,mypy-docs,ts-compile-format-lint-www",
+ "skip-pre-commits": "identity,mypy-core,mypy-dev,mypy-docs,mypy-providers,ts-compile-format-lint-www",
"run-kubernetes-tests": "true",
"upgrade-to-newer-dependencies": "false",
"skip-provider-tests": "false",
@@ -885,6 +935,8 @@ def test_expected_output_pull_request_v2_7(
"Providers[apache.beam,apache.cassandra,cncf.kubernetes,common.sql,facebook,hashicorp,"
"microsoft.azure,microsoft.mssql,mysql,openlineage,oracle,postgres,presto,salesforce,"
"samba,sftp,ssh,trino] Providers[google]",
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow', 'airflow/providers']",
},
id="CLI tests and Google-related provider tests should run if cli/chart files changed but "
"prod image should be build too and k8s tests too",
@@ -906,12 +958,13 @@ def test_expected_output_pull_request_v2_7(
"run-tests": "true",
"docs-build": "true",
"docs-list-as-string": "apache-airflow",
- "skip-pre-commits": "check-provider-yaml-valid,identity,lint-helm-chart,mypy-dev,"
- "mypy-docs,mypy-providers,ts-compile-format-lint-www",
+ "skip-pre-commits": "check-provider-yaml-valid,identity,lint-helm-chart,mypy-core,mypy-dev,mypy-docs,mypy-providers,ts-compile-format-lint-www",
"run-kubernetes-tests": "false",
"upgrade-to-newer-dependencies": "false",
"skip-provider-tests": "true",
"parallel-test-types-list-as-string": "API Always CLI Operators WWW",
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow']",
},
id="No providers tests should run if only CLI/API/Operators/WWW file changed",
),
@@ -927,12 +980,13 @@ def test_expected_output_pull_request_v2_7(
"run-tests": "true",
"docs-build": "true",
"docs-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
- "skip-pre-commits": "check-provider-yaml-valid,identity,lint-helm-chart,mypy-dev,"
- "mypy-docs,mypy-providers,ts-compile-format-lint-www",
+ "skip-pre-commits": "check-provider-yaml-valid,identity,lint-helm-chart,mypy-core,mypy-dev,mypy-docs,mypy-providers,ts-compile-format-lint-www",
"run-kubernetes-tests": "false",
"upgrade-to-newer-dependencies": "false",
"skip-provider-tests": "false",
"parallel-test-types-list-as-string": ALL_CI_SELECTIVE_TEST_TYPES,
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow', 'airflow/providers']",
},
id="Tests for all providers should run if model file changed",
),
@@ -948,12 +1002,13 @@ def test_expected_output_pull_request_v2_7(
"run-tests": "true",
"docs-build": "true",
"docs-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
- "skip-pre-commits": "check-provider-yaml-valid,identity,lint-helm-chart,mypy-dev,"
- "mypy-docs,mypy-providers,ts-compile-format-lint-www",
+ "skip-pre-commits": "check-provider-yaml-valid,identity,lint-helm-chart,mypy-core,mypy-dev,mypy-docs,mypy-providers,ts-compile-format-lint-www",
"run-kubernetes-tests": "false",
"upgrade-to-newer-dependencies": "false",
"skip-provider-tests": "false",
"parallel-test-types-list-as-string": ALL_CI_SELECTIVE_TEST_TYPES,
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow', 'airflow/providers']",
},
id="Tests for all providers should run if any other than API/WWW/CLI/Operators file changed.",
),
@@ -990,9 +1045,11 @@ def test_expected_output_pull_request_target(
"run-tests": "true",
"docs-build": "true",
"docs-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
- "skip-pre-commits": "identity",
+ "skip-pre-commits": "identity,mypy-core,mypy-dev,mypy-docs,mypy-providers",
"upgrade-to-newer-dependencies": "true",
"parallel-test-types-list-as-string": ALL_CI_SELECTIVE_TEST_TYPES,
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow', 'airflow/providers', 'docs', 'dev']",
},
id="All tests run on push even if unimportant file changed",
),
@@ -1009,12 +1066,13 @@ def test_expected_output_pull_request_target(
"needs-helm-tests": "false",
"run-tests": "true",
"docs-build": "true",
- "skip-pre-commits": "check-airflow-provider-compatibility,check-extra-packages-references,"
- "check-provider-yaml-valid,identity,lint-helm-chart,mypy-providers",
+ "skip-pre-commits": "check-airflow-provider-compatibility,check-extra-packages-references,check-provider-yaml-valid,identity,lint-helm-chart,mypy-core,mypy-dev,mypy-docs,mypy-providers",
"docs-list-as-string": "apache-airflow docker-stack",
"upgrade-to-newer-dependencies": "true",
"parallel-test-types-list-as-string": "API Always BranchExternalPython BranchPythonVenv "
"CLI Core ExternalPython Operators Other PlainAsserts PythonVenv Serialization WWW",
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow', 'docs', 'dev']",
},
id="All tests except Providers and Helm run on push"
" even if unimportant file changed in non-main branch",
@@ -1032,10 +1090,12 @@ def test_expected_output_pull_request_target(
"needs-helm-tests": "true",
"run-tests": "true",
"docs-build": "true",
- "skip-pre-commits": "identity",
+ "skip-pre-commits": "identity,mypy-core,mypy-dev,mypy-docs,mypy-providers",
"docs-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
"upgrade-to-newer-dependencies": "true",
"parallel-test-types-list-as-string": ALL_CI_SELECTIVE_TEST_TYPES,
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow', 'airflow/providers', 'docs', 'dev']",
},
id="All tests run on push if core file changed",
),
@@ -1084,11 +1144,13 @@ def test_no_commit_provided_trigger_full_build_for_any_event_type(github_event):
"needs-helm-tests": "true",
"run-tests": "true",
"docs-build": "true",
- "skip-pre-commits": "identity",
+ "skip-pre-commits": "identity,mypy-core,mypy-dev,mypy-docs,mypy-providers",
"upgrade-to-newer-dependencies": "true"
if github_event in [GithubEvents.PUSH, GithubEvents.SCHEDULE]
else "false",
"parallel-test-types-list-as-string": ALL_CI_SELECTIVE_TEST_TYPES,
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow', 'airflow/providers', 'docs', 'dev']",
},
str(stderr),
)
@@ -1577,3 +1639,91 @@ def test_provider_compatibility_checks(labels: tuple[str, ...], expected_outputs
default_branch="main",
)
assert_outputs_are_printed(expected_outputs, str(stderr))
+
+
+@pytest.mark.parametrize(
+ "files, expected_outputs, default_branch, pr_labels",
+ [
+ pytest.param(
+ ("README.md",),
+ {
+ "needs-mypy": "false",
+ "mypy-packages": "[]",
+ },
+ "main",
+ (),
+ id="No mypy checks on non-python files",
+ ),
+ pytest.param(
+ ("airflow/cli/file.py",),
+ {
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow']",
+ },
+ "main",
+ (),
+ id="Airflow mypy checks on airflow regular files",
+ ),
+ pytest.param(
+ ("airflow/models/file.py",),
+ {
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow', 'airflow/providers']",
+ },
+ "main",
+ (),
+ id="Airflow mypy checks on airflow files that can trigger provider tests",
+ ),
+ pytest.param(
+ ("airflow/providers/a_file.py",),
+ {
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow/providers']",
+ },
+ "main",
+ (),
+ id="Airflow mypy checks on provider files",
+ ),
+ pytest.param(
+ ("docs/a_file.py",),
+ {
+ "needs-mypy": "true",
+ "mypy-packages": "['docs']",
+ },
+ "main",
+ (),
+ id="Doc checks on doc files",
+ ),
+ pytest.param(
+ ("dev/a_package/a_file.py",),
+ {
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow', 'airflow/providers', 'docs', 'dev']",
+ },
+ "main",
+ (),
+ id="All mypy checks on def files changed (full tests needed are implicit)",
+ ),
+ pytest.param(
+ ("readme.md",),
+ {
+ "needs-mypy": "true",
+ "mypy-packages": "['airflow', 'airflow/providers', 'docs', 'dev']",
+ },
+ "main",
+ ("full tests needed",),
+ id="All mypy checks on full tests needed",
+ ),
+ ],
+)
+def test_mypy_matches(
+ files: tuple[str, ...], expected_outputs: dict[str, str], default_branch: str, pr_labels: tuple[str, ...]
+):
+ stderr = SelectiveChecks(
+ files=files,
+ commit_ref="HEAD",
+ default_branch=default_branch,
+ github_event=GithubEvents.PULL_REQUEST,
+ pr_labels=pr_labels,
+ )
+ assert_outputs_are_printed(expected_outputs, str(stderr))
diff --git a/images/breeze/output_static-checks.svg b/images/breeze/output_static-checks.svg
index e601050d09..f5aa3bba12 100644
--- a/images/breeze/output_static-checks.svg
+++ b/images/breeze/output_static-checks.svg
@@ -343,18 +343,18 @@
</text><text class="breeze-static-checks-r5" x="0" y="922.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-37)">│</text><text class="breeze-static-checks-r7" x="451.4" y="922.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-37)">doctoc | end-of-file-fixer | fix-encoding-pragma | flynt |                       </t [...]
</text><text class="breeze-static-checks-r5" x="0" y="947.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-38)">│</text><text class="breeze-static-checks-r7" x="451.4" y="947.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-38)">generate-airflow-diagrams | generate-pypi-readme | identity | insert-license |   </text><text class="breeze-static-checks-r5" x="1451.8" y="947.2" textLength="12.2" clip-path="url(#bre [...]
</text><text class="breeze-static-checks-r5" x="0" y="971.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-39)">│</text><text class="breeze-static-checks-r7" x="451.4" y="971.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-39)">lint-chart-schema | lint-css | lint-dockerfile | lint-helm-chart |               </text><text class="breeze-static-checks-r5 [...]
-</text><text class="breeze-static-checks-r5" x="0" y="996" textLength="12.2" clip-path="url(#breeze-static-checks-line-40)">│</text><text class="breeze-static-checks-r7" x="451.4" y="996" textLength="988.2" clip-path="url(#breeze-static-checks-line-40)">lint-json-schema | lint-markdown | lint-openapi | mixed-line-ending | mypy-core |</text><text class="breeze-static-checks-r5" x="1451.8" y="996" textLength="12.2" clip-path="url(#breeze-static- [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1020.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-41)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1020.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-41)">mypy-dev | mypy-docs | mypy-providers | pretty-format-json | python-no-log-warn |</text><text class="breeze-static-checks-r5" x="1451.8" y="1020.4" textLength="12.2" clip-path="url(#breez [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1044.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-42)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1044.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-42)">replace-bad-characters | rst-backticks | ruff | ruff-format | shellcheck |       </text><text class="breeze-static-checks-r5" x="1451.8" y="1044.8" text [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1069.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-43)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1069.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-43)">trailing-whitespace | ts-compile-format-lint-www | update-black-version |        </text><text class="breeze-static-checks-r5" x="1451.8" y="1069.2" textLength="12.2" c [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1093.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-44)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1093.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-44)">update-breeze-cmd-output | update-breeze-readme-config-hash |                    </text><text class="breeze-static-ch [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1118" textLength="12.2" clip-path="url(#breeze-static-checks-line-45)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1118" textLength="988.2" clip-path="url(#breeze-static-checks-line-45)">update-common-sql-api-stubs | update-er-diagram | update-extras |                </text><text class="breeze-static-checks-r5" x="14 [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1142.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-46)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1142.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-46)">update-in-the-wild-to-be-sorted | update-inlined-dockerfile-scripts |            </text><text class="breeze-static-checks-r5" x="1451.8" y="1142.4" textLengt [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1166.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-47)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1166.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-47)">update-installed-providers-to-be-sorted | update-local-yml-file |                </text><text class="breeze-static-checks-r5" x="1451.8" [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1191.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-48)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1191.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-48)">update-migration-references | update-providers-dependencies |                    </text><text class="breeze-static-ch [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1215.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-49)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1215.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-49)">update-spelling-wordlist-to-be-sorted | update-supported-versions |              </text><text class="breeze-static-checks-r5" x="1451.8" y="1215.6" [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1240" textLength="12.2" clip-path="url(#breeze-static-checks-line-50)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1240" textLength="988.2" clip-path="url(#breeze-static-checks-line-50)">update-vendored-in-k8s-json-schema | update-version | validate-pyproject |       </text><text class="breeze-static-checks-r5" x="1451.8" y="1240" textLength="12.2" clip-path="u [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1264.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-51)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1264.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-51)">yamllint)                                     & [...]
+</text><text class="breeze-static-checks-r5" x="0" y="996" textLength="12.2" clip-path="url(#breeze-static-checks-line-40)">│</text><text class="breeze-static-checks-r7" x="451.4" y="996" textLength="988.2" clip-path="url(#breeze-static-checks-line-40)">lint-json-schema | lint-markdown | lint-openapi | mixed-line-ending | mypy |     </text><text class="breeze-static-checks-r5" x="1451.8" y="996" textLength="12.2" clip- [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1020.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-41)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1020.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-41)">mypy-core | mypy-dev | mypy-docs | mypy-providers | pretty-format-json |         </text><text class="breeze-static-checks-r5" x="1451.8" y="10 [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1044.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-42)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1044.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-42)">python-no-log-warn | replace-bad-characters | rst-backticks | ruff | ruff-format </text><text class="breeze-static-checks-r5" x="1451.8" y="1044.8" textLength="12.2" clip-path="url(#breez [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1069.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-43)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1069.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-43)">| shellcheck | trailing-whitespace | ts-compile-format-lint-www |                </text><text class="breeze-static-checks- [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1093.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-44)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1093.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-44)">update-black-version | update-breeze-cmd-output |                             [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1118" textLength="12.2" clip-path="url(#breeze-static-checks-line-45)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1118" textLength="988.2" clip-path="url(#breeze-static-checks-line-45)">update-breeze-readme-config-hash | update-common-sql-api-stubs |                 </text><text class="breeze-static-checks-r5" x="1451.8" [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1142.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-46)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1142.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-46)">update-er-diagram | update-extras | update-in-the-wild-to-be-sorted |            </text><text class="breeze-static-checks-r5" x="1451.8" y="1142.4" [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1166.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-47)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1166.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-47)">update-inlined-dockerfile-scripts | update-installed-providers-to-be-sorted |    </text><text class="breeze-static-checks-r5" x="1451.8" y="1166.8" textLength="12.2" clip-path="url(#breeze-static-c [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1191.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-48)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1191.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-48)">update-local-yml-file | update-migration-references |                           &# [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1215.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-49)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1215.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-49)">update-providers-dependencies | update-spelling-wordlist-to-be-sorted |          </text><text class="breeze-static-checks-r5" x="1451.8" y="1215.6" textLength="12.2" c [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1240" textLength="12.2" clip-path="url(#breeze-static-checks-line-50)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1240" textLength="988.2" clip-path="url(#breeze-static-checks-line-50)">update-supported-versions | update-vendored-in-k8s-json-schema | update-version |</text><text class="breeze-static-checks-r5" x="1451.8" y="1240" textLength="12.2" clip-path="url(#breeze-static-checks-line-50)"> [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1264.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-51)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1264.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-51)">validate-pyproject | yamllint)                                 [...]
</text><text class="breeze-static-checks-r5" x="0" y="1288.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-52)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1288.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-52)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1288.8" textLength="61" clip-path="url(#breeze-static-checks-line-52)">-show</text><text class="breeze-static-checks-r4" x="97.6" y="1288.8" textLength="195.2" clip-path="url(# [...]
</text><text class="breeze-static-checks-r5" x="0" y="1313.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-53)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1313.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-53)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1313.2" textLength="134.2" clip-path="url(#breeze-static-checks-line-53)">-initialize</text><text class="breeze-static-checks-r4" x="170.8" y="1313.2" textLength="146.4" clip-p [...]
</text><text class="breeze-static-checks-r5" x="0" y="1337.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-54)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1337.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-54)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1337.6" textLength="48.8" clip-path="url(#breeze-static-checks-line-54)">-max</text><text class="breeze-static-checks-r4" x="85.4" y="1337.6" textLength="292.8" clip-path="url( [...]
diff --git a/images/breeze/output_static-checks.txt b/images/breeze/output_static-checks.txt
index 00157e7cd8..d135c297d0 100644
--- a/images/breeze/output_static-checks.txt
+++ b/images/breeze/output_static-checks.txt
@@ -1 +1 @@
-65040ded8dc5fc072fa49219f88feb36
+3131ead122c4ff23924ceef326f7ce04
diff --git a/scripts/ci/pre_commit/common_precommit_utils.py b/scripts/ci/pre_commit/common_precommit_utils.py
index 76370b15ad..8926bc1823 100644
--- a/scripts/ci/pre_commit/common_precommit_utils.py
+++ b/scripts/ci/pre_commit/common_precommit_utils.py
@@ -45,12 +45,32 @@ def read_airflow_version() -> str:
raise RuntimeError("Couldn't find __version__ in AST")
-def filter_out_providers_on_non_main_branch(files: list[str]) -> list[str]:
- """When running build on non-main branch do not take providers into account"""
+def pre_process_files(files: list[str]) -> list[str]:
+ """Pre-process files passed to mypy.
+
+ * When running build on non-main branch do not take providers into account.
+ * When running "airflow/providers" package, then we need to add --namespace-packages flag.
+ * When running "airflow" package, then we need to exclude providers.
+ """
default_branch = os.environ.get("DEFAULT_BRANCH")
if not default_branch or default_branch == "main":
return files
- return [file for file in files if not file.startswith(f"airflow{os.sep}providers")]
+ result = [file for file in files if not file.startswith(f"airflow{os.sep}providers")]
+ if "airflow/providers" in files:
+ if len(files) > 1:
+ raise RuntimeError(
+ "When running `airflow/providers` package, you cannot run any other packages because only "
+ "airflow/providers package requires --namespace-packages flag to be set"
+ )
+ result.append("--namespace-packages")
+ if "airflow" in files:
+ if len(files) > 1:
+ raise RuntimeError(
+ "When running `airflow` package, you cannot run any other packages because only "
+ "airflow/providers package requires --exclude airflow/providers/.* flag to be set"
+ )
+ result.extend(["--exclude", "airflow/providers/.*"])
+ return result
def insert_documentation(file_path: Path, content: list[str], header: str, footer: str):
diff --git a/scripts/ci/pre_commit/pre_commit_mypy.py b/scripts/ci/pre_commit/pre_commit_mypy.py
index 61fc4baede..8f38f27ac4 100755
--- a/scripts/ci/pre_commit/pre_commit_mypy.py
+++ b/scripts/ci/pre_commit/pre_commit_mypy.py
@@ -18,6 +18,7 @@
from __future__ import annotations
import os
+import shlex
import sys
from pathlib import Path
@@ -25,15 +26,18 @@ sys.path.insert(0, str(Path(__file__).parent.resolve()))
from common_precommit_utils import (
console,
- filter_out_providers_on_non_main_branch,
initialize_breeze_precommit,
+ pre_process_files,
run_command_via_breeze_shell,
)
initialize_breeze_precommit(__name__, __file__)
-files_to_test = filter_out_providers_on_non_main_branch(sys.argv[1:])
-if files_to_test == ["--namespace-packages"]:
+files_to_test = pre_process_files(sys.argv[1:])
+mypy_packages = os.environ.get("MYPY_PACKAGES")
+if mypy_packages:
+ files_to_test += shlex.split(mypy_packages)
+if files_to_test == ["--namespace-packages"] or files_to_test == []:
print("No files to tests. Quitting")
sys.exit(0)
@@ -51,19 +55,26 @@ res = run_command_via_breeze_shell(
"MOUNT_SOURCES": "selected",
},
)
+ci_environment = os.environ.get("CI")
if res.returncode != 0:
+ if mypy_packages and ci_environment:
+ console.print(
+ "[yellow]You are running mypy with the packages selected. If you want to"
+ "reproduce it locally, you need to run the following command:\n"
+ )
+ console.print(
+ f'MYPY_PACKAGES="{mypy_packages}" pre-commit run --hook-stage manual mypy --all-files\n'
+ )
upgrading = os.environ.get("UPGRADE_TO_NEWER_DEPENDENCIES", "false") != "false"
if upgrading:
console.print(
- "[yellow]You are running mypy with the image that has dependencies upgraded automatically."
+ "[yellow]You are running mypy with the image that has dependencies upgraded automatically.\n"
)
flag = " --upgrade-to-newer-dependencies" if upgrading else ""
console.print(
- "[yellow]If you see strange stacktraces above, "
- f"run `breeze ci-image build --python 3.8{flag}` and try again. "
- "You can also run `breeze down --cleanup-mypy-cache` to clean up the cache used. "
- "Still sometimes diff heuristic in mypy is behaving abnormal, to double check you can "
- "call `breeze static-checks --type mypy-[dev|core|providers|docs] --all-files` "
- 'and then commit via `git commit --no-verify -m "commit message"`. CI will do a full check.'
+ "[yellow]If you see strange stacktraces above, and can't reproduce it, please run"
+ " this command and try again:\n"
)
+ console.print(f"breeze ci-image build --python 3.8{flag}\n")
+ console.print("[yellow]You can also run `breeze down --cleanup-mypy-cache` to clean up the cache used.\n")
sys.exit(res.returncode)
(airflow) 10/20: Bump min version of mysql-connector-python (#36668)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 8344febc9855c268df820d7758e8df2642c0800a
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Mon Jan 8 21:30:44 2024 +0100
Bump min version of mysql-connector-python (#36668)
As part of extraction from #36537 I noticed that mysql-connector-python
causes also a lot of backtracking.
This PR bumps minimum version of the mysql-connection-python from
REALLY OLD (2018) to JUST OLD (2022).
(cherry picked from commit c29632aab4cf13a861bfffcbd11f1c9964d8910a)
---
airflow/providers/mysql/provider.yaml | 2 +-
generated/provider_dependencies.json | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/airflow/providers/mysql/provider.yaml b/airflow/providers/mysql/provider.yaml
index 9712ec8676..22d1f364ef 100644
--- a/airflow/providers/mysql/provider.yaml
+++ b/airflow/providers/mysql/provider.yaml
@@ -58,7 +58,7 @@ dependencies:
- apache-airflow>=2.6.0
- apache-airflow-providers-common-sql>=1.3.1
- mysqlclient>=1.3.6
- - mysql-connector-python>=8.0.11
+ - mysql-connector-python>=8.0.29
integrations:
- integration-name: MySQL
diff --git a/generated/provider_dependencies.json b/generated/provider_dependencies.json
index c4eb6d64dc..3751cb0811 100644
--- a/generated/provider_dependencies.json
+++ b/generated/provider_dependencies.json
@@ -691,7 +691,7 @@
"deps": [
"apache-airflow-providers-common-sql>=1.3.1",
"apache-airflow>=2.6.0",
- "mysql-connector-python>=8.0.11",
+ "mysql-connector-python>=8.0.29",
"mysqlclient>=1.3.6"
],
"cross-providers-deps": [
(airflow) 17/20: Standardize airflow build process and switch to Hatchling build backend (#36537)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit f3d81268123b87bfec223dcefb91724d5ef0bcf5
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Wed Jan 10 21:19:02 2024 +0100
Standardize airflow build process and switch to Hatchling build backend (#36537)
This PR changes Airflow installation and build backend to use new
standard Python ways of building Python applications.
We've been trying to do it for quite a while. Airflow tranditionally
has been using complex and convoluted build process based on
setuptools and (extremely) custom setup.py file. It survived
migration to Airflow 2.0 and splitting Airlfow monorepo into
Airflow and Providers, adding pre-installed providers and switching
providers to use flit (and follow build standards).
So far tooling in Python ecosystme had not been able to fuflill our
needs and we refrained to develop our own tooling, but finally with
appearance of Hatch (managed by Python Packaging Authority) and
few recent advancements there we are finally able to swtich to
Python standard ways of managing project dependnecy configuration
and project build setup (with a few customizations).
This PR makes airflow build process follow those standard PEPs:
* Airflow has all build configuration stored in pyproject.toml
following PEP 518 which allows any fronted (`pip`, `poetry`,
`hatch`, `flit`, or whatever other frontend is used to
install required build dependendencies to install Airflow
locally and to build distribution pacakges (sdist/wheel)
* Hatchling backend follows PEP 517 for standard source tree and build
backend implementation that allows to execute the build in a
frontend-independent way
* We store all project metadata in pyprooject.toml - following
PEP 621 where all necessary project metadata components were
defined.
* We plug-in into Hatchling "editable build" hooks following
PEP 660. Hatchling internally builds editable wheel that
is used as ephemeral step and communication between backend
and frontend (and this ephemeral wheel is used to make
editable installation of the projeect - suitable for fast
iteration of code without reinstalling the package.
With Airflow having many provider packages in single source tree
where we want to be able to install and develop airflow and
providers together, this is not a small feat to implement the
case wher editable installation has to behave quite a bit
differently when it comes to packaging and dependencies for
editable install (when you want to edit sources directly) and
installable package (where you want to have separate Airflow
package and provider packages). Fortunately the standardisation
efforts in the Python Packaging community and tooling implementing
it had finally made it possible.
Some of the important ways bow this has been achieved:
* We continue using provider.yaml in providers as the single source
of trutgh for per-provider dependencies. We added a possibility
to specify "devel-dependencies" in provider.yaml so that all
per-provider dependencies in `generated/provider_dependencies.json`
and `pyproject.toml` are generated from those dependencies via
update-providers-dependencies pre-commit.
* Pyproject.toml is generally managed manually, but the part where
provider dependencies and bundle dependencies are used is
automatically updated by a pre-commit whenever provider
dependencies change. Those generated provider dependencies contain
just dependencies of providers - not the provider packages, but
in the final "standard" wheel file they are replaced with
"apache-airflow-providers-PROVIDER" dependencies - so that the
wheel package will only install the provider and use the
dependencies of that version of provider it installs.
* We are utilising custom hatchiling build hooks (PEP 660 standard)
that allow to modify 'standard' wheel package on-the-fly when
the wheel is being prepared by adding preinstalled package
dependencies (which are not needed in editable build) and by
removing all devel extras (that are not needed in the PyPI
distributed wheel package). This allows to solve the conundrum
of having different "editable" and "standard" behaviour while
keeping the same project specification in pyproject.toml.
* We added description of how `Hatch` can be employed as build
frontend in order to manage local virtualenv and install Airflow
in editable way easily - while keeping all properties of the
installed application (including working airflow cli and
package metadata discovery) as well as how to use PEP-standard
ways of bulding wheel and sdist packages.
* We have a custom step (following PEP-standards) to inject
airflow-specific build steps - compiling www assets and
generating git commit hash version to display it in the UI
* We also show how all this makes it possible to make it easy to
manage local virtualenvs and editable installations for Airflow
contributors - without vendor lock-in of the build tools as
by following standard PEPs Airflow can be locally and editably
installed by anyone using any build front-end tools following
the standards - whether you use `pip`, `poetry`, `Hatch`, `flit`
or any other frontent build tools, Airflow local installation
and package building will work the same way for all of them,
where both "editable" and "standard" package prepration is
managed by `hatchling` backend in the same way.
* Previously our extras contained a "." which is not normalized
name for extras - `pip` and other tools replaced it automatically
with `_'. This change updates the extra names to contain
'-' rather than '.' in the name, following PEP-685. This should be
fully backwards compatible, users will still be able to use "." but it
will be normalized to "-" in Airflow packages. This is also future
proof as it is expected that all package managers and tools
will eventually use PEP-685 applied to extras, even if currently
some of the tools (pip + setuptools) might generate warnings.
* Additionally, this change organizes the documentation around
the extras and dependencies, explaining the reasoning behind
all the different extras we have.
* As a bonus (and this is what we used to test it all) we are
documenting how to use Hatch frontend to:
* manage multiple Python installations
* manage multiple Pythob virtualenv environments
* build Airflow packages for release management
(cherry picked from commit c439ab87c421aaa6bd5d8074780e4f63606a1ef1)
---
.dockerignore | 3 -
.github/actions/build-prod-images/action.yml | 2 +-
.github/workflows/ci.yml | 39 +-
.gitignore | 7 +-
.pre-commit-config.yaml | 45 +-
.rat-excludes | 5 +-
.../LICENSE-bootstrap.txt | 0
.../LICENSE-bootstrap3-typeahead.txt | 0
.../LICENSE-d3-shape.txt | 0
.../LICENSE-d3-tip.txt | 0
{licenses => 3rd-party-licenses}/LICENSE-d3js.txt | 0
.../LICENSE-dagre-d3.txt | 0
.../LICENSE-datatables.txt | 0
.../LICENSE-elasticmock.txt | 0
.../LICENSE-eonasdan-bootstrap-datetimepicker.txt | 0
.../LICENSE-flask-kerberos.txt | 0
{licenses => 3rd-party-licenses}/LICENSE-hue.txt | 0
.../LICENSE-jqclock.txt | 0
.../LICENSE-jquery.txt | 0
.../LICENSE-moment.txt | 0
.../LICENSE-normalize.txt | 0
.../LICENSE-pytest-capture-warnings.txt | 0
.../LICENSE-unicodecsv.txt | 0
BREEZE.rst | 22 +-
CI.rst | 4 +-
CONTRIBUTING.rst | 146 ++-
CONTRIBUTORS_QUICK_START.rst | 5 +-
Dockerfile | 19 +-
Dockerfile.ci | 43 +-
IMAGES.rst | 2 +-
INSTALL | 342 ++++--
LOCAL_VIRTUALENV.rst | 337 +++--
MANIFEST.in | 43 -
README.md | 4 +-
STATIC_CODE_CHECKS.rst | 8 +-
airflow/_vendor/README.md | 2 +-
airflow/cli/cli_parser.py | 9 +-
airflow/configuration.py | 2 +-
airflow/provider.yaml.schema.json | 7 +
airflow/providers/MANAGING_PROVIDERS_LIFECYCLE.rst | 83 +-
airflow/providers/amazon/provider.yaml | 50 +-
airflow/providers/apache/beam/provider.yaml | 10 +-
airflow/providers/apache/hive/provider.yaml | 2 +-
airflow/providers/apache/impala/provider.yaml | 11 +-
airflow/providers/apache/spark/provider.yaml | 10 +-
airflow/providers/celery/provider.yaml | 11 +-
airflow/providers/databricks/provider.yaml | 17 +-
airflow/providers/google/provider.yaml | 40 +-
airflow/providers/microsoft/azure/provider.yaml | 3 +
airflow/providers/mongo/provider.yaml | 3 +
airflow/providers/mysql/provider.yaml | 10 +-
airflow/providers/oracle/provider.yaml | 11 +-
airflow/providers/postgres/provider.yaml | 10 +-
airflow/providers/tabular/provider.yaml | 3 +
airflow/providers_manager.py | 8 +
airflow/www/webpack.config.js | 2 +-
clients/gen/common.sh | 3 +-
...UALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md | 7 +-
dev/README.md | 11 +-
dev/README_RELEASE_AIRFLOW.md | 6 +-
dev/README_RELEASE_PROVIDER_PACKAGES.md | 38 +-
dev/airflow_pre_installed_providers.txt | 7 +
dev/breeze/README.md | 2 +-
dev/breeze/SELECTIVE_CHECKS.md | 29 +-
.../adr/0003-bootstrapping-virtual-environment.md | 4 +-
dev/breeze/pyproject.toml | 3 +-
.../airflow_breeze/commands/ci_image_commands.py | 4 -
.../commands/ci_image_commands_config.py | 1 -
.../commands/common_image_options.py | 6 -
.../airflow_breeze/commands/developer_commands.py | 12 +-
.../commands/developer_commands_config.py | 2 +-
.../commands/production_image_commands.py | 4 -
.../commands/production_image_commands_config.py | 1 -
.../commands/release_candidate_command.py | 5 +-
.../commands/release_management_commands.py | 165 ++-
.../commands/release_management_commands_config.py | 1 -
dev/breeze/src/airflow_breeze/global_constants.py | 11 +-
.../src/airflow_breeze/params/build_ci_params.py | 2 +-
.../src/airflow_breeze/params/build_prod_params.py | 1 -
.../airflow_breeze/params/common_build_params.py | 1 -
.../src/airflow_breeze/params/shell_params.py | 2 -
dev/breeze/src/airflow_breeze/pre_commit_ids.py | 3 +-
.../prepare_providers/provider_documentation.py | 2 +-
.../prepare_providers/provider_packages.py | 3 +-
.../airflow_breeze/utils/docker_command_utils.py | 5 +-
.../src/airflow_breeze/utils/kubernetes_utils.py | 1 -
dev/breeze/src/airflow_breeze/utils/packages.py | 12 +-
.../src/airflow_breeze/utils/selective_checks.py | 7 +-
dev/breeze/tests/test_selective_checks.py | 16 +-
dev/hatch_build.py | 176 +++
.../prod_image_installed_providers.txt | 1 +
dev/refresh_images.sh | 2 +-
dev/sign.sh | 4 +-
docker_tests/docker_tests_utils.py | 3 +-
docker_tests/test_prod_image.py | 37 +-
.../apache-airflow-providers-apache-hive/index.rst | 2 +-
.../howto/create-custom-providers.rst | 36 +-
.../modules_management.rst | 54 +-
docs/apache-airflow/extra-packages-ref.rst | 200 +--
.../installation/installing-from-pypi.rst | 6 +-
docs/docker-stack/build-arg-ref.rst | 16 +-
docs/spelling_wordlist.txt | 2 +
generated/PYPI_README.md | 2 +-
generated/provider_dependencies.json | 108 +-
images/breeze/output_ci-image_build.svg | 118 +-
images/breeze/output_ci-image_build.txt | 2 +-
images/breeze/output_prod-image_build.svg | 148 ++-
images/breeze/output_prod-image_build.txt | 2 +-
images/breeze/output_release-management.svg | 2 +-
images/breeze/output_release-management.txt | 2 +-
...put_release-management_generate-constraints.svg | 2 +-
...put_release-management_generate-constraints.txt | 2 +-
..._release-management_prepare-airflow-package.svg | 40 +-
..._release-management_prepare-airflow-package.txt | 2 +-
images/breeze/output_start-airflow.svg | 8 +-
images/breeze/output_start-airflow.txt | 2 +-
images/breeze/output_static-checks.svg | 114 +-
images/breeze/output_static-checks.txt | 2 +-
newsfragments/36537.significant.rst | 41 +
pyproject.toml | 1288 +++++++++++++++++++-
scripts/ci/docker-compose/devcontainer.env | 1 -
scripts/ci/docker-compose/local.yml | 9 -
scripts/ci/kubernetes/k8s_requirements.txt | 2 +-
.../pre_commit_check_extra_packages_ref.py | 97 ++
.../pre_commit_check_order_dockerfile_extras.py | 2 +-
.../pre_commit_check_order_pyproject_toml.py | 103 ++
.../ci/pre_commit/pre_commit_check_order_setup.py | 135 --
.../pre_commit_check_setup_extra_packages_ref.py | 260 ----
.../ci/pre_commit/pre_commit_compile_www_assets.py | 18 +-
scripts/ci/pre_commit/pre_commit_insert_extras.py | 91 +-
.../pre_commit_sort_installed_providers.py | 15 +-
.../pre_commit_update_providers_dependencies.py | 171 ++-
..._commit_update_providers_dependencies.py.md5sum | 1 +
scripts/docker/entrypoint_ci.sh | 2 +-
scripts/docker/install_airflow.sh | 7 +-
...install_airflow_dependencies_from_branch_tip.sh | 4 +-
scripts/in_container/run_generate_constraints.py | 38 +-
.../in_container/run_prepare_airflow_packages.py | 192 +--
scripts/in_container/verify_providers.py | 11 +-
scripts/tools/initialize_virtualenv.py | 9 +-
setup.cfg | 244 ----
setup.py | 1084 ----------------
tests/cli/commands/test_task_command.py | 10 +-
tests/plugins/test_plugins_manager.py | 3 +-
.../providers/google/cloud/utils/base_gcp_mock.py | 4 +-
145 files changed, 3648 insertions(+), 3046 deletions(-)
diff --git a/.dockerignore b/.dockerignore
index 803b75ea80..5240d438d7 100644
--- a/.dockerignore
+++ b/.dockerignore
@@ -49,7 +49,6 @@
!.dockerignore
!RELEASE_NOTES.rst
!LICENSE
-!MANIFEST.in
!NOTICE
!.github
!empty
@@ -69,8 +68,6 @@
!.bash_completion.d
# Setup/version configuration
-!setup.cfg
-!setup.py
!pyproject.toml
!manifests
!generated
diff --git a/.github/actions/build-prod-images/action.yml b/.github/actions/build-prod-images/action.yml
index 14a5aa8de9..b58aa3a2ba 100644
--- a/.github/actions/build-prod-images/action.yml
+++ b/.github/actions/build-prod-images/action.yml
@@ -41,7 +41,7 @@ runs:
shell: bash
run: >
breeze release-management prepare-provider-packages
- --package-list-file ./airflow/providers/installed_providers.txt
+ --package-list-file ./dev/prod_image_installed_providers.txt
--package-format wheel --version-suffix-for-pypi dev0
if: ${{ inputs.build-provider-packages == 'true' }}
- name: "Prepare chicken-eggs provider packages"
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index f321ec2619..6ab4213153 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -195,7 +195,7 @@ jobs:
# Push early BuildX cache to GitHub Registry in Apache repository, This cache does not wait for all the
# tests to complete - it is run very early in the build process for "main" merges in order to refresh
- # cache using the current constraints. This will speed up cache refresh in cases when setup.py
+ # cache using the current constraints. This will speed up cache refresh in cases when pyproject.toml
# changes or in case of Dockerfile changes. Failure in this step is not a problem (at most it will
# delay cache refresh. It does not attempt to upgrade to newer dependencies.
# We only push CI cache as PROD cache usually does not gain as much from fresh cache because
@@ -489,7 +489,7 @@ jobs:
# And when we prepare them from sources they will have apache-airflow>=X.Y.Z.dev0
shell: bash
run: >
- breeze release-management prepare-provider-packages
+ breeze release-management prepare-provider-packages --include-not-ready-providers
--package-format wheel --version-suffix-for-pypi dev0
${{ needs.build-info.outputs.chicken-egg-providers }}
if: needs.build-info.outputs.chicken-egg-providers != ''
@@ -681,9 +681,9 @@ jobs:
id: cache-doc-inventories
with:
path: ./docs/_inventory_cache/
- key: docs-inventory-${{ hashFiles('setup.py','setup.cfg','pyproject.toml;') }}
+ key: docs-inventory-${{ hashFiles('pyproject.toml;') }}
restore-keys: |
- docs-inventory-${{ hashFiles('setup.py','setup.cfg','pyproject.toml;') }}
+ docs-inventory-${{ hashFiles('pyproject.toml;') }}
docs-inventory-
- name: "Build docs"
run: >
@@ -745,9 +745,9 @@ jobs:
id: cache-doc-inventories
with:
path: ./docs/_inventory_cache/
- key: docs-inventory-${{ hashFiles('setup.py','setup.cfg','pyproject.toml;') }}
+ key: docs-inventory-${{ hashFiles('pyproject.toml;') }}
restore-keys: |
- docs-inventory-${{ hashFiles('setup.py','setup.cfg','pyproject.toml;') }}
+ docs-inventory-${{ hashFiles('pyproject.toml;') }}
docs-inventory-
- name: "Spellcheck docs"
run: >
@@ -776,11 +776,13 @@ jobs:
run: rm -fv ./dist/*
- name: "Prepare provider documentation"
run: >
- breeze release-management prepare-provider-documentation --non-interactive
+ breeze release-management prepare-provider-documentation --include-not-ready-providers
+ --non-interactive
${{ needs.build-info.outputs.affected-providers-list-as-string }}
- name: "Prepare provider packages: wheel"
run: >
- breeze release-management prepare-provider-packages --version-suffix-for-pypi dev0
+ breeze release-management prepare-provider-packages --include-not-ready-providers
+ --version-suffix-for-pypi dev0
--package-format wheel ${{ needs.build-info.outputs.affected-providers-list-as-string }}
- name: "Prepare airflow package: wheel"
run: breeze release-management prepare-airflow-package --version-suffix-for-pypi dev0
@@ -849,7 +851,7 @@ jobs:
run: rm -fv ./dist/*
- name: "Prepare provider packages: sdist"
run: >
- breeze release-management prepare-provider-packages
+ breeze release-management prepare-provider-packages --include-not-ready-providers
--version-suffix-for-pypi dev0 --package-format sdist
${{ needs.build-info.outputs.affected-providers-list-as-string }}
- name: "Prepare airflow package: sdist"
@@ -916,7 +918,7 @@ jobs:
run: rm -fv ./dist/*
- name: "Prepare provider packages: wheel"
run: >
- breeze release-management prepare-provider-packages
+ breeze release-management prepare-provider-packages --include-not-ready-providers
--package-format wheel ${{ needs.build-info.outputs.affected-providers-list-as-string }}
- name: >
Remove incompatible Airflow
@@ -925,17 +927,9 @@ jobs:
rm -vf ${{ matrix.remove-providers }}
working-directory: ./dist
if: matrix.remove-providers != ''
- - name: "Checkout ${{matrix.airflow-version}} of Airflow"
- uses: actions/checkout@v4
- with:
- persist-credentials: false
- ref: ${{matrix.airflow-version}}
- path: old-airflow
- - name: "Prepare airflow package: wheel"
+ - name: "Download airflow package: wheel"
run: |
- pip install pip==23.3.2 wheel==0.36.2 gitpython==3.1.40
- python setup.py egg_info --tag-build ".dev0" bdist_wheel -d ../dist
- working-directory: ./old-airflow
+ pip download "apache-airflow==${{matrix.airflow-version}}" -d dist --no-deps
- name: >
Install and verify all provider packages and airflow on
Airflow ${{matrix.airflow-version}}:Python ${{matrix.python-version}}
@@ -2124,8 +2118,7 @@ jobs:
path: ".build/.k8s-env"
key: "\
k8s-env-${{steps.breeze.outputs.host-python-version}}-\
- ${{ hashFiles('scripts/ci/kubernetes/k8s_requirements.txt','setup.cfg',\
- 'setup.py','pyproject.toml','generated/provider_dependencies.json') }}"
+ ${{ hashFiles('scripts/ci/kubernetes/k8s_requirements.txt','pyproject.toml') }}"
- name: Run complete K8S tests ${{needs.build-info.outputs.kubernetes-combos-list-as-string}}
run: breeze k8s run-complete-tests --run-in-parallel --upgrade
env:
@@ -2256,7 +2249,7 @@ jobs:
- name: "Prepare providers packages for PROD build"
run: >
breeze release-management prepare-provider-packages
- --package-list-file ./airflow/providers/installed_providers.txt
+ --package-list-file ./dev/prod_image_installed_providers.txt
--package-format wheel
env:
VERSION_SUFFIX_FOR_PYPI: "dev0"
diff --git a/.gitignore b/.gitignore
index 32e202a43e..cf1a075778 100644
--- a/.gitignore
+++ b/.gitignore
@@ -190,8 +190,6 @@ dmypy.json
log.txt*
# Provider-related ignores
-/provider_packages/CHANGELOG.txt
-/provider_packages/MANIFEST.in
/airflow/providers/__init__.py
# Docker context files
@@ -219,7 +217,7 @@ pip-wheel-metadata
/dev/Dockerfile.pmc
# Generated UI licenses
-licenses/LICENSES-ui.txt
+3rd-party-licenses/LICENSES-ui.txt
# Packaged breeze on Windows
/breeze.exe
@@ -240,3 +238,6 @@ licenses/LICENSES-ui.txt
# Dask Executor tests generate this directory
/tests/executors/dask-worker-space/
+
+# airflow-build-dockerfile and correconding ignore file
+airflow-build-dockerfile*
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 51aba2e09f..f50076ba61 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -274,11 +274,6 @@ repos:
- --ignore-words=docs/spelling_wordlist.txt
- --skip=airflow/providers/*/*.rst,airflow/www/*.log,docs/*/commits.rst,docs/apache-airflow/tutorial/pipeline_example.csv,*.min.js,*.lock,INTHEWILD.md
- --exclude-file=.codespellignorelines
- - repo: https://github.com/abravalheri/validate-pyproject
- rev: v0.15
- hooks:
- - id: validate-pyproject
- name: Validate pyproject.toml
- repo: local
# Note that this is the 2nd "local" repo group in the .pre-commit-config.yaml file. This is because
# we try to minimise the number of passes that must happen in order to apply some of the changes
@@ -333,13 +328,6 @@ repos:
files: Dockerfile.*$
pass_filenames: true
require_serial: true
- - id: check-setup-order
- name: Check order of dependencies in setup.cfg and setup.py
- language: python
- files: ^setup\.cfg$|^setup\.py$
- pass_filenames: false
- entry: ./scripts/ci/pre_commit/pre_commit_check_order_setup.py
- additional_dependencies: ['rich>=12.4.4']
- id: check-airflow-k8s-not-used
name: Check airflow.kubernetes imports are not used
language: python
@@ -363,14 +351,6 @@ repos:
exclude: ^airflow/kubernetes/|^airflow/providers/
entry: ./scripts/ci/pre_commit/pre_commit_check_cncf_k8s_used_for_k8s_executor_only.py
additional_dependencies: ['rich>=12.4.4']
- - id: check-extra-packages-references
- name: Checks setup extra packages
- description: Checks if all the libraries in setup.py are listed in extra-packages-ref.rst file
- language: python
- files: ^setup\.py$|^docs/apache-airflow/extra-packages-ref\.rst$|^airflow/providers/.*/provider\.yaml$
- pass_filenames: false
- entry: ./scripts/ci/pre_commit/pre_commit_check_setup_extra_packages_ref.py
- additional_dependencies: ['rich>=12.4.4']
- id: check-airflow-provider-compatibility
name: Check compatibility of Providers with Airflow
entry: ./scripts/ci/pre_commit/pre_commit_check_provider_airflow_compatibility.py
@@ -400,19 +380,34 @@ repos:
files: ^airflow/providers/.*/hooks/.*\.py$
additional_dependencies: ['rich>=12.4.4', 'pyyaml', 'packaging']
- id: update-providers-dependencies
- name: Update cross-dependencies for providers packages
+ name: Update dependencies for provider packages
entry: ./scripts/ci/pre_commit/pre_commit_update_providers_dependencies.py
language: python
- files: ^airflow/providers/.*\.py$|^airflow/providers/.*/provider\.yaml$|^tests/providers/.*\.py$|^tests/system/providers/.*\.py$
+ files: ^airflow/providers/.*\.py$|^airflow/providers/.*/provider\.yaml$|^tests/providers/.*\.py$|^tests/system/providers/.*\.py$|^scripts/ci/pre_commit/pre_commit_update_providers_dependencies\.py$
pass_filenames: false
- additional_dependencies: ['setuptools', 'rich>=12.4.4', 'pyyaml']
+ additional_dependencies: ['setuptools', 'rich>=12.4.4', 'pyyaml', 'tomli']
+ - id: check-extra-packages-references
+ name: Checks setup extra packages
+ description: Checks if all the extras defined in pyproject.toml are listed in extra-packages-ref.rst file
+ language: python
+ files: ^docs/apache-airflow/extra-packages-ref\.rst$|^pyproject.toml
+ pass_filenames: false
+ entry: ./scripts/ci/pre_commit/pre_commit_check_extra_packages_ref.py
+ additional_dependencies: ['rich>=12.4.4', 'tomli', 'tabulate']
+ - id: check-pyproject-toml-order
+ name: Check order of dependencies in pyproject.toml
+ language: python
+ files: ^pyproject\.toml$
+ pass_filenames: false
+ entry: ./scripts/ci/pre_commit/pre_commit_check_order_pyproject_toml.py
+ additional_dependencies: ['rich>=12.4.4']
- id: update-extras
name: Update extras in documentation
entry: ./scripts/ci/pre_commit/pre_commit_insert_extras.py
language: python
files: ^setup\.py$|^CONTRIBUTING\.rst$|^INSTALL$|^airflow/providers/.*/provider\.yaml$
pass_filenames: false
- additional_dependencies: ['rich>=12.4.4']
+ additional_dependencies: ['rich>=12.4.4', 'tomli']
- id: check-extras-order
name: Check order of extras in Dockerfile
entry: ./scripts/ci/pre_commit/pre_commit_check_order_dockerfile_extras.py
@@ -712,7 +707,7 @@ repos:
name: Sort alphabetically and uniquify installed_providers.txt
entry: ./scripts/ci/pre_commit/pre_commit_sort_installed_providers.py
language: python
- files: ^\.pre-commit-config\.yaml$|^airflow/providers/installed_providers\.txt$
+ files: ^\.pre-commit-config\.yaml$|^dev/.*_installed_providers\.txt$
pass_filenames: false
require_serial: true
- id: update-spelling-wordlist-to-be-sorted
diff --git a/.rat-excludes b/.rat-excludes
index d881787de9..ab2296b487 100644
--- a/.rat-excludes
+++ b/.rat-excludes
@@ -43,7 +43,8 @@ venv
files
airflow.iml
.gitmodules
-installed_providers.txt
+prod_image_installed_providers.txt
+airflow_pre_installed_providers.txt
# Generated doc files
.*html
@@ -61,7 +62,7 @@ spelling_wordlist.txt
# it is compatible according to http://www.apache.org/legal/resolved.html#category-a
kerberos_auth.py
airflow_api_auth_backend_kerberos_auth_py.html
-licenses/*
+3rd-party-licenses/*
parallel.js
underscore.js
jquery.dataTables.min.js
diff --git a/licenses/LICENSE-bootstrap.txt b/3rd-party-licenses/LICENSE-bootstrap.txt
similarity index 100%
rename from licenses/LICENSE-bootstrap.txt
rename to 3rd-party-licenses/LICENSE-bootstrap.txt
diff --git a/licenses/LICENSE-bootstrap3-typeahead.txt b/3rd-party-licenses/LICENSE-bootstrap3-typeahead.txt
similarity index 100%
rename from licenses/LICENSE-bootstrap3-typeahead.txt
rename to 3rd-party-licenses/LICENSE-bootstrap3-typeahead.txt
diff --git a/licenses/LICENSE-d3-shape.txt b/3rd-party-licenses/LICENSE-d3-shape.txt
similarity index 100%
rename from licenses/LICENSE-d3-shape.txt
rename to 3rd-party-licenses/LICENSE-d3-shape.txt
diff --git a/licenses/LICENSE-d3-tip.txt b/3rd-party-licenses/LICENSE-d3-tip.txt
similarity index 100%
rename from licenses/LICENSE-d3-tip.txt
rename to 3rd-party-licenses/LICENSE-d3-tip.txt
diff --git a/licenses/LICENSE-d3js.txt b/3rd-party-licenses/LICENSE-d3js.txt
similarity index 100%
rename from licenses/LICENSE-d3js.txt
rename to 3rd-party-licenses/LICENSE-d3js.txt
diff --git a/licenses/LICENSE-dagre-d3.txt b/3rd-party-licenses/LICENSE-dagre-d3.txt
similarity index 100%
rename from licenses/LICENSE-dagre-d3.txt
rename to 3rd-party-licenses/LICENSE-dagre-d3.txt
diff --git a/licenses/LICENSE-datatables.txt b/3rd-party-licenses/LICENSE-datatables.txt
similarity index 100%
rename from licenses/LICENSE-datatables.txt
rename to 3rd-party-licenses/LICENSE-datatables.txt
diff --git a/licenses/LICENSE-elasticmock.txt b/3rd-party-licenses/LICENSE-elasticmock.txt
similarity index 100%
rename from licenses/LICENSE-elasticmock.txt
rename to 3rd-party-licenses/LICENSE-elasticmock.txt
diff --git a/licenses/LICENSE-eonasdan-bootstrap-datetimepicker.txt b/3rd-party-licenses/LICENSE-eonasdan-bootstrap-datetimepicker.txt
similarity index 100%
rename from licenses/LICENSE-eonasdan-bootstrap-datetimepicker.txt
rename to 3rd-party-licenses/LICENSE-eonasdan-bootstrap-datetimepicker.txt
diff --git a/licenses/LICENSE-flask-kerberos.txt b/3rd-party-licenses/LICENSE-flask-kerberos.txt
similarity index 100%
rename from licenses/LICENSE-flask-kerberos.txt
rename to 3rd-party-licenses/LICENSE-flask-kerberos.txt
diff --git a/licenses/LICENSE-hue.txt b/3rd-party-licenses/LICENSE-hue.txt
similarity index 100%
rename from licenses/LICENSE-hue.txt
rename to 3rd-party-licenses/LICENSE-hue.txt
diff --git a/licenses/LICENSE-jqclock.txt b/3rd-party-licenses/LICENSE-jqclock.txt
similarity index 100%
rename from licenses/LICENSE-jqclock.txt
rename to 3rd-party-licenses/LICENSE-jqclock.txt
diff --git a/licenses/LICENSE-jquery.txt b/3rd-party-licenses/LICENSE-jquery.txt
similarity index 100%
rename from licenses/LICENSE-jquery.txt
rename to 3rd-party-licenses/LICENSE-jquery.txt
diff --git a/licenses/LICENSE-moment.txt b/3rd-party-licenses/LICENSE-moment.txt
similarity index 100%
rename from licenses/LICENSE-moment.txt
rename to 3rd-party-licenses/LICENSE-moment.txt
diff --git a/licenses/LICENSE-normalize.txt b/3rd-party-licenses/LICENSE-normalize.txt
similarity index 100%
rename from licenses/LICENSE-normalize.txt
rename to 3rd-party-licenses/LICENSE-normalize.txt
diff --git a/licenses/LICENSE-pytest-capture-warnings.txt b/3rd-party-licenses/LICENSE-pytest-capture-warnings.txt
similarity index 100%
rename from licenses/LICENSE-pytest-capture-warnings.txt
rename to 3rd-party-licenses/LICENSE-pytest-capture-warnings.txt
diff --git a/licenses/LICENSE-unicodecsv.txt b/3rd-party-licenses/LICENSE-unicodecsv.txt
similarity index 100%
rename from licenses/LICENSE-unicodecsv.txt
rename to 3rd-party-licenses/LICENSE-unicodecsv.txt
diff --git a/BREEZE.rst b/BREEZE.rst
index 85164fb456..61b42c510f 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -1569,7 +1569,7 @@ The CI image is built automatically as needed, however it can be rebuilt manuall
Building the image first time pulls a pre-built version of images from the Docker Hub, which may take some
time. But for subsequent source code changes, no wait time is expected.
-However, changes to sensitive files like ``setup.py`` or ``Dockerfile.ci`` will trigger a rebuild
+However, changes to sensitive files like ``pyproject.toml`` or ``Dockerfile.ci`` will trigger a rebuild
that may take more time though it is highly optimized to only rebuild what is needed.
Breeze has built in mechanism to check if your local image has not diverged too much from the
@@ -2299,7 +2299,7 @@ These are all available flags of ``release-management add-back-references`` comm
Generating constraints
""""""""""""""""""""""
-Whenever setup.py gets modified, the CI main job will re-generate constraint files. Those constraint
+Whenever ``pyproject.toml`` gets modified, the CI main job will re-generate constraint files. Those constraint
files are stored in separated orphan branches: ``constraints-main``, ``constraints-2-0``.
Those are constraint files as described in detail in the
@@ -2341,14 +2341,14 @@ These are all available flags of ``generate-constraints`` command:
:width: 100%
:alt: Breeze generate-constraints
-In case someone modifies setup.py, the scheduled CI Tests automatically upgrades and
+In case someone modifies ``pyproject.toml``, the scheduled CI Tests automatically upgrades and
pushes changes to the constraint files, however you can also perform test run of this locally using
the procedure described in the
`Manually generating image cache and constraints <dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md>`_
which utilises multiple processors on your local machine to generate such constraints faster.
-This bumps the constraint files to latest versions and stores hash of setup.py. The generated constraint
-and setup.py hash files are stored in the ``files`` folder and while generating the constraints diff
+This bumps the constraint files to latest versions and stores hash of ``pyproject.toml``. The generated constraint
+and ``pyproject.toml`` hash files are stored in the ``files`` folder and while generating the constraints diff
of changes vs the previous constraint files is printed.
Updating constraints
@@ -2697,18 +2697,18 @@ disappear when you exit Breeze shell.
When you want to add dependencies permanently, then it depends what kind of dependency you add.
-If you want to add core dependency that should always be installed - you need to add it to ``setup.cfg``
-to ``install_requires`` section. If you want to add it to one of the optional core extras, you should
-add it in the extra definition in ``setup.py`` (you need to find out where it is defined). If you want
-to add it to one of the providers, you need to add it to the ``provider.yaml`` file in the provider
+If you want to add core dependency that should always be installed - you need to add it to ``pyproject.toml``
+to ``dependencies`` section. If you want to add it to one of the optional core extras, you should
+add it in the extra definition in ``pyproject.toml`` (you need to find out where it is defined).
+If you want to add it to one of the providers, you need to add it to the ``provider.yaml`` file in the provider
directory - but remember that this should be followed by running pre-commit that will automatically update
-the ``generated/provider_dependencies.json`` directory with the new dependencies:
+the ``pyproject.toml`` with the new dependencies as the ``provider.yaml`` files are not used directly, they
+are used to update ``pyproject.toml`` file:
.. code-block:: bash
pre-commit run update-providers-dependencies --all-files
-
You can also run the pre-commit by ``breeze static-checks --type update-providers-dependencies --all-files``
command - which provides autocomplete.
diff --git a/CI.rst b/CI.rst
index e8d7200027..af37ff2fd9 100644
--- a/CI.rst
+++ b/CI.rst
@@ -604,7 +604,7 @@ those via corresponding command line flags passed to ``breeze shell`` command.
| ``UPGRADE_TO_NEWER_DEPENDENCIES`` | false | false | false\* | Determines whether the build should |
| | | | | attempt to upgrade Python base image and all |
| | | | | PIP dependencies to latest ones matching |
-| | | | | ``setup.py`` limits. This tries to replicate |
+| | | | | ``pyproject.toml`` limits. Tries to replicate |
| | | | | the situation of "fresh" user who just installs |
| | | | | airflow and uses latest version of matching |
| | | | | dependencies. By default we are using a |
@@ -625,7 +625,7 @@ those via corresponding command line flags passed to ``breeze shell`` command.
| | | | | |
| | | | | Setting the value to random value is best way |
| | | | | to assure that constraints are upgraded even if |
-| | | | | there is no change to setup.py |
+| | | | | there is no change to ``pyproject.toml`` |
| | | | | |
| | | | | This way our constraints are automatically |
| | | | | tested and updated whenever new versions |
diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
index ba3ff9ec98..a7bf23effe 100644
--- a/CONTRIBUTING.rst
+++ b/CONTRIBUTING.rst
@@ -660,29 +660,66 @@ Extras
------
There are a number of extras that can be specified when installing Airflow. Those
-extras can be specified after the usual pip install - for example
-``pip install -e .[ssh]``. For development purpose there is a ``devel`` extra that
-installs all development dependencies. There is also ``devel_ci`` that installs
-all dependencies needed in the CI environment.
+extras can be specified after the usual pip install - for example ``pip install -e.[ssh]`` for editable
+installation. Note that there are two kinds of those extras - ``regular`` extras (used when you install
+airflow as a user, but in ``editable`` mode you can also install ``devel`` extras that are necessary if
+you want to run airflow locally for testing and ``doc`` extras that install tools needed to build
+the documentation.
This is the full list of those extras:
- .. START EXTRAS HERE
-aiobotocore, airbyte, alibaba, all, all_dbs, amazon, apache.atlas, apache.beam, apache.cassandra,
-apache.drill, apache.druid, apache.flink, apache.hdfs, apache.hive, apache.impala, apache.kafka,
-apache.kylin, apache.livy, apache.pig, apache.pinot, apache.spark, apache.webhdfs, apprise,
-arangodb, asana, async, atlas, atlassian.jira, aws, azure, cassandra, celery, cgroups, cloudant,
-cncf.kubernetes, cohere, common.io, common.sql, crypto, databricks, datadog, dbt.cloud,
-deprecated_api, devel, devel_all, devel_ci, devel_hadoop, dingding, discord, doc, doc_gen, docker,
-druid, elasticsearch, exasol, facebook, ftp, gcp, gcp_api, github, github_enterprise, google,
-google_auth, graphviz, grpc, hashicorp, hdfs, hive, http, imap, influxdb, jdbc, jenkins, kerberos,
-kubernetes, ldap, leveldb, microsoft.azure, microsoft.mssql, microsoft.psrp, microsoft.winrm, mongo,
-mssql, mysql, neo4j, odbc, openai, openfaas, openlineage, opensearch, opsgenie, oracle, otel,
-pagerduty, pandas, papermill, password, pgvector, pinecone, pinot, postgres, presto, rabbitmq,
-redis, s3, s3fs, salesforce, samba, saml, segment, sendgrid, sentry, sftp, singularity, slack, smtp,
-snowflake, spark, sqlite, ssh, statsd, tableau, tabular, telegram, trino, vertica, virtualenv,
-weaviate, webhdfs, winrm, yandex, zendesk
- .. END EXTRAS HERE
+Devel extras
+.............
+
+The ``devel`` extras are not available in the released packages. They are only available when you install
+Airflow from sources in ``editable`` installation - i.e. one that you are usually using to contribute to
+Airflow. They provide tools such as ``pytest`` and ``mypy`` for general purpose development and testing, also
+some providers have their own development-related extras tbat allow to install tools necessary to run tests,
+where the tools are specific for the provider.
+
+
+ .. START DEVEL EXTRAS HERE
+devel, devel-all, devel-all-dbs, devel-ci, devel-debuggers, devel-devscripts, devel-duckdb, devel-
+hadoop, devel-mypy, devel-sentry, devel-static-checks, devel-tests
+ .. END DEVEL EXTRAS HERE
+
+Doc extras
+...........
+
+The ``doc`` extras are not available in the released packages. They are only available when you install
+Airflow from sources in ``editable`` installation - i.e. one that you are usually using to contribute to
+Airflow. They provide tools needed when you want to build Airflow documentation (note that you also need
+``devel`` extras installed for airflow and providers in order to build documentation for airflow and
+provider packages respectively). The ``doc`` package is enough to build regular documentation, where
+``doc_gen`` is needed to generate ER diagram we have describing our database.
+
+ .. START DOC EXTRAS HERE
+doc, doc-gen
+ .. END DOC EXTRAS HERE
+
+
+Regular extras
+..............
+
+Those extras are available as regular Airflow extras and are targeted to be used by Airflow users and
+contributors to select features of Airflow they want to use They might install additional providers or
+just install dependencies that are necessary to enable the feature.
+
+ .. START REGULAR EXTRAS HERE
+aiobotocore, airbyte, alibaba, all, all-core, all-dbs, amazon, apache-atlas, apache-beam, apache-
+cassandra, apache-drill, apache-druid, apache-flink, apache-hdfs, apache-hive, apache-impala,
+apache-kafka, apache-kylin, apache-livy, apache-pig, apache-pinot, apache-spark, apache-webhdfs,
+apprise, arangodb, asana, async, atlas, atlassian-jira, aws, azure, cassandra, celery, cgroups,
+cloudant, cncf-kubernetes, cohere, common-io, common-sql, crypto, databricks, datadog, dbt-cloud,
+deprecated-api, dingding, discord, docker, druid, elasticsearch, exasol, facebook, ftp, gcp,
+gcp_api, github, github-enterprise, google, google-auth, graphviz, grpc, hashicorp, hdfs, hive,
+http, imap, influxdb, jdbc, jenkins, kerberos, kubernetes, ldap, leveldb, microsoft-azure,
+microsoft-mssql, microsoft-psrp, microsoft-winrm, mongo, mssql, mysql, neo4j, odbc, openai,
+openfaas, openlineage, opensearch, opsgenie, oracle, otel, pagerduty, pandas, papermill, password,
+pgvector, pinecone, pinot, postgres, presto, rabbitmq, redis, s3, s3fs, salesforce, samba, saml,
+segment, sendgrid, sentry, sftp, singularity, slack, smtp, snowflake, spark, sqlite, ssh, statsd,
+tableau, tabular, telegram, trino, vertica, virtualenv, weaviate, webhdfs, winrm, yandex, zendesk
+ .. END REGULAR EXTRAS HERE
Provider packages
-----------------
@@ -704,29 +741,25 @@ of ``airflow\providers``. This file contains:
* list of integrations, operators, hooks, sensors, transfers provided by the provider (useful for documentation generation)
* list of connection types, extra-links, secret backends, auth backends, and logging handlers (useful to both
register them as they are needed by Airflow and to include them in documentation automatically).
+* and more ...
If you want to add dependencies to the provider, you should add them to the corresponding ``provider.yaml``
and Airflow pre-commits and package generation commands will use them when preparing package information.
In Airflow 1.10 all those providers were installed together within one single package and when you installed
airflow locally, from sources, they were also installed. In Airflow 2.0, providers are separated out,
-and not packaged together with the core, unless you set ``INSTALL_PROVIDERS_FROM_SOURCES`` environment
-variable to ``true``.
+and not packaged together with the core when you build "apache-airflow" package, however when you install
+airflow project locally with ``pip install -e ".[devel]"`` they are available on the same
+environment as Airflow.
-In Breeze - which is a development environment, ``INSTALL_PROVIDERS_FROM_SOURCES`` variable is set to true,
-but you can add ``--install-providers-from-sources=false`` flag to Breeze to install providers from PyPI instead of source files when
-building the images.
-
-One watch-out - providers are still always installed (or rather available) if you install airflow from
-sources using ``-e`` (or ``--editable``) flag. In such case airflow is read directly from the sources
-without copying airflow packages to the usual installation location, and since 'providers' folder is
-in this airflow folder - the providers package is importable.
+You should only update dependencies for the provider in the corresponding ``provider.yaml`` which is the
+source of truth for all information about the provider.
Some of the packages have cross-dependencies with other providers packages. This typically happens for
transfer operators where operators use hooks from the other providers in case they are transferring
-data between the providers. The list of dependencies is maintained (automatically with pre-commits)
-in the ``generated/provider_dependencies.json``. Pre-commits are also used to generate dependencies.
-The dependency list is automatically used during PyPI packages generation.
+data between the providers. The list of dependencies is maintained (automatically with the
+``update-providers-dependencies`` pre-commit) in the ``generated/provider_dependencies.json``.
+Same pre-commit also updates generate dependencies in ``pyproject.toml``.
Cross-dependencies between provider packages are converted into extras - if you need functionality from
the other provider package you can install it adding [extra] after the
@@ -735,8 +768,9 @@ the other provider package you can install it adding [extra] after the
transfer operators from Amazon ECS.
If you add a new dependency between different providers packages, it will be detected automatically during
-and pre-commit will generate new entry in ``generated/provider_dependencies.json`` so that
-the package extra dependencies are properly handled when package is installed.
+and pre-commit will generate new entry in ``generated/provider_dependencies.json`` and update
+``pyproject.toml`` so that the package extra dependencies are properly handled when package
+might be installed when breeze is restarted or by your IDE or by running ``pip install -e ".[devel]"``.
Developing community managed provider packages
----------------------------------------------
@@ -746,27 +780,26 @@ They are part of the same repository as Apache Airflow (we use ``monorepo`` appr
parts of the system are developed in the same repository but then they are packaged and released separately).
All the community-managed providers are in 'airflow/providers' folder and they are all sub-packages of
'airflow.providers' package. All the providers are available as ``apache-airflow-providers-<PROVIDER_ID>``
-packages.
+packages when installed by users, but when you contribute to providers you can work on airflow main
+and install provider dependencies via ``editable`` extras - without having to manage and install providers
+separately, you can easily run tests for the providers and when you run airflow from the ``main``
+sources, all community providers are automatically available for you.
The capabilities of the community-managed providers are the same as the third-party ones. When
the providers are installed from PyPI, they provide the entry-point containing the metadata as described
in the previous chapter. However when they are locally developed, together with Airflow, the mechanism
of discovery of the providers is based on ``provider.yaml`` file that is placed in the top-folder of
-the provider. Similarly as in case of the ``provider.yaml`` file is compliant with the
-`json-schema specification <https://github.com/apache/airflow/blob/main/airflow/provider.yaml.schema.json>`_.
-Thanks to that mechanism, you can develop community managed providers in a seamless way directly from
-Airflow sources, without preparing and releasing them as packages. This is achieved by:
+the provider. The ``provider.yaml`` is the single source of truth for the provider metadata and it is
+there where you should add and remove dependencies for providers (following by running
+``update-providers-dependencies`` pre-commit to synchronize the dependencies with ``pyproject.toml``
+of Airflow).
-* When Airflow is installed locally in editable mode (``pip install -e``) the provider packages installed
- from PyPI are uninstalled and the provider discovery mechanism finds the providers in the Airflow
- sources by searching for provider.yaml files.
+The ``provider.yaml`` file is compliant with the schema that is available in
+`json-schema specification <https://github.com/apache/airflow/blob/main/airflow/provider.yaml.schema.json>`_.
-* When you want to install Airflow from sources you can set ``INSTALL_PROVIDERS_FROM_SOURCES`` variable
- to ``true`` and then the providers will not be installed from PyPI packages, but they will be installed
- from local sources as part of the ``apache-airflow`` package, but additionally the ``provider.yaml`` files
- are copied together with the sources, so that capabilities and names of the providers can be discovered.
- This mode is especially useful when you are developing a new provider, that cannot be installed from
- PyPI and you want to check if it installs cleanly.
+Thanks to that mechanism, you can develop community managed providers in a seamless way directly from
+Airflow sources, without preparing and releasing them as packages separately, which would be rather
+complicated.
Regardless if you plan to contribute your provider, when you are developing your own, custom providers,
you can use the above functionality to make your development easier. You can add your provider
@@ -774,6 +807,7 @@ as a sub-folder of the ``airflow.providers`` package, add the ``provider.yaml``
in development mode - then capabilities of your provider will be discovered by airflow and you will see
the provider among other providers in ``airflow providers`` command output.
+
Documentation for the community managed providers
-------------------------------------------------
@@ -799,12 +833,13 @@ You can see for example ``google`` provider which has very comprehensive documen
* `Documentation <docs/apache-airflow-providers-google>`_
* `Example DAGs <airflow/providers/google/cloud/example_dags>`_
-Part of the documentation are example dags. We are using the example dags for various purposes in
-providers:
+Part of the documentation are example dags (placed in the ``tests/system`` folder). The reason why
+they are in ``tests/system`` is because we are using the example dags for various purposes:
* showing real examples of how your provider classes (Operators/Sensors/Transfers) can be used
* snippets of the examples are embedded in the documentation via ``exampleinclude::`` directive
-* examples are executable as system tests
+* examples are executable as system tests and some of our stakeholders run them regularly to
+ check if ``system`` level instagration is still working, before releasing a new version of the provider.
Testing the community managed providers
---------------------------------------
@@ -841,8 +876,7 @@ be open to allow several different libraries with the same requirements to be in
The problem is that Apache Airflow is a bit of both - application to install and library to be used when
you are developing your own operators and DAGs.
-This - seemingly unsolvable - puzzle is solved by having pinned constraints files. Those are available
-as of airflow 1.10.10 and further improved with 1.10.12 (moved to separate orphan branches)
+This - seemingly unsolvable - puzzle is solved by having pinned constraints files.
Pinned constraint files
=======================
@@ -906,7 +940,7 @@ requirements).
.. code-block:: bash
- pip install -e . \
+ pip install -e ".[devel]" \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.8.txt"
@@ -931,7 +965,7 @@ If you want to update just airflow dependencies, without paying attention to pro
The ``constraints-<PYTHON_MAJOR_MINOR_VERSION>.txt`` and ``constraints-no-providers-<PYTHON_MAJOR_MINOR_VERSION>.txt``
-will be automatically regenerated by CI job every time after the ``setup.py`` is updated and pushed
+will be automatically regenerated by CI job every time after the ``pyproject.toml`` is updated and pushed
if the tests are successful.
diff --git a/CONTRIBUTORS_QUICK_START.rst b/CONTRIBUTORS_QUICK_START.rst
index e79c9cbe02..ffe30e753d 100644
--- a/CONTRIBUTORS_QUICK_START.rst
+++ b/CONTRIBUTORS_QUICK_START.rst
@@ -151,14 +151,13 @@ Pyenv and setting up virtual-env
basic system-level dependencies on Debian/Ubuntu-like system. You will have to adapt it to install similar packages
if your operating system is MacOS or another flavour of Linux
-
.. code-block:: bash
sudo apt install openssl sqlite default-libmysqlclient-dev libmysqlclient-dev postgresql
If you want to install all airflow providers, more system dependencies might be needed. For example on Debian/Ubuntu
-like system, this command will install all necessary dependencies that should be installed when you use ``devel_all``
-extra while installing airflow.
+like system, this command will install all necessary dependencies that should be installed when you use
+``devel-all`` extra while installing airflow.
.. code-block:: bash
diff --git a/Dockerfile b/Dockerfile
index e493d99ec0..47c904e566 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -35,7 +35,7 @@
# much smaller.
#
# Use the same builder frontend version for everyone
-ARG AIRFLOW_EXTRAS="aiobotocore,amazon,async,celery,cncf.kubernetes,common.io,docker,elasticsearch,ftp,google,google_auth,graphviz,grpc,hashicorp,http,ldap,microsoft.azure,mysql,odbc,openlineage,pandas,postgres,redis,sendgrid,sftp,slack,snowflake,ssh,statsd,virtualenv"
+ARG AIRFLOW_EXTRAS="aiobotocore,amazon,async,celery,cncf-kubernetes,common-io,docker,elasticsearch,ftp,google,google-auth,graphviz,grpc,hashicorp,http,ldap,microsoft-azure,mysql,odbc,openlineage,pandas,postgres,redis,sendgrid,sftp,slack,snowflake,ssh,statsd,virtualenv"
ARG ADDITIONAL_AIRFLOW_EXTRAS=""
ARG ADDITIONAL_PYTHON_DEPS=""
@@ -760,10 +760,11 @@ function install_airflow() {
"${AIRFLOW_INSTALLATION_METHOD}[${AIRFLOW_EXTRAS}]${AIRFLOW_VERSION_SPECIFICATION}" \
${EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS=}
if [[ -n "${AIRFLOW_INSTALL_EDITABLE_FLAG}" ]]; then
- # Remove airflow and reinstall it using editable flag
+ # Remove airflow and all providers and reinstall it using editable flag
# We can only do it when we install airflow from sources
set -x
- pip uninstall apache-airflow --yes
+ pip freeze | grep apache-airflow-providers | xargs pip uninstall --yes 2>/dev/null || true
+ pip uninstall apache-airflow --yes 2>/dev/null || true
pip install --root-user-action ignore ${AIRFLOW_INSTALL_EDITABLE_FLAG} \
${ADDITIONAL_PIP_INSTALL_FLAGS} \
"${AIRFLOW_INSTALLATION_METHOD}[${AIRFLOW_EXTRAS}]${AIRFLOW_VERSION_SPECIFICATION}"
@@ -785,7 +786,7 @@ function install_airflow() {
"${AIRFLOW_INSTALLATION_METHOD}[${AIRFLOW_EXTRAS}]${AIRFLOW_VERSION_SPECIFICATION}" \
--constraint "${AIRFLOW_CONSTRAINTS_LOCATION}"
common::install_pip_version
- # then upgrade if needed without using constraints to account for new limits in setup.py
+ # then upgrade if needed without using constraints to account for new limits in pyproject.toml
pip install --root-user-action ignore --upgrade --upgrade-strategy only-if-needed \
${ADDITIONAL_PIP_INSTALL_FLAGS} \
${AIRFLOW_INSTALL_EDITABLE_FLAG} \
@@ -1288,17 +1289,13 @@ ARG DEFAULT_CONSTRAINTS_BRANCH="constraints-main"
# By default PIP has progress bar but you can disable it.
ARG PIP_PROGRESS_BAR
# By default we do not use pre-cached packages, but in CI/Breeze environment we override this to speed up
-# builds in case setup.py/setup.cfg changed. This is pure optimisation of CI/Breeze builds.
+# builds in case pyproject.toml changed. This is pure optimisation of CI/Breeze builds.
ARG AIRFLOW_PRE_CACHED_PIP_PACKAGES="false"
# This is airflow version that is put in the label of the image build
ARG AIRFLOW_VERSION
# By default latest released version of airflow is installed (when empty) but this value can be overridden
# and we can install version according to specification (For example ==2.0.2 or <3.0.0).
ARG AIRFLOW_VERSION_SPECIFICATION
-# By default we install providers from PyPI but in case of Breeze build we want to install providers
-# from local sources without the need of preparing provider packages upfront. This value is
-# automatically overridden by Breeze scripts.
-ARG INSTALL_PROVIDERS_FROM_SOURCES="false"
# Determines the way airflow is installed. By default we install airflow from PyPI `apache-airflow` package
# But it also can be `.` from local installation or GitHub URL pointing to specific branch or tag
# Of Airflow. Note That for local source installation you need to have local sources of
@@ -1327,7 +1324,6 @@ ARG ADDITIONAL_PIP_INSTALL_FLAGS=""
ENV AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION} \
AIRFLOW_PRE_CACHED_PIP_PACKAGES=${AIRFLOW_PRE_CACHED_PIP_PACKAGES} \
- INSTALL_PROVIDERS_FROM_SOURCES=${INSTALL_PROVIDERS_FROM_SOURCES} \
AIRFLOW_VERSION=${AIRFLOW_VERSION} \
AIRFLOW_INSTALLATION_METHOD=${AIRFLOW_INSTALLATION_METHOD} \
AIRFLOW_VERSION_SPECIFICATION=${AIRFLOW_VERSION_SPECIFICATION} \
@@ -1372,8 +1368,7 @@ ARG USE_CONSTRAINTS_FOR_CONTEXT_PACKAGES="false"
# In case of Production build image segment we want to pre-install main version of airflow
# dependencies from GitHub so that we do not have to always reinstall it from the scratch.
-# The Airflow (and providers in case INSTALL_PROVIDERS_FROM_SOURCES is "false")
-# are uninstalled, only dependencies remain
+# The Airflow and providers are uninstalled, only dependencies remain
# the cache is only used when "upgrade to newer dependencies" is not set to automatically
# account for removed dependencies (we do not install them in the first place) and in case
# INSTALL_PACKAGES_FROM_CONTEXT is not set (because then caching it from main makes no sense).
diff --git a/Dockerfile.ci b/Dockerfile.ci
index 3364ee234f..0f10757de5 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -598,10 +598,11 @@ function install_airflow() {
"${AIRFLOW_INSTALLATION_METHOD}[${AIRFLOW_EXTRAS}]${AIRFLOW_VERSION_SPECIFICATION}" \
${EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS=}
if [[ -n "${AIRFLOW_INSTALL_EDITABLE_FLAG}" ]]; then
- # Remove airflow and reinstall it using editable flag
+ # Remove airflow and all providers and reinstall it using editable flag
# We can only do it when we install airflow from sources
set -x
- pip uninstall apache-airflow --yes
+ pip freeze | grep apache-airflow-providers | xargs pip uninstall --yes 2>/dev/null || true
+ pip uninstall apache-airflow --yes 2>/dev/null || true
pip install --root-user-action ignore ${AIRFLOW_INSTALL_EDITABLE_FLAG} \
${ADDITIONAL_PIP_INSTALL_FLAGS} \
"${AIRFLOW_INSTALLATION_METHOD}[${AIRFLOW_EXTRAS}]${AIRFLOW_VERSION_SPECIFICATION}"
@@ -623,7 +624,7 @@ function install_airflow() {
"${AIRFLOW_INSTALLATION_METHOD}[${AIRFLOW_EXTRAS}]${AIRFLOW_VERSION_SPECIFICATION}" \
--constraint "${AIRFLOW_CONSTRAINTS_LOCATION}"
common::install_pip_version
- # then upgrade if needed without using constraints to account for new limits in setup.py
+ # then upgrade if needed without using constraints to account for new limits in pyproject.toml
pip install --root-user-action ignore --upgrade --upgrade-strategy only-if-needed \
${ADDITIONAL_PIP_INSTALL_FLAGS} \
${AIRFLOW_INSTALL_EDITABLE_FLAG} \
@@ -899,7 +900,7 @@ function check_download_sqlalchemy() {
if [[ ${DOWNGRADE_SQLALCHEMY=} != "true" ]]; then
return
fi
- min_sqlalchemy_version=$(grep "sqlalchemy>=" setup.cfg | sed "s/.*>=\([0-9\.]*\).*/\1/")
+ min_sqlalchemy_version=$(grep "\"sqlalchemy>=" pyproject.toml | sed "s/.*>=\([0-9\.]*\).*/\1/" | xargs)
echo
echo "${COLOR_BLUE}Downgrading sqlalchemy to minimum supported version: ${min_sqlalchemy_version}${COLOR_RESET}"
echo
@@ -1057,8 +1058,6 @@ ARG DEFAULT_CONSTRAINTS_BRANCH="constraints-main"
# It can also be overwritten manually by setting the AIRFLOW_CI_BUILD_EPOCH environment variable.
ARG AIRFLOW_CI_BUILD_EPOCH="6"
ARG AIRFLOW_PRE_CACHED_PIP_PACKAGES="true"
-# By default in the image, we are installing all providers when installing from sources
-ARG INSTALL_PROVIDERS_FROM_SOURCES="true"
ARG AIRFLOW_PIP_VERSION=23.3.2
# Setup PIP
# By default PIP install run without cache to make image smaller
@@ -1088,7 +1087,6 @@ ENV AIRFLOW_REPO=${AIRFLOW_REPO}\
DEFAULT_CONSTRAINTS_BRANCH=${DEFAULT_CONSTRAINTS_BRANCH} \
AIRFLOW_CI_BUILD_EPOCH=${AIRFLOW_CI_BUILD_EPOCH} \
AIRFLOW_PRE_CACHED_PIP_PACKAGES=${AIRFLOW_PRE_CACHED_PIP_PACKAGES} \
- INSTALL_PROVIDERS_FROM_SOURCES=${INSTALL_PROVIDERS_FROM_SOURCES} \
AIRFLOW_VERSION=${AIRFLOW_VERSION} \
AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION} \
# In the CI image we always:
@@ -1116,11 +1114,12 @@ RUN echo "Airflow version: ${AIRFLOW_VERSION}"
# force them on the main Airflow package. Currently we need no extra limits as PIP 23.1+ has much better
# dependency resolution and we do not need to limit the versions of the dependencies
#
-# Aiobotocore is limited for eager upgrade because it either causes a long backtracking or
-# conflict when we do not limit it. It seems that `pip` has a hard time figuring the right
-# combination of dependencies for aiobotocore, botocore, boto3 and s3fs together
+# boto3 is limited to <1.34 because of aiobotocore that only works with 1.33 and we want to help
+# `pip` to limit the versions it checks and limit backtracking, by explicitly specifying these limits
+# when performing eager upgrade of dependencies - this way it won't even consider 1.34 versions of boto
+# We should update it every time a new version of aiobotocore is released supporting 1.34
#
-ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="aiobotocore>=2.5.4"
+ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="boto3>=1.33,<1.34"
ARG UPGRADE_TO_NEWER_DEPENDENCIES="false"
ARG VERSION_SUFFIX_FOR_PYPI=""
@@ -1138,8 +1137,7 @@ COPY --from=scripts install_pip_version.sh install_airflow_dependencies_from_bra
# In case of CI builds we want to pre-install main version of airflow dependencies so that
# We do not have to always reinstall it from the scratch.
# And is automatically reinstalled from the scratch every time patch release of python gets released
-# The Airflow (and providers in case INSTALL_PROVIDERS_FROM_SOURCES is "false")
-# are uninstalled, only dependencies remain.
+# The Airflow and providers are uninstalled, only dependencies remain.
# the cache is only used when "upgrade to newer dependencies" is not set to automatically
# account for removed dependencies (we do not install them in the first place)
RUN bash /scripts/docker/install_pip_version.sh; \
@@ -1158,21 +1156,22 @@ COPY --from=scripts install_pipx_tools.sh /scripts/docker/
RUN bash /scripts/docker/install_pipx_tools.sh
# Airflow sources change frequently but dependency configuration won't change that often
-# We copy setup.py and other files needed to perform setup of dependencies
-# So in case setup.py changes we can install latest dependencies required.
-COPY setup.py ${AIRFLOW_SOURCES}/setup.py
-COPY setup.cfg ${AIRFLOW_SOURCES}/setup.cfg
+# We copy pyproject.toml and other files needed to perform setup of dependencies
+# So in case pyproject.toml changes we can install latest dependencies required.
+COPY pyproject.toml ${AIRFLOW_SOURCES}/pyproject.toml
COPY airflow/__init__.py ${AIRFLOW_SOURCES}/airflow/
-COPY generated/provider_dependencies.json ${AIRFLOW_SOURCES}/generated/
+COPY generated/* ${AIRFLOW_SOURCES}/generated/
COPY constraints/* ${AIRFLOW_SOURCES}/constraints/
-
+COPY LICENSE ${AIRFLOW_SOURCES}/LICENSE
+COPY dev/airflow_pre_installed_providers.txt ${AIRFLOW_SOURCES}/dev/airflow_pre_installed_providers.txt
+COPY dev/hatch_build.py ${AIRFLOW_SOURCES}/dev/hatch_build.py
COPY --from=scripts install_airflow.sh /scripts/docker/
-# The goal of this line is to install the dependencies from the most current setup.py from sources
+# The goal of this line is to install the dependencies from the most current pyproject.toml from sources
# This will be usually incremental small set of packages in CI optimized build, so it will be very fast
# In non-CI optimized build this will install all dependencies before installing sources.
-# Usually we will install versions based on the dependencies in setup.py and upgraded only if needed.
-# But in cron job we will install latest versions matching setup.py to see if there is no breaking change
+# Usually we will install versions based on the dependencies in pyproject.toml and upgraded only if needed.
+# But in cron job we will install latest versions matching pyproject.toml to see if there is no breaking change
# and push the constraints if everything is successful
RUN bash /scripts/docker/install_airflow.sh
diff --git a/IMAGES.rst b/IMAGES.rst
index 9cb6d94bb3..809d33a385 100644
--- a/IMAGES.rst
+++ b/IMAGES.rst
@@ -54,7 +54,7 @@ CI image
The CI image is used by `Breeze <BREEZE.rst>`_ as the shell image but it is also used during CI tests.
The image is single segment image that contains Airflow installation with "all" dependencies installed.
It is optimised for rebuild speed. It installs PIP dependencies from the current branch first -
-so that any changes in ``setup.py`` do not trigger reinstalling of all dependencies.
+so that any changes in ``pyproject.toml`` do not trigger reinstalling of all dependencies.
There is a second step of installation that re-installs the dependencies
from the latest sources so that we are sure that latest dependencies are installed.
diff --git a/INSTALL b/INSTALL
index 832fd8603d..8fdd933e90 100644
--- a/INSTALL
+++ b/INSTALL
@@ -1,118 +1,304 @@
# INSTALL / BUILD instructions for Apache Airflow
-This is a generic installation method that requires a number of dependencies to be installed.
+## Basic installation of Airflow from sources and development environment setup
+
+This is a generic installation method that requires minimum starndard tools to develop airflow and
+test it in local virtual environment (using standard CPyhon installation and `pip`).
Depending on your system you might need different prerequisites, but the following
systems/prerequisites are known to work:
-Linux (Debian Bullseye, Bookworm and Linux Mint Debbie):
+Linux (Debian Bookworm):
+
+ sudo apt install -y --no-install-recommends apt-transport-https apt-utils ca-certificates \
+ curl dumb-init freetds-bin gosu krb5-user libgeos-dev \
+ ldap-utils libsasl2-2 libsasl2-modules libxmlsec1 locales libffi8 libldap-2.5-0 libssl3 netcat-openbsd \
+ lsb-release openssh-client python3-selinux rsync sasl2-bin sqlite3 sudo unixodbc
+
+You might need to install MariaDB development headers to build some of the dependencies
+
+ sudo apt-get install libmariadb-dev libmariadbclient-dev
+
+MacOS (Mojave/Catalina) you might need to to install XCode command line tools and brew and those packages:
+
+ brew install sqlite mysql postgresql
+
+## Downloading and installing Airflow from sources
+
+While you can get Airflow sources in various ways (including cloning https://github.com/apache/airflow/), the
+canonical way to download it is to fetch the tarball published at https://downloads.apache.org where you can
+also verify checksum, signatures of the downloaded file. You can then and un-tar the source move into the
+directory that was un-tarred.
+
+When you download source packages from https://downloads.apache.org, you download sources of Airflow and
+all providers separately, however when you clone the GitHub repository at https://github.com/apache/airflow/
+you get all sources in one place. This is the most convenient way to develop Airflow and Providers together.
+otherwise you have to separately install Airflow and Providers from sources in the same environment, which
+is not as convenient.
+
+## Creating virtualenv
+
+Airflow pulls in quite a lot of dependencies in order to connect to other services. You generally want to
+test or run Airflow from a virtual env to make sure those dependencies are separated from your system
+wide versions. Using system-installed Python installation is strongly discouraged as the versions of Python
+shipped with operating system often have a number of limitations and are not up to date. It is recommended
+to install Python using either https://www.python.org/downloads/ or other tools that use them. See later
+for description of `Hatch` as one of the tools that is Airflow's tool of choice to build Airflow packages.
+
+Once you have a suitable Python version installed, you can create a virtualenv and activate it:
+
+ python3 -m venv PATH_TO_YOUR_VENV
+ source PATH_TO_YOUR_VENV/bin/activate
+
+## Installing airflow locally
+
+Installing airflow locally can be done using pip - note that this will install "development" version of
+Airflow, where all providers are installed from local sources (if they are available), not from `pypi`.
+It will also not include pre-installed providers installed from PyPI. In case you install from sources of
+just Airflow, you need to install separately each provider that you want to develop. In case you install
+from GitHub repository, all the current providers are available after installing Airflow.
+
+ pip install .
+
+If you develop Airflow and iterate on it you should install it in editable mode (with -e) flag and then
+you do not need to re-install it after each change to sources. This is useful if you want to develop and
+iterate on Airflow and Providers (together) if you install sources from cloned GitHub repository.
+
+Note that you might want to install `devel` extra when you install airflow for development in editable env
+as this one contains minimum set of tools and dependencies that are needed to run unit tests.
+
+
+ pip install -e ".[devel]"
+
+
+You can also install optional packages that are needed to run certain tests. In case of local installation
+for example you can install all prerequisites for google provider, tests and
+all hadoop providers with this command:
+
+ pip install -e ".[google,devel-tests,devel-hadoop]"
+
+
+or you can install all packages needed to run tests for core, providers and all extensions of airflow:
+
+ pip install -e ".[devel-all]"
+
+You can see the list of all available extras below.
+
+# Using Hatch to manage your Python, virtualenvs and build packages
+
+Airflow uses [hatch](https://hatch.pypa.io/) as a build and development tool of choice. It is one of popular
+build tools and environment managers for Python, maintained by the Python Packaging Authority.
+It is an optional tool that is only really needed when you want to build packages from sources, but
+it is also very convenient to manage your Python versions and virtualenvs.
+
+Airflow project contains some pre-defined virtualenv definitions in ``pyproject.toml`` that can be
+easily used by hatch to create your local venvs. This is not necessary for you to develop and test
+Airflow, but it is a convenient way to manage your local Python versions and virtualenvs.
+
+## Installing Hatch
+
+You can install hat using various other ways (including Gui installers).
+
+Example using `pipx`:
+
+ pipx install hatch
+
+We recommend using `pipx` as you can manage installed Python apps easily and later use it
+to upgrade `hatch` easily as needed with:
-sudo apt install build-essential python3-dev libsqlite3-dev openssl \
- sqlite default-libmysqlclient-dev libmysqlclient-dev postgresql
+ pipx upgrade hatch
-On Ubuntu 20.04 you may get an error of mariadb_config not found
-and mysql_config not found.
+## Using Hatch to manage your Python versions
-Install MariaDB development headers:
-sudo apt-get install libmariadb-dev libmariadbclient-dev
+You can also use hatch to install and manage airflow virtualenvs and development
+environments. For example, you can install Python 3.10 with this command:
-MacOS (Mojave/Catalina):
+ hatch python install 3.10
-brew install sqlite mysql postgresql
+or install all Python versions that are used in Airflow:
-# [required] fetch the tarball and untar the source move into the directory that was untarred.
+ hatch python install all
-# [optional] run Apache RAT (release audit tool) to validate license headers
-# RAT docs here: https://creadur.apache.org/rat/. Requires Java and Apache Rat
-java -jar apache-rat.jar -E ./.rat-excludes -d .
+## Using Hatch to manage your virtualenvs
-# [optional] Airflow pulls in quite a lot of dependencies in order
-# to connect to other services. You might want to test or run Airflow
-# from a virtual env to make sure those dependencies are separated
-# from your system wide versions
+Airflow has some pre-defined virtualenvs that you can use to develop and test airflow.
+You can see the list of available envs with:
-python3 -m venv PATH_TO_YOUR_VENV
-source PATH_TO_YOUR_VENV/bin/activate
+ hatch show env
-# [required] building and installing by pip (preferred)
-pip install .
+This is what it shows currently:
-# or directly
-python setup.py install
+┏━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
+┃ Name ┃ Type ┃ Features ┃ Description ┃
+┡━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
+│ default │ virtual │ devel │ Default environment with Python 3.8 for maximum compatibility │
+├─────────────┼─────────┼──────────┼───────────────────────────────────────────────────────────────┤
+│ airflow-38 │ virtual │ │ Environment with Python 3.8. No devel installed. │
+├─────────────┼─────────┼──────────┼───────────────────────────────────────────────────────────────┤
+│ airflow-39 │ virtual │ │ Environment with Python 3.9. No devel installed. │
+├─────────────┼─────────┼──────────┼───────────────────────────────────────────────────────────────┤
+│ airflow-310 │ virtual │ │ Environment with Python 3.10. No devel installed. │
+├─────────────┼─────────┼──────────┼───────────────────────────────────────────────────────────────┤
+│ airflow-311 │ virtual │ │ Environment with Python 3.11. No devel installed │
+└─────────────┴─────────┴──────────┴───────────────────────────────────────────────────────────────┘
-# You can also install recommended version of the dependencies by using
-# constraint-python<PYTHON_MAJOR_MINOR_VERSION>.txt files as constraint file. This is needed in case
-# you have problems with installing the current requirements from PyPI.
-# There are different constraint files for different python versions. For example"
+The default env (if you have not used one explicitly) is `default` and it is a Python 3.8
+virtualenv for maximum compatibility with `devel` extra installed - this devel extra contains the minimum set
+of dependencies and tools that should be used during unit testing of core Airflow and running all `airflow`
+CLI commands - without support for providers or databases.
-pip install . \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.8.txt"
+The other environments are just bare-bones Python virtualenvs with Airflow core requirements only,
+without any extras installed and without any tools. They are much faster to create than the default
+environment, and you can manually install either appropriate extras or directly tools that you need for
+testing or development.
+ hatch env create
-By default `pip install` in Airflow 2.0 installs only the provider packages that are needed by the extras and
-install them as packages from PyPI rather than from local sources:
+You can create specific environment by using them in create command:
-pip install .[google,amazon] \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.8.txt"
+ hatch env create airflow-310
+You can install extras in the environment by running pip command:
-You can upgrade just airflow, without paying attention to provider's dependencies by using 'constraints-no-providers'
-constraint files. This allows you to keep installed provider packages.
+ hatch -e airflow-310 run -- pip install -e ".[devel,google]"
-pip install . --upgrade \
+And you can enter the environment with running a shell of your choice (for example zsh) where you
+can run any commands
+
+ hatch -e airflow-310 shell
+
+Once you are in the environment (indicated usually by updated prompt), you can just install
+extra dependencies you need:
+
+ [~/airflow] [airflow-310] pip install -e ".[devel,google]"
+
+You can exit the environment by just exiting the shell.
+
+You can also see where hatch created the virtualenvs and use it in your IDE or activate it manually:
+
+ hatch env find airflow-310
+
+You will get path similar to:
+
+ /Users/jarek/Library/Application Support/hatch/env/virtual/apache-airflow/TReRdyYt/apache-airflow
+
+Then you will find `python` binary and `activate` script in the `bin` sub-folder of this directory and
+you can configure your IDE to use this python virtualenv if you want to use that environment in your IDE.
+
+You can also set default environment name by HATCH_ENV environment variable.
+
+You can clean the env by running:
+
+ hatch env prune
+
+More information about hatch can be found in https://hatch.pypa.io/1.9/environment/
+
+## Using Hatch to build your packages
+
+You can use hatch to build installable package from the airflow sources. Such package will
+include all metadata that is configured in `pyproject.toml` and will be installable with pip.
+
+The packages will have pre-installed dependencies for providers that are always
+installed when Airflow is installed from PyPI. By default both `wheel` and `sdist` packages are built.
+
+ hatch build
+
+You can also build only `wheel` or `sdist` packages:
+
+ hatch build -t wheel
+ hatch build -t sdist
+
+## Installing recommended version of dependencies
+
+Whatever virtualenv solution you use, when you want to make sure you are using the same
+version of dependencies as in main, you can install recommended version of the dependencies by using
+constraint-python<PYTHON_MAJOR_MINOR_VERSION>.txt files as `constraint` file. This might be useful
+to avoid "works-for-me" syndrome, where you use different version of dependencies than the ones
+that are used in main, CI tests and by other contributors.
+
+There are different constraint files for different python versions. For example this command will install
+all basic devel requirements and requirements of google provider as last successfully tested for Python 3.8:
+
+ pip install -e ".[devel,google]"" \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.8.txt"
+
+You can upgrade just airflow, without paying attention to provider's dependencies by using
+the 'constraints-no-providers' constraint files. This allows you to keep installed provider dependencies
+and install to latest supported ones by pure airflow core.
+
+pip install -e ".[devel]" \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-no-providers-3.8.txt"
+## All airflow extras
+
+Airflow has a number of extras that you can install to get additional dependencies. They sometimes install
+providers, sometimes enable other features where packages are not installed by default.
+
+You can read more about those extras in the extras reference:
+https://airflow.apache.org/docs/apache-airflow/stable/extra-packages-ref.html
+
+The list of available extras is below.
+
+Regular extras that are available for users in the Airflow package.
+
+# START REGULAR EXTRAS HERE
+aiobotocore, airbyte, alibaba, all, all-core, all-dbs, amazon, apache-atlas, apache-beam, apache-
+cassandra, apache-drill, apache-druid, apache-flink, apache-hdfs, apache-hive, apache-impala,
+apache-kafka, apache-kylin, apache-livy, apache-pig, apache-pinot, apache-spark, apache-webhdfs,
+apprise, arangodb, asana, async, atlas, atlassian-jira, aws, azure, cassandra, celery, cgroups,
+cloudant, cncf-kubernetes, cohere, common-io, common-sql, crypto, databricks, datadog, dbt-cloud,
+deprecated-api, dingding, discord, docker, druid, elasticsearch, exasol, facebook, ftp, gcp,
+gcp_api, github, github-enterprise, google, google-auth, graphviz, grpc, hashicorp, hdfs, hive,
+http, imap, influxdb, jdbc, jenkins, kerberos, kubernetes, ldap, leveldb, microsoft-azure,
+microsoft-mssql, microsoft-psrp, microsoft-winrm, mongo, mssql, mysql, neo4j, odbc, openai,
+openfaas, openlineage, opensearch, opsgenie, oracle, otel, pagerduty, pandas, papermill, password,
+pgvector, pinecone, pinot, postgres, presto, rabbitmq, redis, s3, s3fs, salesforce, samba, saml,
+segment, sendgrid, sentry, sftp, singularity, slack, smtp, snowflake, spark, sqlite, ssh, statsd,
+tableau, tabular, telegram, trino, vertica, virtualenv, weaviate, webhdfs, winrm, yandex, zendesk
+# END REGULAR EXTRAS HERE
+
+Devel extras - used to install development-related tools. Only available during editable install.
+
+# START DEVEL EXTRAS HERE
+devel, devel-all, devel-all-dbs, devel-ci, devel-debuggers, devel-devscripts, devel-duckdb, devel-
+hadoop, devel-mypy, devel-sentry, devel-static-checks, devel-tests
+# END DEVEL EXTRAS HERE
+
+Doc extras - used to install dependencies that are needed to build documentation. Only available during
+editable install.
+
+# START DOC EXTRAS HERE
+doc, doc-gen
+# END DOC EXTRAS HERE
-You can also install airflow in "editable mode" (with -e) flag and then provider packages are
-available directly from the sources (and the provider packages installed from PyPI are UNINSTALLED in
-order to avoid having providers in two places. And `provider.yaml` files are used to discover capabilities
-of the providers which are part of the airflow source code.
+## Compiling front end assets
-You can read more about `provider.yaml` and community-managed providers in
-https://airflow.apache.org/docs/apache-airflow-providers/index.html for developing custom providers
-and in ``CONTRIBUTING.rst`` for developing community maintained providers.
+Sometimes you can see that front-end assets are missing and website looks broken. This is because
+you need to compile front-end assets. This is done automatically when you create a virtualenv
+with hatch, but if you want to do it manually, you can do it after installing node and yarn and running:
-This is useful if you want to develop providers:
+ yarn install --frozen-lockfile
+ yarn run build
-pip install -e . \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.8.txt"
+Currently we are running yarn coming with note 18.6.0, but you should check the version in
+our `.pre-commit-config.yaml` file (node version).
-You can also skip installing provider packages from PyPI by setting INSTALL_PROVIDERS_FROM_SOURCE to "true".
-In this case Airflow will be installed in non-editable mode with all providers installed from the sources.
-Additionally `provider.yaml` files will also be copied to providers folders which will make the providers
-discoverable by Airflow even if they are not installed from packages in this case.
+Installing yarn is described in https://classic.yarnpkg.com/en/docs/install
-INSTALL_PROVIDERS_FROM_SOURCES="true" pip install . \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.8.txt"
+Also - in case you use `breeze` or have `pre-commit` installed you can build the assets with:
-Airflow can be installed with extras to install some additional features (for example 'async' or 'doc' or
-to install automatically providers and all dependencies needed by that provider:
+ pre-commit run --hook-stage manual compile-www-assets --all-files
-pip install .[async,google,amazon] \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.8.txt"
+or
-The list of available extras:
+ breeze compile-www-assets
-# START EXTRAS HERE
-aiobotocore, airbyte, alibaba, all, all_dbs, amazon, apache.atlas, apache.beam, apache.cassandra,
-apache.drill, apache.druid, apache.flink, apache.hdfs, apache.hive, apache.impala, apache.kafka,
-apache.kylin, apache.livy, apache.pig, apache.pinot, apache.spark, apache.webhdfs, apprise,
-arangodb, asana, async, atlas, atlassian.jira, aws, azure, cassandra, celery, cgroups, cloudant,
-cncf.kubernetes, cohere, common.io, common.sql, crypto, databricks, datadog, dbt.cloud,
-deprecated_api, devel, devel_all, devel_ci, devel_hadoop, dingding, discord, doc, doc_gen, docker,
-druid, elasticsearch, exasol, facebook, ftp, gcp, gcp_api, github, github_enterprise, google,
-google_auth, graphviz, grpc, hashicorp, hdfs, hive, http, imap, influxdb, jdbc, jenkins, kerberos,
-kubernetes, ldap, leveldb, microsoft.azure, microsoft.mssql, microsoft.psrp, microsoft.winrm, mongo,
-mssql, mysql, neo4j, odbc, openai, openfaas, openlineage, opensearch, opsgenie, oracle, otel,
-pagerduty, pandas, papermill, password, pgvector, pinecone, pinot, postgres, presto, rabbitmq,
-redis, s3, s3fs, salesforce, samba, saml, segment, sendgrid, sentry, sftp, singularity, slack, smtp,
-snowflake, spark, sqlite, ssh, statsd, tableau, tabular, telegram, trino, vertica, virtualenv,
-weaviate, webhdfs, winrm, yandex, zendesk
-# END EXTRAS HERE
+Both commands will install node and yarn if needed to a dedicated pre-commit node environment and
+then build the assets.
-# For installing Airflow in development environments - see CONTRIBUTING.rst
+Finally you can also clean and recompile assets with ``custom`` build target when running hatch build
-# COMPILING FRONT-END ASSETS (in case you see "Please make sure to build the frontend in static/ directory and then restart the server")
-# Optional : Installing yarn - https://classic.yarnpkg.com/en/docs/install
+ hatch build -t custom -t wheel -t sdist
-python setup.py compile_assets
+This will also update `git_version` file in airflow package that should contain the git commit hash of the
+build. This is used to display the commit hash in the UI.
diff --git a/LOCAL_VIRTUALENV.rst b/LOCAL_VIRTUALENV.rst
index 753f3a9f29..e7dc4d4ee4 100644
--- a/LOCAL_VIRTUALENV.rst
+++ b/LOCAL_VIRTUALENV.rst
@@ -21,26 +21,29 @@
Local Virtual Environment (virtualenv)
======================================
-Use the local virtualenv development option in combination with the `Breeze
-<BREEZE.rst#using-local-virtualenv-environment-in-your-host-ide>`_ development environment. This option helps
-you benefit from the infrastructure provided
-by your IDE (for example, IntelliJ PyCharm/IntelliJ Idea) and work in the
-environment where all necessary dependencies and tests are available and set up
-within Docker images.
+The easiest wey to run tests for Airflow is to use local virtualenv. While Breeze is the recommended
+way to run tests - because it provides a reproducible environment and is easy to set up, it is not
+always the best option as you need to run your tests inside a docker container. This might make it
+harder to debug the tests and to use your IDE to run them.
-But you can also use the local virtualenv as a standalone development option if you
-develop Airflow functionality that does not incur large external dependencies and
-CI test coverage.
+That's why we recommend using local virtualenv for development and testing.
-These are examples of the development options available with the local virtualenv in your IDE:
+The simplest way to install Airflow in local virtualenv is to use ``pip``:
-* local debugging;
-* Airflow source view;
-* auto-completion;
-* documentation support;
-* unit tests.
+.. code:: bash
-This document describes minimum requirements and instructions for using a standalone version of the local virtualenv.
+ pip install -e ".[devel,<OTHER EXTRAS>]" # for example: pip install -e ".[devel,google,postgres]"
+
+This will install Airflow in 'editable' mode - where sources of Airflow are taken directly from the source
+code rather than moved to the installation directory. You need to run this command in the virtualenv you
+want to install Airflow in - and you need to have the virtualenv activated.
+
+While you can use any virtualenv manager, we recommend using `Hatch <https://hatch.pypa.io/latest/>`__
+as your development environment front-end, and we already use Hatch backend ``hatchling`` for Airflow.
+
+Hatchling is automatically installed when you build Airflow but since airflow build system uses
+``PEP`` compliant ``pyproject.toml`` file, you can use any front-end build system that supports
+``PEP 517`` and ``PEP 518``. You can also use ``pip`` to install Airflow in editable mode.
Prerequisites
=============
@@ -88,7 +91,7 @@ Extra Packages
You can also install extra packages (like ``[ssh]``, etc) via
-``pip install -e [EXTRA1,EXTRA2 ...]``. However, some of them may
+``pip install -e [devel,EXTRA1,EXTRA2 ...]``. However, some of them may
have additional install and setup requirements for your local system.
For example, if you have a trouble installing the mysql client on macOS and get
@@ -108,129 +111,165 @@ You are STRONGLY encouraged to also install and use `pre-commit hooks <STATIC_CO
for your local virtualenv development environment. Pre-commit hooks can speed up your
development cycle a lot.
-The full list of extras is available in `<setup.py>`_.
+The full list of extras is available in `<pyproject.toml>`_ and can be easily retrieved using hatch via
-Creating a Local virtualenv
-===========================
-To use your IDE for Airflow development and testing, you need to configure a virtual
-environment. Ideally you should set up virtualenv for all Python versions that Airflow
-supports (3.8, 3.9, 3.10, 3.11).
+Using Hatch
+===========
-To create and initialize the local virtualenv:
+Airflow uses [hatch](https://hatch.pypa.io/) as a build and development tool of choice. It is one of popular
+build tools and environment managers for Python, maintained by the Python Packaging Authority.
+It is an optional tool that is only really needed when you want to build packages from sources, but
+it is also very convenient to manage your Python versions and virtualenvs.
-1. Create an environment with one of the two options:
+Airflow project contains some pre-defined virtualenv definitions in ``pyproject.toml`` that can be
+easily used by hatch to create your local venvs. This is not necessary for you to develop and test
+Airflow, but it is a convenient way to manage your local Python versions and virtualenvs.
- - Option 1: consider using one of the following utilities to create virtual environments and easily switch between them with the ``workon`` command:
+Installing Hatch
+----------------
- - `pyenv <https://github.com/pyenv/pyenv>`_
- - `pyenv-virtualenv <https://github.com/pyenv/pyenv-virtualenv>`_
- - `virtualenvwrapper <https://virtualenvwrapper.readthedocs.io/en/latest/>`_
+You can install hat using various other ways (including Gui installers).
- ``mkvirtualenv <ENV_NAME> --python=python<VERSION>``
+Example using ``pipx``:
- - Option 2: create a local virtualenv with Conda
+.. code:: bash
- - install `miniconda3 <https://docs.conda.io/en/latest/miniconda.html>`_
+ pipx install hatch
- .. code-block:: bash
+We recommend using ``pipx`` as you can manage installed Python apps easily and later use it
+to upgrade ``hatch`` easily as needed with:
- conda create -n airflow python=3.8 # or 3.9, 3.10, 3.11
- conda activate airflow
+.. code:: bash
-2. Install Python PIP requirements:
+ pipx upgrade hatch
-.. note::
+## Using Hatch to manage your Python versions
- Only ``pip`` installation is currently officially supported.
+You can also use hatch to install and manage airflow virtualenvs and development
+environments. For example, you can install Python 3.10 with this command:
- While they are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
- `pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as
- ``pip`` - especially when it comes to constraint vs. requirements management.
- Installing via ``Poetry`` or ``pip-tools`` is not currently supported.
+.. code:: bash
- There are known issues with ``bazel`` that might lead to circular dependencies when using it to install
- Airflow. Please switch to ``pip`` if you encounter such problems. ``Bazel`` community works on fixing
- the problem in `this PR <https://github.com/bazelbuild/rules_python/pull/1166>`_ so it might be that
- newer versions of ``bazel`` will handle it.
+ hatch python install 3.10
- If you wish to install airflow using those tools you should use the constraint files and convert
- them to appropriate format and workflow that your tool requires.
+or install all Python versions that are used in Airflow:
+.. code:: bash
- .. code-block:: bash
+ hatch python install all
- pip install --upgrade -e ".[devel,<OTHER EXTRAS>]" # for example: pip install --upgrade -e ".[devel,google,postgres]"
+Manage your virtualenvs with Hatch
+----------------------------------
-In case you have problems with installing airflow because of some requirements are not installable, you can
-try to install it with the set of working constraints (note that there are different constraint files
-for different python versions). For development on current main source:
+Airflow has some pre-defined virtualenvs that you can use to develop and test airflow.
+You can see the list of available envs with:
- .. code-block:: bash
+.. code:: bash
- # use the same version of python as you are working with, 3.8, 3.9, 3.10 or 3.11
- pip install -e ".[devel,<OTHER EXTRAS>]" \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.8.txt"
+ hatch show env
+
+This is what it shows currently:
+
++-------------+---------+----------+---------------------------------------------------------------+
+| Name | Type | Features | Description |
++=============+=========+==========+===============================================================+
+| default | virtual | devel | Default environment with Python 3.8 for maximum compatibility |
++-------------+---------+----------+---------------------------------------------------------------+
+| airflow-38 | virtual | devel | Environment with Python 3.8 |
++-------------+---------+----------+---------------------------------------------------------------+
+| airflow-39 | virtual | devel | Environment with Python 3.9 |
++-------------+---------+----------+---------------------------------------------------------------+
+| airflow-310 | virtual | devel | Environment with Python 3.10 |
++-------------+---------+----------+---------------------------------------------------------------+
+| airflow-311 | virtual | devel | Environment with Python 3.11 |
++-------------+---------+----------+---------------------------------------------------------------+
+
+The default env (if you have not used one explicitly) is ``default`` and it is a Python 3.8
+virtualenv for maximum compatibility with ``devel`` extra installed - this devel extra contains the minimum set
+of dependencies and tools that should be used during unit testing of core Airflow and running all ``airflow``
+CLI commands - without support for providers or databases.
+
+The other environments are just bare-bones Python virtualenvs with Airflow core requirements only,
+without any extras installed and without any tools. They are much faster to create than the default
+environment, and you can manually install either appropriate extras or directly tools that you need for
+testing or development.
-This will install Airflow in 'editable' mode - where sources of Airflow are taken directly from the source
-code rather than moved to the installation directory. During the installation airflow will install - but then
-automatically remove all provider packages installed from PyPI - instead it will automatically use the
-provider packages available in your local sources.
+.. code:: bash
+
+ hatch env create
+
+You can create specific environment by using them in create command:
+
+.. code:: bash
+
+ hatch env create airflow-310
+
+You can install extras in the environment by running pip command:
+
+.. code:: bash
+
+ hatch -e airflow-310 run -- pip install -e ".[devel,google]"
+
+And you can enter the environment with running a shell of your choice (for example zsh) where you
+can run any commands
+
+.. code:: bash
+
+ hatch -e airflow-310 shell
+
+
+Once you are in the environment (indicated usually by updated prompt), you can just install
+extra dependencies you need:
-You can also install Airflow in non-editable mode:
+.. code:: bash
+
+ [~/airflow] [airflow-310] pip install -e ".[devel,google]"
- .. code-block:: bash
- # use the same version of python as you are working with, 3.8, 3.9, 3.10 or 3.11
- pip install ".[devel,<OTHER EXTRAS>]" \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.8.txt"
+You can also see where hatch created the virtualenvs and use it in your IDE or activate it manually:
-This will copy the sources to directory where usually python packages are installed. You can see the list
-of directories via ``python -m site`` command. In this case the providers are installed from PyPI, not from
-sources, unless you set ``INSTALL_PROVIDERS_FROM_SOURCES`` environment variable to ``true``
+.. code:: bash
- .. code-block:: bash
+ hatch env find airflow-310
- # use the same version of python as you are working with, 3.8, 3.9, 3.10 or 3.11
- INSTALL_PROVIDERS_FROM_SOURCES="true" pip install ".[devel,<OTHER EXTRAS>]" \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.8.txt"
+You will get path similar to:
+.. code::
-Note: when you first initialize database (the next step), you may encounter some problems.
-This is because airflow by default will try to load in example dags where some of them requires dependencies ``google`` and ``postgres``.
-You can solve the problem by:
+ /Users/jarek/Library/Application Support/hatch/env/virtual/apache-airflow/TReRdyYt/apache-airflow
-- installing the extras i.e. ``[devel,google,postgres]`` or
-- disable the example dags with environment variable: ``export AIRFLOW__CORE__LOAD_EXAMPLES=False`` or
-- simply ignore the error messages and proceed
+Then you will find ``python`` binary and ``activate`` script in the ``bin`` sub-folder of this directory and
+you can configure your IDE to use this python virtualenv if you want to use that environment in your IDE.
-*In addition to above, you may also encounter problems during database migration.*
-*This is a known issue and please see the progress here:* `AIRFLOW-6265 <https://issues.apache.org/jira/browse/AIRFLOW-6265>`_
+You can also set default environment name by HATCH_ENV environment variable.
-3. Create the Airflow sqlite database:
+You can clean the env by running:
- .. code-block:: bash
+.. code:: bash
- # if necessary, start with a clean AIRFLOW_HOME, e.g.
- # rm -rf ~/airflow
- airflow db migrate
+ hatch env prune
-4. Select the virtualenv you created as the project's default virtualenv in your IDE.
+More information about hatch can be found in https://hatch.pypa.io/1.9/environment/
-Note that if you have the Breeze development environment installed, the ``breeze``
-script can automate initializing the created virtualenv (steps 2 and 3).
-Activate your virtualenv, e.g. by using ``workon``, and once you are in it, run:
+## Using Hatch to build your packages
-.. code-block:: bash
+You can use hatch to build installable package from the airflow sources. Such package will
+include all metadata that is configured in ``pyproject.toml`` and will be installable with pip.
- ./scripts/tools/initialize_virtualenv.py
+The packages will have pre-installed dependencies for providers that are always
+installed when Airflow is installed from PyPI. By default both ``wheel`` and ``sdist`` packages are built.
-By default Breeze installs the ``devel`` extra only. You can optionally control which extras are
-Adding extra dependencies as parameter.
+.. code:: bash
-.. code-block:: bash
+ hatch build
- ./scripts/tools/initialize_virtualenv.py devel,google,postgres
+You can also build only ``wheel`` or ``sdist`` packages:
+
+.. code:: bash
+
+ hatch build -t wheel
+ hatch build -t sdist
Developing Providers
@@ -240,22 +279,84 @@ In Airflow 2.0 we introduced split of Apache Airflow into separate packages - th
apache-airflow package with core of Airflow and 70+ packages for all providers (external services
and software Airflow can communicate with).
-Developing providers is part of Airflow development, but when you install airflow as editable in your local
-development environment, the corresponding provider packages will be also installed from PyPI. However, the
-providers will also be present in your "airflow/providers" folder. This might lead to confusion,
-which sources of providers are imported during development. It will depend on your
-environment's PYTHONPATH setting in general.
+When you install airflow from sources using editable install, you can develop together both - main version
+of Airflow and providers, which is pretty convenient, because you can use the same environment for both.
+
+
+Running ``pipinstall -e .`` will install Airflow in editable mode, but all provider code will also be
+available in the same environment. However, most provider need some additional dependencies.
+
+You can install the dependencies of the provider you want to develop by installing airflow in editable
+mode with provider id as extra. You can see the list of provider's extras in the
+`extras reference <./docs/apache-airflow/extra-packages-ref.rst>`_.
+
+For example, if you want to develop Google provider, you can install it with:
+
+.. code:: bash
+
+ pip install -e ".[devel,google]"
+
+In case of a provider has name compose of several segments, you can use ``-`` to separate them. You can also
+install multiple extra dependencies at a time:
+
+.. code:: bash
+
+ pip install -e ".[devel,apache-beam,dbt-cloud]"
+
+The dependencies for providers are configured in ``airflow/providers/PROVIDERS_FOLDER/provider.yaml`` file -
+separately for each provider. You can find there two types of ``dependencies`` - production runtime
+dependencies, and sometimes ``devel-dependencies`` which are needed to run tests. While ``provider.yaml``
+file is the single source of truth for the dependencies, eventually they need to find its way to Airflow`s
+``pyproject.toml``. This is done by running:
+
+.. code:: bash
+
+ pre-commit run update-providers-dependencies --all-files
+
+This will update ``pyproject.toml`` with the dependencies from ``provider.yaml`` files and from there
+it will be used automatically when you install Airflow in editable mode.
+
+If you want to add another dependency to a provider, you should add it to corresponding ``provider.yaml``,
+run the command above and commit the changes to ``pyproject.toml``. Then running
+``pip install -e .[devel,PROVIDER_EXTRA]`` will install the new dependencies. Tools like ``hatch`` can also
+install the dependencies automatically when you create or switch to a development environment.
+
+
+Installing recommended version of dependencies
+==============================================
+
+Whatever virtualenv solution you use, when you want to make sure you are using the same
+version of dependencies as in main, you can install recommended version of the dependencies by using
+constraint-python<PYTHON_MAJOR_MINOR_VERSION>.txt files as ``constraint`` file. This might be useful
+to avoid "works-for-me" syndrome, where you use different version of dependencies than the ones
+that are used in main, CI tests and by other contributors.
+
+There are different constraint files for different python versions. For example this command will install
+all basic devel requirements and requirements of google provider as last successfully tested for Python 3.8:
+
+.. code:: bash
+
+ pip install -e ".[devel,google]"" \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.8.txt"
+
+You can upgrade just airflow, without paying attention to provider's dependencies by using
+the 'constraints-no-providers' constraint files. This allows you to keep installed provider dependencies
+and install to latest supported ones by pure airflow core.
-In order to avoid the confusion, you can set ``INSTALL_PROVIDERS_FROM_SOURCES`` environment to ``true``
-before running ``pip install`` command:
+.. code:: bash
-.. code-block:: bash
+ pip install -e ".[devel]" \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-no-providers-3.8.txt"
- INSTALL_PROVIDERS_FROM_SOURCES="true" pip install -U -e ".[devel,<OTHER EXTRAS>]" \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.8.txt"
+These are examples of the development options available with the local virtualenv in your IDE:
-This way no providers packages will be installed and they will always be imported from the "airflow/providers"
-folder.
+* local debugging;
+* Airflow source view;
+* auto-completion;
+* documentation support;
+* unit tests.
+
+This document describes minimum requirements and instructions for using a standalone version of the local virtualenv.
Running Tests
@@ -263,23 +364,23 @@ Running Tests
Running tests is described in `TESTING.rst <TESTING.rst>`_.
-While most of the tests are typical unit tests that do not
-require external components, there are a number of Integration tests. You can technically use local
-virtualenv to run those tests, but it requires to set up a number of
-external components (databases/queues/kubernetes and the like). So, it is
-much easier to use the `Breeze <BREEZE.rst>`__ development environment
-for Integration tests.
+While most of the tests are typical unit tests that do not require external components, there are a number
+of Integration tests. You can technically use local virtualenv to run those tests, but it requires to
+set up all necessary dependencies for all the providers you are going to tests and also setup
+databases - and sometimes other external components (for integration test).
+
+So, generally it should be easier to use the `Breeze <BREEZE.rst>`__ development environment (especially
+for Integration tests).
-Note: Soon we will separate the integration and system tests out via pytest
-so that you can clearly know which tests are unit tests and can be run in
-the local virtualenv and which should be run using Breeze.
Connecting to database
----------------------
When analyzing the situation, it is helpful to be able to directly query the database. You can do it using
-the built-in Airflow command:
+the built-in Airflow command (however you needs a CLI client tool for each database to be installed):
.. code:: bash
airflow db shell
+
+The command will explain what CLI tool is needed for the database you have configured.
diff --git a/MANIFEST.in b/MANIFEST.in
deleted file mode 100644
index 5a636212b8..0000000000
--- a/MANIFEST.in
+++ /dev/null
@@ -1,43 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-include NOTICE
-include LICENSE
-include RELEASE_NOTES.rst
-include README.md
-graft licenses
-graft airflow/www
-graft airflow/www/static
-graft airflow/www/templates
-graft scripts/systemd
-graft airflow/config_templates
-recursive-exclude airflow/www/node_modules *
-global-exclude __pycache__ *.pyc
-exclude airflow/www/yarn.lock
-exclude airflow/www/*.sh
-include airflow/alembic.ini
-include airflow/api_connexion/openapi/v1.yaml
-include airflow/auth/managers/fab/openapi/v1.yaml
-include airflow/git_version
-include airflow/provider_info.schema.json
-include airflow/customized_form_field_behaviours.schema.json
-include airflow/serialization/schema.json
-include airflow/utils/python_virtualenv_script.jinja2
-include airflow/utils/context.pyi
-include airflow/example_dags/sql/sample.sql
-include generated
diff --git a/README.md b/README.md
index d55d4e8814..b992bd77ea 100644
--- a/README.md
+++ b/README.md
@@ -152,7 +152,7 @@ Documentation for dependent projects like provider packages, Docker image, Helm
We publish Apache Airflow as `apache-airflow` package in PyPI. Installing it however might be sometimes tricky
because Airflow is a bit of both a library and application. Libraries usually keep their dependencies open, and
applications usually pin them, but we should do neither and both simultaneously. We decided to keep
-our dependencies as open as possible (in `setup.py`) so users can install different versions of libraries
+our dependencies as open as possible (in `pyproject.toml`) so users can install different versions of libraries
if needed. This means that `pip install apache-airflow` will not work from time to time or will
produce unusable Airflow installation.
@@ -384,7 +384,7 @@ binding.
### Approach for dependencies for Airflow Core
-Those `extras` and `providers` dependencies are maintained in `setup.cfg`.
+Those dependencies are maintained in ``pyproject.toml``.
There are few dependencies that we decided are important enough to upper-bound them by default, as they are
known to follow predictable versioning scheme, and we know that new versions of those are very likely to
diff --git a/STATIC_CODE_CHECKS.rst b/STATIC_CODE_CHECKS.rst
index 81911d7375..68a0aef451 100644
--- a/STATIC_CODE_CHECKS.rst
+++ b/STATIC_CODE_CHECKS.rst
@@ -262,12 +262,12 @@ require Breeze Docker image to be built locally.
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
| check-pydevd-left-in-code | Check for pydevd debug statements accidentally left | |
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
+| check-pyproject-toml-order | Check order of dependencies in pyproject.toml | |
++-----------------------------------------------------------+--------------------------------------------------------------+---------+
| check-revision-heads-map | Check that the REVISION_HEADS_MAP is up-to-date | |
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
| check-safe-filter-usage-in-html | Don't use safe in templates | |
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-setup-order | Check order of dependencies in setup.cfg and setup.py | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
| check-start-date-not-used-in-defaults | start_date not to be defined in default_args in example_dags | |
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
| check-system-tests-present | Check if system tests have required segments of code | |
@@ -392,7 +392,7 @@ require Breeze Docker image to be built locally.
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
| update-migration-references | Update migration ref doc | * |
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| update-providers-dependencies | Update cross-dependencies for providers packages | |
+| update-providers-dependencies | Update dependencies for provider packages | |
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
| update-spelling-wordlist-to-be-sorted | Sort alphabetically and uniquify spelling_wordlist.txt | |
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
@@ -402,8 +402,6 @@ require Breeze Docker image to be built locally.
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
| update-version | Update version to the latest version in the documentation | |
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| validate-pyproject | Validate pyproject.toml | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
| yamllint | Check YAML files with yamllint | |
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
diff --git a/airflow/_vendor/README.md b/airflow/_vendor/README.md
index e76d8beea3..9d26a0257f 100644
--- a/airflow/_vendor/README.md
+++ b/airflow/_vendor/README.md
@@ -30,7 +30,7 @@ Way to vendor a library or update a version:
3. Replace them with new files (only replace relevant python packages:move LICENSE )
* move license files to [licenses](../../licenses) folder
* remove README and any other supporting files (they can be found in PyPI)
- * make sure to add requirements from setup.py to airflow's setup.py with appropriate comment stating
+ * make sure to add requirements to airflow's ``pyproject.toml`` with appropriate comment stating
why the requirements are added and when they should be removed
4. If you replace previous version, re-apply historical fixes from the "package" folder by
cherry-picking them.
diff --git a/airflow/cli/cli_parser.py b/airflow/cli/cli_parser.py
index e98e8ce52c..a295bd21ba 100644
--- a/airflow/cli/cli_parser.py
+++ b/airflow/cli/cli_parser.py
@@ -25,6 +25,7 @@ from __future__ import annotations
import argparse
import logging
+import sys
from argparse import Action
from collections import Counter
from functools import lru_cache
@@ -72,9 +73,13 @@ except Exception:
try:
auth_mgr = get_auth_manager_cls()
airflow_commands.extend(auth_mgr.get_cli_commands())
-except Exception:
- log.exception("cannot load CLI commands from auth manager")
+except Exception as e:
+ log.warning("cannot load CLI commands from auth manager: %s", e)
+ log.warning("Authentication manager is not configured and webserver will not be able to start.")
# do not re-raise for the same reason as above
+ if len(sys.argv) > 1 and sys.argv[1] == "webserver":
+ log.exception(e)
+ sys.exit(1)
ALL_COMMANDS_DICT: dict[str, CLICommand] = {sp.name: sp for sp in airflow_commands}
diff --git a/airflow/configuration.py b/airflow/configuration.py
index 7dca8a44d1..e4e3dab4b3 100644
--- a/airflow/configuration.py
+++ b/airflow/configuration.py
@@ -1191,7 +1191,7 @@ class AirflowConfigParser(ConfigParser):
try:
return import_string(full_qualified_path)
except ImportError as e:
- log.error(e)
+ log.warning(e)
raise AirflowConfigException(
f'The object could not be loaded. Please check "{key}" key in "{section}" section. '
f'Current value: "{full_qualified_path}".'
diff --git a/airflow/provider.yaml.schema.json b/airflow/provider.yaml.schema.json
index 2d439b7fda..faf66d8565 100644
--- a/airflow/provider.yaml.schema.json
+++ b/airflow/provider.yaml.schema.json
@@ -38,6 +38,13 @@
"type": "string"
}
},
+ "devel-dependencies": {
+ "description": "Dependencies that should be added to development requirements of the provider",
+ "type": "array",
+ "items": {
+ "type": "string"
+ }
+ },
"excluded-python-versions": {
"description": "List of python versions excluded for that provider",
"type": "array",
diff --git a/airflow/providers/MANAGING_PROVIDERS_LIFECYCLE.rst b/airflow/providers/MANAGING_PROVIDERS_LIFECYCLE.rst
index 9170786aff..0e4aa13081 100644
--- a/airflow/providers/MANAGING_PROVIDERS_LIFECYCLE.rst
+++ b/airflow/providers/MANAGING_PROVIDERS_LIFECYCLE.rst
@@ -195,11 +195,16 @@ Documentation
An important part of building a new provider is the documentation.
Some steps for documentation occurs automatically by ``pre-commit`` see `Installing pre-commit guide <https://github.com/apache/airflow/blob/main/CONTRIBUTORS_QUICK_START.rst#pre-commit>`_
+Those are important files in the airflow source tree that affect providers. The ``pyproject.toml`` in root
+Airflow folder is automatically generated based on content of ``provider.yaml`` file in each provider
+when ``pre-commit`` is run. Files such as ``extra-packages-ref.rst`` should be manually updated because
+they are manually formatted for better layout and ``pre-commit`` will just verify if the information
+about provider is updated there. Files like ``commit.rst`` and ``CHANGELOG`` are automatically updated
+by ``breeze release-management`` command by release manager when providers are released.
+
.. code-block:: bash
- ├── INSTALL
- ├── CONTRIBUTING.rst
- ├── setup.py
+ ├── pyproject.toml
├── airflow/
│ └── providers/
│ └── <NEW_PROVIDER>/
@@ -207,7 +212,6 @@ Some steps for documentation occurs automatically by ``pre-commit`` see `Install
│ └── CHANGELOG.rst
│
└── docs/
- ├── spelling_wordlist.txt
├── apache-airflow/
│ └── extra-packages-ref.rst
├── integration-logos/<NEW_PROVIDER>/
@@ -220,36 +224,8 @@ Some steps for documentation occurs automatically by ``pre-commit`` see `Install
└── <NEW_PROVIDER>.rst
-Files automatically updated by pre-commit:
-
-- ``INSTALL`` in provider
-
-Files automatically created when the provider is released:
-
-- ``docs/apache-airflow-providers-<NEW_PROVIDER>/commits.rst``
-- ``/airflow/providers/<NEW_PROVIDER>/CHANGELOG``
-
There is a chance that your provider's name is not a common English word.
-In this case is necessary to add it to the file ``docs/spelling_wordlist.txt``. This file begin with capitalized words and
-lowercase in the second block.
-
- .. code-block:: bash
-
- Namespace
- Neo4j
- Nextdoor
- <NEW_PROVIDER> (new line)
- Nones
- NotFound
- Nullable
- ...
- neo4j
- neq
- networkUri
- <NEW_PROVIDER> (new line)
- nginx
- nobr
- nodash
+In this case is necessary to add it to the file ``docs/spelling_wordlist.txt``.
Add your provider dependencies into ``provider.yaml`` under ``dependencies`` key..
If your provider doesn't have any dependency add a empty list.
@@ -258,9 +234,9 @@ In the ``docs/apache-airflow-providers-<NEW_PROVIDER>/connections.rst``:
- add information how to configure connection for your provider.
-In the ``docs/apache-airflow-providers-<NEW_PROVIDER>/operators/<NEW_PROVIDER>.rst``:
-
-- add information how to use the Operator. It's important to add examples and additional information if your Operator has extra-parameters.
+In the ``docs/apache-airflow-providers-<NEW_PROVIDER>/operators/<NEW_PROVIDER>.rst`` add information
+how to use the Operator. It's important to add examples and additional information if your
+Operator has extra-parameters.
.. code-block:: RST
@@ -284,7 +260,7 @@ In the ``docs/apache-airflow-providers-<NEW_PROVIDER>/operators/<NEW_PROVIDER>.r
:end-before: [END howto_operator_<NEW_PROVIDER>]
-Copy from another, similar provider the docs: ``docs/apache-airflow-providers-new_provider/*.rst``:
+Copy from another, similar provider the docs: ``docs/apache-airflow-providers-<NEW_PROVIDER>/*.rst``:
At least those docs should be present
@@ -336,20 +312,6 @@ In the ``airflow/providers/<NEW_PROVIDER>/provider.yaml`` add information of you
- hook-class-name: airflow.providers.<NEW_PROVIDER>.hooks.<NEW_PROVIDER>.NewProviderHook
- connection-type: provider-connection-type
- hook-class-names: # deprecated in Airflow 2.2.0
- - airflow.providers.<NEW_PROVIDER>.hooks.<NEW_PROVIDER>.NewProviderHook
-
-.. note:: Defining your own connection types
-
- You only need to add ``connection-types`` in case you have some hooks that have customized UI behavior. However,
- it is only supported for Airflow 2.2.0. If your providers are also targeting Airflow below 2.2.0 you should
- provide the deprecated ``hook-class-names`` array. The ``connection-types`` array allows for optimization
- of importing of individual connections and while Airflow 2.2.0 is able to handle both definition, the
- ``connection-types`` is recommended.
-
- For more information see `Custom connection types <http://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#custom-connection-types>`_
-
-
After changing and creating these files you can build the documentation locally. The two commands below will
serve to accomplish this. The first will build your provider's documentation. The second will ensure that the
main Airflow documentation that involves some steps with the providers is also working.
@@ -460,6 +422,19 @@ the compatibility checks should be updated when min airflow version is updated.
Details on how this should be done are described in
`Provider policies <https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md>`_
+Releasing pre-installed providers for the first time
+====================================================
+
+When releasing providers for the first time, you need to release them in state ``not-ready``.
+This will make it available for release management commands, but it will not be added to airflow's
+preinstalled providers list - allowing airflow in main ``CI`` builds to be built without expecting the
+provider to be available in PyPI.
+
+You need to add ``--include-not-ready-providers`` if you want to add them to the list of providers
+considered by the release management commands.
+
+As soon as the provider is released, you should update the provider to ``state: ready``.
+
Suspending providers
====================
@@ -488,14 +463,13 @@ to do.
* You will have to run ``breeze setup regenerate-command-images`` to regenerate breeze help files
* you will need to update ``extra-packages-ref.rst`` and in some cases - when mentioned there explicitly -
- ``setup.py`` to remove the provider from list of dependencies.
+ ``pyproject.toml`` to remove the provider from list of dependencies.
-What happens under-the-hood as a result, is that ``generated/providers.json`` file is updated with
+What happens under-the-hood as a result, is that ``pyproject.toml`` file is updated with
the information about available providers and their dependencies and it is used by our tooling to
exclude suspended providers from all relevant parts of the build and CI system (such as building CI image
with dependencies, building documentation, running tests, etc.)
-
Resuming providers
==================
@@ -503,7 +477,6 @@ Resuming providers is done by reverting the original change that suspended it. I
needed to fix problems in the reverted provider, our CI will detect them and you will have to fix them
as part of the PR reverting the suspension.
-
Removing providers
==================
diff --git a/airflow/providers/amazon/provider.yaml b/airflow/providers/amazon/provider.yaml
index f9687dcbfa..8fe5b1496e 100644
--- a/airflow/providers/amazon/provider.yaml
+++ b/airflow/providers/amazon/provider.yaml
@@ -92,6 +92,36 @@ dependencies:
- sqlalchemy_redshift>=0.8.6
- asgiref
+additional-extras:
+ - name: pandas
+ dependencies:
+ - pandas>=1.2.5
+ # There is conflict between boto3 and aiobotocore dependency botocore.
+ # TODO: We can remove it once boto3 and aiobotocore both have compatible botocore version or
+ # boto3 have native async support and we move away from aio aiobotocore
+ - name: aiobotocore
+ dependencies:
+ - aiobotocore[boto3]>=2.5.3
+ - name: cncf.kubernetes
+ dependencies:
+ - apache-airflow-providers-cncf-kubernetes>=7.2.0
+ - name: s3fs
+ dependencies:
+ - s3fs>=2023.10.0
+ - name: python3-saml
+ dependencies:
+ - python3-saml>=1.16.0
+
+devel-dependencies:
+ - aiobotocore>=2.7.0
+ - aws_xray_sdk>=2.12.0
+ - moto[cloudformation,glue]>=4.2.12
+ - mypy-boto3-appflow>=1.33.0
+ - mypy-boto3-rds>=1.33.0
+ - mypy-boto3-redshift-data>=1.33.0
+ - mypy-boto3-s3>=1.33.0
+ - s3fs>=2023.10.0
+
integrations:
- integration-name: Amazon Athena
external-doc-url: https://aws.amazon.com/athena/
@@ -735,26 +765,6 @@ logging:
- airflow.providers.amazon.aws.log.s3_task_handler.S3TaskHandler
- airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler
-additional-extras:
- - name: pandas
- dependencies:
- - pandas>=1.2.5
- # There is conflict between boto3 and aiobotocore dependency botocore.
- # TODO: We can remove it once boto3 and aiobotocore both have compatible botocore version or
- # boto3 have native async support and we move away from aio aiobotocore
- - name: aiobotocore
- dependencies:
- - aiobotocore[boto3]>=2.5.3
- - name: cncf.kubernetes
- dependencies:
- - apache-airflow-providers-cncf-kubernetes>=7.2.0
- - name: s3fs
- dependencies:
- - s3fs>=2023.10.0
- - name: python3-saml
- dependencies:
- - python3-saml>=1.16.0
-
config:
aws:
description: This section contains settings for Amazon Web Services (AWS) integration.
diff --git a/airflow/providers/apache/beam/provider.yaml b/airflow/providers/apache/beam/provider.yaml
index ddf8100ede..339f0c23e5 100644
--- a/airflow/providers/apache/beam/provider.yaml
+++ b/airflow/providers/apache/beam/provider.yaml
@@ -56,6 +56,11 @@ dependencies:
- apache-beam>=2.53.0
- pyarrow>=14.0.1
+additional-extras:
+ - name: google
+ dependencies:
+ - apache-beam[gcp]
+
integrations:
- integration-name: Apache Beam
external-doc-url: https://beam.apache.org/
@@ -77,8 +82,3 @@ triggers:
- integration-name: Apache Beam
python-modules:
- airflow.providers.apache.beam.triggers.beam
-
-additional-extras:
- - name: google
- dependencies:
- - apache-beam[gcp]
diff --git a/airflow/providers/apache/hive/provider.yaml b/airflow/providers/apache/hive/provider.yaml
index a3aefe854c..51114871a4 100644
--- a/airflow/providers/apache/hive/provider.yaml
+++ b/airflow/providers/apache/hive/provider.yaml
@@ -66,7 +66,7 @@ dependencies:
- apache-airflow-providers-common-sql>=1.3.1
- hmsclient>=0.1.0
- pandas>=1.2.5
- - pyhive[hive_pure_sasl]>=0.7.0
+ - pyhive[hive-pure-sasl]>=0.7.0
- thrift>=0.9.2
integrations:
diff --git a/airflow/providers/apache/impala/provider.yaml b/airflow/providers/apache/impala/provider.yaml
index b6cc987088..53cc680669 100644
--- a/airflow/providers/apache/impala/provider.yaml
+++ b/airflow/providers/apache/impala/provider.yaml
@@ -37,6 +37,12 @@ dependencies:
- impyla>=0.18.0,<1.0
- apache-airflow>=2.6.0
+additional-extras:
+ - name: kerberos
+ dependencies:
+ - kerberos>=1.3.0
+
+
integrations:
- integration-name: Apache Impala
external-doc-url: https://impala.apache.org
@@ -50,8 +56,3 @@ hooks:
connection-types:
- hook-class-name: airflow.providers.apache.impala.hooks.impala.ImpalaHook
connection-type: impala
-
-additional-extras:
- - name: kerberos
- dependencies:
- - kerberos>=1.3.0
diff --git a/airflow/providers/apache/spark/provider.yaml b/airflow/providers/apache/spark/provider.yaml
index 992deffadd..a8062c4734 100644
--- a/airflow/providers/apache/spark/provider.yaml
+++ b/airflow/providers/apache/spark/provider.yaml
@@ -56,6 +56,11 @@ dependencies:
- pyspark
- grpcio-status>=1.59.0
+additional-extras:
+ - name: cncf.kubernetes
+ dependencies:
+ - apache-airflow-providers-cncf-kubernetes>=7.4.0
+
integrations:
- integration-name: Apache Spark
external-doc-url: https://spark.apache.org/
@@ -94,8 +99,3 @@ connection-types:
task-decorators:
- class-name: airflow.providers.apache.spark.decorators.pyspark.pyspark_task
name: pyspark
-
-additional-extras:
- - name: cncf.kubernetes
- dependencies:
- - apache-airflow-providers-cncf-kubernetes>=7.4.0
diff --git a/airflow/providers/celery/provider.yaml b/airflow/providers/celery/provider.yaml
index 962fe57c9d..ad32e9f7d6 100644
--- a/airflow/providers/celery/provider.yaml
+++ b/airflow/providers/celery/provider.yaml
@@ -57,6 +57,12 @@ dependencies:
- flower>=1.0.0
- google-re2>=1.0
+additional-extras:
+ - name: cncf.kubernetes
+ dependencies:
+ - apache-airflow-providers-cncf-kubernetes>=7.4.0
+
+
integrations:
- integration-name: Celery
external-doc-url: https://docs.celeryq.dev/en/stable/
@@ -324,8 +330,3 @@ config:
sensitive: true
example: '{"password": "password_for_redis_server"}'
default: ~
-
-additional-extras:
- - name: cncf.kubernetes
- dependencies:
- - apache-airflow-providers-cncf-kubernetes>=7.4.0
diff --git a/airflow/providers/databricks/provider.yaml b/airflow/providers/databricks/provider.yaml
index de7829f1cf..a8c65d76a4 100644
--- a/airflow/providers/databricks/provider.yaml
+++ b/airflow/providers/databricks/provider.yaml
@@ -67,6 +67,16 @@ dependencies:
- databricks-sql-connector>=2.0.0, <3.0.0, !=2.9.0
- aiohttp>=3.6.3, <4
+additional-extras:
+ # pip install apache-airflow-providers-databricks[sdk]
+ - name: sdk
+ description: Install Databricks SDK
+ dependencies:
+ - databricks-sdk==0.10.0
+
+devel-dependencies:
+ - deltalake>=0.12.0
+
integrations:
- integration-name: Databricks
external-doc-url: https://databricks.com/
@@ -129,10 +139,3 @@ connection-types:
extra-links:
- airflow.providers.databricks.operators.databricks.DatabricksJobRunLink
-
-additional-extras:
- # pip install apache-airflow-providers-databricks[sdk]
- - name: sdk
- description: Install Databricks SDK
- dependencies:
- - databricks-sdk==0.10.0
diff --git a/airflow/providers/google/provider.yaml b/airflow/providers/google/provider.yaml
index aefc4d4997..eb8a728211 100644
--- a/airflow/providers/google/provider.yaml
+++ b/airflow/providers/google/provider.yaml
@@ -145,6 +145,26 @@ dependencies:
- sqlalchemy-bigquery>=1.2.1
- sqlalchemy-spanner>=1.6.2
+additional-extras:
+ - name: apache.beam
+ dependencies:
+ - apache-beam[gcp]
+ - name: cncf.kubernetes
+ dependencies:
+ - apache-airflow-providers-cncf-kubernetes>=7.2.0
+ - name: leveldb
+ dependencies:
+ - plyvel
+ - name: oracle
+ dependencies:
+ - apache-airflow-providers-oracle>=3.1.0
+ - name: facebook
+ dependencies:
+ - apache-airflow-providers-facebook>=2.2.0
+ - name: amazon
+ dependencies:
+ - apache-airflow-providers-amazon>=2.6.0
+
integrations:
- integration-name: Google Analytics (GA4)
external-doc-url: https://analytics.google.com/
@@ -1218,26 +1238,6 @@ extra-links:
- airflow.providers.google.common.links.storage.FileDetailsLink
- airflow.providers.google.marketing_platform.links.analytics_admin.GoogleAnalyticsPropertyLink
-additional-extras:
- - name: apache.beam
- dependencies:
- - apache-beam[gcp]
- - name: cncf.kubernetes
- dependencies:
- - apache-airflow-providers-cncf-kubernetes>=7.2.0
- - name: leveldb
- dependencies:
- - plyvel
- - name: oracle
- dependencies:
- - apache-airflow-providers-oracle>=3.1.0
- - name: facebook
- dependencies:
- - apache-airflow-providers-facebook>=2.2.0
- - name: amazon
- dependencies:
- - apache-airflow-providers-amazon>=2.6.0
-
secrets-backends:
- airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
diff --git a/airflow/providers/microsoft/azure/provider.yaml b/airflow/providers/microsoft/azure/provider.yaml
index fc3aefaf9c..ffe397024d 100644
--- a/airflow/providers/microsoft/azure/provider.yaml
+++ b/airflow/providers/microsoft/azure/provider.yaml
@@ -95,6 +95,9 @@ dependencies:
- azure-mgmt-containerregistry>=8.0.0
- azure-mgmt-containerinstance>=9.0.0
+devel-dependencies:
+ - pywinrm
+
integrations:
- integration-name: Microsoft Azure Batch
external-doc-url: https://azure.microsoft.com/en-us/services/batch/
diff --git a/airflow/providers/mongo/provider.yaml b/airflow/providers/mongo/provider.yaml
index 9e2bea6859..f22d7035de 100644
--- a/airflow/providers/mongo/provider.yaml
+++ b/airflow/providers/mongo/provider.yaml
@@ -48,6 +48,9 @@ dependencies:
- dnspython>=1.13.0
- pymongo>=3.6.0
+devel-dependencies:
+ - mongomock
+
integrations:
- integration-name: MongoDB
external-doc-url: https://www.mongodb.com/
diff --git a/airflow/providers/mysql/provider.yaml b/airflow/providers/mysql/provider.yaml
index 22d1f364ef..48c74bcd7c 100644
--- a/airflow/providers/mysql/provider.yaml
+++ b/airflow/providers/mysql/provider.yaml
@@ -60,6 +60,11 @@ dependencies:
- mysqlclient>=1.3.6
- mysql-connector-python>=8.0.29
+additional-extras:
+ # only needed for backwards compatibility
+ - name: mysql-connector-python
+ dependencies: []
+
integrations:
- integration-name: MySQL
external-doc-url: https://www.mysql.com/
@@ -97,8 +102,3 @@ transfers:
connection-types:
- hook-class-name: airflow.providers.mysql.hooks.mysql.MySqlHook
connection-type: mysql
-
-additional-extras:
- # only needed for backwards compatibility
- - name: mysql-connector-python
- dependencies: []
diff --git a/airflow/providers/oracle/provider.yaml b/airflow/providers/oracle/provider.yaml
index 89b6480d0e..59bc4b2ab5 100644
--- a/airflow/providers/oracle/provider.yaml
+++ b/airflow/providers/oracle/provider.yaml
@@ -56,17 +56,18 @@ dependencies:
- apache-airflow-providers-common-sql>=1.3.1
- oracledb>=1.0.0
+additional-extras:
+ - name: numpy
+ dependencies:
+ - numpy
+
+
integrations:
- integration-name: Oracle
external-doc-url: https://www.oracle.com/en/database/
logo: /integration-logos/oracle/Oracle.png
tags: [software]
-additional-extras:
- - name: numpy
- dependencies:
- - numpy
-
operators:
- integration-name: Oracle
python-modules:
diff --git a/airflow/providers/postgres/provider.yaml b/airflow/providers/postgres/provider.yaml
index c6e4338978..5fcb7068d4 100644
--- a/airflow/providers/postgres/provider.yaml
+++ b/airflow/providers/postgres/provider.yaml
@@ -60,6 +60,11 @@ dependencies:
- apache-airflow-providers-common-sql>=1.3.1
- psycopg2-binary>=2.8.0
+additional-extras:
+ - name: amazon
+ dependencies:
+ - apache-airflow-providers-amazon>=2.6.0
+
integrations:
- integration-name: PostgreSQL
external-doc-url: https://www.postgresql.org/
@@ -81,8 +86,3 @@ hooks:
connection-types:
- hook-class-name: airflow.providers.postgres.hooks.postgres.PostgresHook
connection-type: postgres
-
-additional-extras:
- - name: amazon
- dependencies:
- - apache-airflow-providers-amazon>=2.6.0
diff --git a/airflow/providers/tabular/provider.yaml b/airflow/providers/tabular/provider.yaml
index 7b161a41d2..47bd18547d 100644
--- a/airflow/providers/tabular/provider.yaml
+++ b/airflow/providers/tabular/provider.yaml
@@ -36,6 +36,9 @@ versions:
dependencies:
- apache-airflow>=2.6.0
+devel-dependencies:
+ - pyiceberg>=0.5.0
+
integrations:
- integration-name: Tabular
external-doc-url: https://tabular.io/docs/
diff --git a/airflow/providers_manager.py b/airflow/providers_manager.py
index 5ca1b6740e..01bc9bc39a 100644
--- a/airflow/providers_manager.py
+++ b/airflow/providers_manager.py
@@ -959,6 +959,14 @@ class ProvidersManager(LoggingMixin, metaclass=Singleton):
field_behaviours = hook_class.get_ui_field_behaviour()
if field_behaviours:
self._add_customized_fields(package_name, hook_class, field_behaviours)
+ except ImportError as e:
+ if "No module named 'flask_appbuilder'" in e.msg:
+ log.warning(
+ "The hook_class '%s' is not fully initialized (UI widgets will be missing), because "
+ "the 'flask_appbuilder' package is not installed, however it is not required for "
+ "Airflow components to work",
+ hook_class_name,
+ )
except Exception as e:
log.warning(
"Exception when importing '%s' from '%s' package: %s",
diff --git a/airflow/www/webpack.config.js b/airflow/www/webpack.config.js
index 6ac1f3a208..4e3de410bc 100644
--- a/airflow/www/webpack.config.js
+++ b/airflow/www/webpack.config.js
@@ -275,7 +275,7 @@ const config = {
}),
new LicensePlugin({
additionalFiles: {
- "../../../../licenses/LICENSES-ui.txt": formatLicenses,
+ "../../../../3rd-party-licenses/LICENSES-ui.txt": formatLicenses,
},
unacceptableLicenseTest: (licenseIdentifier) =>
[
diff --git a/clients/gen/common.sh b/clients/gen/common.sh
index cd78d6d039..0437028351 100755
--- a/clients/gen/common.sh
+++ b/clients/gen/common.sh
@@ -59,8 +59,7 @@ function validate_input {
git_push.sh
.gitlab-ci.yml
requirements.txt
-setup.cfg
-setup.py
+pyproject.toml
test-requirements.txt
tox.ini
EOF
diff --git a/dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md b/dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md
index 29b25f5f11..d74514d31e 100644
--- a/dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md
+++ b/dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md
@@ -424,9 +424,10 @@ The slight risk is that if there is a constraint problem that impacts regular PR
make all PRs "red" until the constraint is fixed. However, if this is the case then usually we should fix
the problem by fixing the tests or dependencies and the automated CI process should be able to self-heal.
The main build does not use constraints and it will attempt to upgrade (or downgrade) the dependencies to
-the latest version matching the dependency specification we have in setup.cfg/setup.py/provider.yaml files.
-Also the constraints are pushed without `--force` so there is no risk of destroying anything.
-The history is kept in Git, so you can always revert to the previous version if needed.
+the latest version matching the dependency specification we have in `pyproject.toml` files (note that provider
+dependencies in `pyproject.toml` are generated from provider.yaml files being the single source of truth for
+provider dependencies). Also, the constraints are pushed without `--force` so there is no risk of destroying
+anything. The history is kept in Git, so you can always revert to the previous version if needed.
# Manually updating already tagged constraint files
diff --git a/dev/README.md b/dev/README.md
index 5ed5929233..9ffd0798cc 100644
--- a/dev/README.md
+++ b/dev/README.md
@@ -23,6 +23,7 @@
- [Apache Airflow source releases](#apache-airflow-source-releases)
- [Apache Airflow Package](#apache-airflow-package)
- [Provider packages](#provider-packages)
+- [Preinstalled providers](#preinstalled-providers)
- [Prerequisites for the release manager preparing the release](#prerequisites-for-the-release-manager-preparing-the-release)
- [Upload Public keys to id.apache.org and GitHub](#upload-public-keys-to-idapacheorg-and-github)
- [Configure PyPI uploads](#configure-pypi-uploads)
@@ -95,7 +96,6 @@ The full provider's list can be found here:
There are also convenience packages released as "apache-airflow-providers"separately in PyPI.
[PyPI query for providers](https://pypi.org/search/?q=apache-airflow-providers)
-
And available in PyPI:
[PyPI query for backport providers](https://pypi.org/search/?q=apache-airflow-backport-providers).
@@ -105,6 +105,15 @@ of Backport Providers was done on March 17, 2021.
Detailed instruction of releasing Provider Packages can be found in the
[README_RELEASE_PROVIDER_PACKAGES.md](README_RELEASE_PROVIDER_PACKAGES.md)
+# Preinstalled providers
+
+The `dev/preinstalled-providers.json` file contains the list of provider ids that are pre-installed.
+Those providers are dynamically added to generated standard wheel packages that are released in PyPI.
+Those packages are not present in pyproject.toml as dependencies, and
+they are not installed when you install Airflow for editable installation for development.
+This way, when you develop Airflow you can work on Airflow and Providers together from the same
+Source tree - without polluting your editable installation with installed provider packages.
+
# Prerequisites for the release manager preparing the release
The person acting as release manager has to fulfill certain pre-requisites. More details and FAQs are
diff --git a/dev/README_RELEASE_AIRFLOW.md b/dev/README_RELEASE_AIRFLOW.md
index 5b1ad59a40..dabc02ea2d 100644
--- a/dev/README_RELEASE_AIRFLOW.md
+++ b/dev/README_RELEASE_AIRFLOW.md
@@ -76,7 +76,7 @@ For obvious reasons, you can't cherry-pick every change from `main` into the rel
some are incompatible without a large set of other changes, some are brand-new features, and some just don't need to be in a release.
In general only security fixes, data-loss bugs and regression fixes are essential to bring into a patch release;
-also changes in dependencies (setup.py, setup.cfg) resulting from releasing newer versions of packages that Airflow depends on.
+also changes in dependencies (pyproject.toml) resulting from releasing newer versions of packages that Airflow depends on.
Other bugfixes can be added on a best-effort basis, but if something is going to be very difficult to backport
(maybe it has a lot of conflicts, or heavily depends on a new feature or API that's not being backported),
it's OK to leave it out of the release at your sole discretion as the release manager -
@@ -772,7 +772,7 @@ that have min-airflow version set to X.Y.0
* NOTE! WE MIGHT WANT TO AUTOMATE THAT STEP IN THE FUTURE
-1. Checkout the constraints-2-* branch and update the ``constraints-3*.txt`` file with the new provider
+1. Checkout the ``constraints-2-*`` branch and update the ``constraints-3*.txt`` file with the new provider
version. Find the place where the provider should be added, add it with the latest provider version.
```
@@ -793,7 +793,7 @@ git push -f apache constraints-X.Y.Z
that have >= ``X.Y.0`` in the corresponding provider.yaml file.
-3. In case the provider should also be installed in the image (it is part of ``airflow/providers/installed_providers.txt``)
+3. In case the provider should also be installed in the image (it is part of ``dev/prod_image_installed_providers.txt``)
it should also be added at this moment to ``Dockerfile`` to the list of default extras in the line with ``AIRFLOW_EXTRAS``:
```Dockerfile
diff --git a/dev/README_RELEASE_PROVIDER_PACKAGES.md b/dev/README_RELEASE_PROVIDER_PACKAGES.md
index ba7c68b103..b97e920685 100644
--- a/dev/README_RELEASE_PROVIDER_PACKAGES.md
+++ b/dev/README_RELEASE_PROVIDER_PACKAGES.md
@@ -127,29 +127,27 @@ Details about maintaining the SEMVER version are going to be discussed and imple
The provider packages can be in one of several states.
-* The `ready` state the provider package is released as part of the regular release cycle (including the
- documentation, package building and publishing). This is the default state for all providers.
-* The `not-ready` state is when the provider has `not-ready` field set to `true` in the `provider.yaml` file.
- This is usually used when the provider has some in-progress changes (usually API changes) that we do not
- want to release yet as part of the regular release cycle. Providers in this state are excluded from being
- released as part of the regular release cycle (including documentation building). You can build and prepare
- such provider when you explicitly specify it as argument of a release command or by passing
+* The `not-ready` state is used when the provider has some in-progress changes (usually API changes) that
+ we do not want to release yet as part of the regular release cycle. Providers in this state are excluded
+ from being released as part of the regular release cycle (including documentation building). You can build
+ and prepare such provider when you explicitly specify it as argument of a release command or by passing
`--include-not-ready-providers` flag in corresponding command. The `not-ready` providers are treated as
regular providers when it comes to running tests and preparing and releasing packages in `CI` - as we want
to make sure they are properly releasable any time and we want them to contribute to dependencies and we
- want to test them.
-* The `suspended` state is when the provider has `suspended` field set to `true` in the `provider.yaml` file.
- This is used when we have a good reason to suspend such provider, following the devlist discussion and
- vote or "lazy consensus". The process of suspension is described in [Provider's docs](../PROVIDERS.rst).
+ want to test them. Also in case of preinstalled providers, the `not-ready` providers are contributing
+ their dependencies rather than the provider package to requirements of Airflow.
+* The `ready` state is the usual state of the provider that is released in the regular release cycle
+ (including the documentation, package building and publishing). This is the state most providers are in.
+* The `suspended` state is used when we have a good reason to suspend such provider, following the devlist
+ discussion and vote or "lazy consensus". The process of suspension is described in [Provider's docs](../PROVIDERS.rst).
The `suspended` providers are excluded from being released as part of the regular release cycle (including
documentation building) but also they do not contribute dependencies to the CI image and their tests are
not run in CI process. You can build and prepare such provider when you explicitly specify it as argument
of a release command or by passing `--include-suspended-providers` flag in corresponding command (but it
might or might not work at any time as the provider release commands are not regularly run on CI for the
suspended providers). The `suspended` providers are not released as part of the regular release cycle.
-* The `removed` state is when the provider is marked as `removed` - usually after some period of time being
- `suspended`. This is a temporary state after the provider has been voted (or agreed in "lazy consensus") to
- be removed and it is only used for exactly one release cycle - in order to produce the final version of
+* The `removed` state is a temporary state after the provider has been voted (or agreed in "lazy consensus")
+ to be removed and it is only used for exactly one release cycle - in order to produce the final version of
the package - identical to the previous version with the exception of the removal notice. The process
of removal is described in [Provider's docs](../PROVIDERS.rst). The `removed` providers are included in
the regular release cycle (including documentation building) because the `--include-removed-providers`
@@ -222,6 +220,9 @@ In case you prepare provider documentation for just a few selected providers, yo
breeze release-management prepare-provider-documentation [packages]
```
+In case you want to also release a pre-installed provider that is in ``not-ready`` state (i.e. when
+you want to release it before you switch their state to ``ready``), you need to pass
+``--include-not-ready-providers`` flag to the command above.
This command will not only prepare documentation but will also help the release manager to review
changes implemented in all providers, and determine which of the providers should be released. For each
@@ -251,6 +252,10 @@ In case you prepare provider documentation for just a few selected providers, yo
breeze release-management prepare-provider-documentation --answer yes [packages]
```
+In case you want to also release a pre-installed provider that is in ``not-ready`` state (i.e. when
+you want to release it before you switch their state to ``ready``), you need to pass
+``--include-not-ready-providers`` flag to the command above.
+
NOTE!! In case you prepare provider's documentation in a branch different than main, you need to manually
specify the base branch via `--base-branch` parameter.
For example if you try to build a `cncf.kubernetes` provider that is build from `provider-cncf-kubernetes/v4-4`
@@ -319,6 +324,11 @@ breeze release-management prepare-provider-packages --include-removed-providers
--package-format both PACKAGE PACKAGE ....
```
+In case you want to also release a pre-installed provider that is in ``not-ready`` state (i.e. when
+you want to release it before you switch their state to ``ready``), you need to pass
+``--include-not-ready-providers`` flag to the command above.
+
+
* Sign all your packages
```shell script
diff --git a/dev/airflow_pre_installed_providers.txt b/dev/airflow_pre_installed_providers.txt
new file mode 100644
index 0000000000..878c309d76
--- /dev/null
+++ b/dev/airflow_pre_installed_providers.txt
@@ -0,0 +1,7 @@
+# List of all the providers that are pre-installed when you run `pip install apache-airflow` without extras
+common.io
+common.sql
+ftp
+http
+imap
+sqlite
diff --git a/dev/breeze/README.md b/dev/breeze/README.md
index 7452be51ae..e64de13a56 100644
--- a/dev/breeze/README.md
+++ b/dev/breeze/README.md
@@ -66,6 +66,6 @@ PLEASE DO NOT MODIFY THE HASH BELOW! IT IS AUTOMATICALLY UPDATED BY PRE-COMMIT.
---------------------------------------------------------------------------------------------------------
-Package config hash: ac1e65234e1a780d1f21ac28e27451c763e308cc901ab8f72acc6414db8da14c8f841e4757b134a4cc6f3c76186af340db0610cd3fa155ed5fcf1abca45381e8
+Package config hash: 5253de328868bc7e725254e1efe2daa57e7b1faee1b0be68aa8609569e47d08a1e0be4760bf889f422c4ce5c2884577c1300cd2f2292f2cb4677b1d1d160916c
---------------------------------------------------------------------------------------------------------
diff --git a/dev/breeze/SELECTIVE_CHECKS.md b/dev/breeze/SELECTIVE_CHECKS.md
index 2f7d962041..627fa5a365 100644
--- a/dev/breeze/SELECTIVE_CHECKS.md
+++ b/dev/breeze/SELECTIVE_CHECKS.md
@@ -44,7 +44,10 @@ We have the following Groups of files for CI that determine which tests are run:
Open API specification and determine that we should run dedicated API tests.
* `Helm files` - change in those files impacts helm "rendering" tests - `chart` folder and `helm_tests` folder.
* `Setup files` - change in the setup files indicates that we should run `upgrade to newer dependencies` -
- setup.* files, pyproject.toml, generated dependencies files in `generated` folder
+ pyproject.toml and generated dependencies files in `generated` folder. The dependency files and part of
+ the pyproject.toml are automatically generated from the provider.yaml files in provider by
+ the `update-providers-dependencies` pre-commit. The provider.yaml is a single source of truth for each
+ provider.
* `DOC files` - change in those files indicate that we should run documentation builds (both airflow sources
and airflow documentation)
* `WWW files` - those are files for the WWW part of our UI (useful to determine if UI tests should run)
@@ -74,9 +77,10 @@ together using `pytest-xdist` (pytest-xdist distributes the tests among parallel
## Selective check decision rules
* `Full tests` case is enabled when the event is PUSH, or SCHEDULE or we miss commit info or any of the
- important environment files (setup.py, setup.cfg, provider.yaml, Dockerfile, build scripts) changed or
- when `full tests needed` label is set. That enables all matrix combinations of variables (representative)
- and all possible test type. No further checks are performed.
+ important environment files (`pyproject.toml`, `Dockerfile`, `scripts`,
+ `generated/provider_dependencies.json` etc.) changed or when `full tests needed` label is set.
+ That enables all matrix combinations of variables (representative) and all possible test type. No further
+ checks are performed. See also [1] note below.
* Python, Kubernetes, Backend, Kind, Helm versions are limited to "defaults" only unless `Full tests` mode
is enabled.
* `Python scans`, `Javascript scans`, `API tests/codegen`, `UI`, `WWW`, `Kubernetes` tests and `DOC builds`
@@ -199,3 +203,20 @@ Github Actions to pass the list of parameters to a command to execute
| skip-provider-tests | When provider tests should be skipped (on non-main branch or when no provider changes detected) | true | |
| sqlite-exclude | Which versions of Sqlite to exclude for tests as JSON array | [] | |
| upgrade-to-newer-dependencies | Whether the image build should attempt to upgrade all dependencies (might be true/false or commit hash) | false | |
+
+
+[1] Note for deciding if `full tests needed` mode is enabled and provider.yaml files.
+
+When we decided whether to run `full tests` we do not check (directly) if provider.yaml files changed,
+even if they are single source of truth for provider dependencies and when you add a dependency there,
+the environment changes and generally full tests are advised.
+
+This is because provider.yaml change will automatically trigger (via `update-provider-dependencies` pre-commit)
+generation of `generated/provider_dependencies.json` and `pyproject.toml` gets updated as well. This is a far
+better indication if we need to run full tests than just checking if provider.yaml files changed, because
+provider.yaml files contain more information than just dependencies - they are the single source of truth
+for a lot of information for each provider and sometimes (for example when we update provider documentation
+or when new Hook class is added), we do not need to run full tests.
+
+That's why we do not base our `full tests needed` decision on changes in dependency files that are generated
+from the `provider.yaml` files.
diff --git a/dev/breeze/doc/adr/0003-bootstrapping-virtual-environment.md b/dev/breeze/doc/adr/0003-bootstrapping-virtual-environment.md
index 5cf11fe327..a2788b57c4 100644
--- a/dev/breeze/doc/adr/0003-bootstrapping-virtual-environment.md
+++ b/dev/breeze/doc/adr/0003-bootstrapping-virtual-environment.md
@@ -101,7 +101,7 @@ version of Breeze will remain as part of the Airflow's source code.
The decision is to implement Breeze in a subfolder (`dev/breeze2/`) of
Apache Airflow as a Python project following the standard setuptools
enabled project. The project contains setup.py and dependencies described
-in setup.cfg and contains both source code and tests for Breeze code.
+in pyproject.toml and contains both source code and tests for Breeze code.
The sub-project could be used in the future to produce a PyPI package
(we reserved such package in PyPI), however its main purpose is
@@ -116,7 +116,7 @@ of Airflow) performs the following tasks:
* when run for the first time it creates `.build/breeze2/venv` virtual
environment (Python3.6+ based) - with locally installed `dev`
- project in editable mode (`pip install -e .`) - this makes sure
+ project in editable mode (`pip install -e ".[devel]"`) - this makes sure
that the users of Breeze will use the latest version of Breeze
available in their version of the repository
* when run subsequently, it will check if setup files changed for
diff --git a/dev/breeze/pyproject.toml b/dev/breeze/pyproject.toml
index 9cae9052e2..438719faa7 100644
--- a/dev/breeze/pyproject.toml
+++ b/dev/breeze/pyproject.toml
@@ -50,6 +50,7 @@ dependencies = [
"filelock>=3.13.0",
"flit>=3.5.0",
"gitpython>=3.1.40",
+ "hatch==1.9.1",
"inputimeout>=1.0.4",
"jinja2>=3.1.0",
"jsonschema>=4.19.1",
@@ -61,7 +62,7 @@ dependencies = [
"pytest-xdist>=3.3.1",
"pytest>=7.4.0",
"pyyaml>=6.0.1",
- "requests>=2.30.0",
+ "requests>=2.31.0",
"rich-click>=1.7.1",
"rich>=13.6.0",
"semver>=3.0.2",
diff --git a/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py b/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
index 981124540f..0682728ff4 100644
--- a/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
@@ -46,7 +46,6 @@ from airflow_breeze.commands.common_image_options import (
option_image_tag_for_pulling,
option_image_tag_for_verifying,
option_install_mysql_client_type,
- option_install_providers_from_sources,
option_platform_multiple,
option_prepare_buildx_cache,
option_pull,
@@ -304,7 +303,6 @@ option_version_suffix_for_pypi_ci = click.option(
@option_install_mysql_client_type
@option_image_tag_for_building
@option_include_success_outputs
-@option_install_providers_from_sources
@option_parallelism
@option_platform_multiple
@option_prepare_buildx_cache
@@ -345,7 +343,6 @@ def build(
image_tag: str,
include_success_outputs,
install_mysql_client_type: str,
- install_providers_from_sources: bool,
parallelism: int,
platform: str | None,
prepare_buildx_cache: bool,
@@ -417,7 +414,6 @@ def build(
github_token=github_token,
image_tag=image_tag,
install_mysql_client_type=install_mysql_client_type,
- install_providers_from_sources=install_providers_from_sources,
prepare_buildx_cache=prepare_buildx_cache,
push=push,
python=python,
diff --git a/dev/breeze/src/airflow_breeze/commands/ci_image_commands_config.py b/dev/breeze/src/airflow_breeze/commands/ci_image_commands_config.py
index d790ae33a1..97f5f23538 100644
--- a/dev/breeze/src/airflow_breeze/commands/ci_image_commands_config.py
+++ b/dev/breeze/src/airflow_breeze/commands/ci_image_commands_config.py
@@ -58,7 +58,6 @@ CI_IMAGE_TOOLS_PARAMETERS: dict[str, list[dict[str, str | list[str]]]] = {
"--commit-sha",
"--debian-version",
"--install-mysql-client-type",
- "--install-providers-from-sources",
"--python-image",
],
},
diff --git a/dev/breeze/src/airflow_breeze/commands/common_image_options.py b/dev/breeze/src/airflow_breeze/commands/common_image_options.py
index dd94ffd499..1b8f946094 100644
--- a/dev/breeze/src/airflow_breeze/commands/common_image_options.py
+++ b/dev/breeze/src/airflow_breeze/commands/common_image_options.py
@@ -146,12 +146,6 @@ option_install_mysql_client_type = click.option(
default=ALLOWED_INSTALL_MYSQL_CLIENT_TYPES[0],
envvar="INSTALL_MYSQL_CLIENT_TYPE",
)
-option_install_providers_from_sources = click.option(
- "--install-providers-from-sources",
- help="Install providers from sources when installing.",
- is_flag=True,
- envvar="INSTALL_PROVIDERS_FROM_SOURCES",
-)
option_platform_multiple = click.option(
"--platform",
help="Platform for Airflow image.",
diff --git a/dev/breeze/src/airflow_breeze/commands/developer_commands.py b/dev/breeze/src/airflow_breeze/commands/developer_commands.py
index fcc9bd57ce..ede6dfd933 100644
--- a/dev/breeze/src/airflow_breeze/commands/developer_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/developer_commands.py
@@ -423,7 +423,7 @@ option_executor_start_airflow = click.option(
@main.command(name="start-airflow")
@click.option(
- "--skip-asset-compilation",
+ "--skip-assets-compilation",
help="Skips compilation of assets when starting airflow even if the content of www changed "
"(mutually exclusive with --dev-mode).",
is_flag=True,
@@ -431,7 +431,7 @@ option_executor_start_airflow = click.option(
@click.option(
"--dev-mode",
help="Starts webserver in dev mode (assets are always recompiled in this case when starting) "
- "(mutually exclusive with --skip-asset-compilation).",
+ "(mutually exclusive with --skip-assets-compilation).",
is_flag=True,
)
@click.argument("extra-args", nargs=-1, type=click.UNPROCESSED)
@@ -512,7 +512,7 @@ def start_airflow(
providers_skip_constraints: bool,
python: str,
restart: bool,
- skip_asset_compilation: bool,
+ skip_assets_compilation: bool,
standalone_dag_processor: bool,
use_airflow_version: str | None,
use_packages_from_dist: bool,
@@ -521,12 +521,12 @@ def start_airflow(
Enter breeze environment and starts all Airflow components in the tmux session.
Compile assets if contents of www directory changed.
"""
- if dev_mode and skip_asset_compilation:
+ if dev_mode and skip_assets_compilation:
get_console().print(
"[warning]You cannot skip asset compilation in dev mode! Assets will be compiled!"
)
- skip_asset_compilation = True
- if use_airflow_version is None and not skip_asset_compilation:
+ skip_assets_compilation = True
+ if use_airflow_version is None and not skip_assets_compilation:
run_compile_www_assets(dev=dev_mode, run_in_background=True, force_clean=False)
airflow_constraints_reference = _determine_constraint_branch_used(
airflow_constraints_reference, use_airflow_version
diff --git a/dev/breeze/src/airflow_breeze/commands/developer_commands_config.py b/dev/breeze/src/airflow_breeze/commands/developer_commands_config.py
index 7fba31b8b0..88b734f513 100644
--- a/dev/breeze/src/airflow_breeze/commands/developer_commands_config.py
+++ b/dev/breeze/src/airflow_breeze/commands/developer_commands_config.py
@@ -228,7 +228,7 @@ DEVELOPER_PARAMETERS: dict[str, list[dict[str, str | list[str]]]] = {
{
"name": "Asset compilation options",
"options": [
- "--skip-asset-compilation",
+ "--skip-assets-compilation",
"--dev-mode",
],
},
diff --git a/dev/breeze/src/airflow_breeze/commands/production_image_commands.py b/dev/breeze/src/airflow_breeze/commands/production_image_commands.py
index 9c2cb3a57c..1266085c79 100644
--- a/dev/breeze/src/airflow_breeze/commands/production_image_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/production_image_commands.py
@@ -41,7 +41,6 @@ from airflow_breeze.commands.common_image_options import (
option_image_tag_for_pulling,
option_image_tag_for_verifying,
option_install_mysql_client_type,
- option_install_providers_from_sources,
option_platform_multiple,
option_prepare_buildx_cache,
option_pull,
@@ -233,7 +232,6 @@ def prod_image():
@option_image_tag_for_building
@option_include_success_outputs
@option_install_mysql_client_type
-@option_install_providers_from_sources
@option_parallelism
@option_platform_multiple
@option_prepare_buildx_cache
@@ -284,7 +282,6 @@ def build(
install_airflow_version: str | None,
install_mysql_client_type: str,
install_packages_from_context: bool,
- install_providers_from_sources: bool,
installation_method: str,
parallelism: int,
platform: str | None,
@@ -348,7 +345,6 @@ def build(
install_airflow_version=install_airflow_version,
install_mysql_client_type=install_mysql_client_type,
install_packages_from_context=install_packages_from_context,
- install_providers_from_sources=install_providers_from_sources,
installation_method=installation_method,
prepare_buildx_cache=prepare_buildx_cache,
push=push,
diff --git a/dev/breeze/src/airflow_breeze/commands/production_image_commands_config.py b/dev/breeze/src/airflow_breeze/commands/production_image_commands_config.py
index 69c7d23c96..78b4b9e00d 100644
--- a/dev/breeze/src/airflow_breeze/commands/production_image_commands_config.py
+++ b/dev/breeze/src/airflow_breeze/commands/production_image_commands_config.py
@@ -57,7 +57,6 @@ PRODUCTION_IMAGE_TOOLS_PARAMETERS: dict[str, list[dict[str, str | list[str]]]] =
"--python-image",
"--commit-sha",
"--additional-pip-install-flags",
- "--install-providers-from-sources",
],
},
{
diff --git a/dev/breeze/src/airflow_breeze/commands/release_candidate_command.py b/dev/breeze/src/airflow_breeze/commands/release_candidate_command.py
index f88388a22a..09cc67c226 100644
--- a/dev/breeze/src/airflow_breeze/commands/release_candidate_command.py
+++ b/dev/breeze/src/airflow_breeze/commands/release_candidate_command.py
@@ -90,7 +90,7 @@ def tarball_release(version, version_without_rc):
def create_artifacts_with_sdist():
- run_command(["python3", "setup.py", "compile_assets", "sdist", "bdist_wheel"], check=True)
+ run_command(["hatch", "build", "-t", "sdist", "-t", "wheel"], check=True)
console_print("Artifacts created")
@@ -100,7 +100,6 @@ def create_artifacts_with_breeze():
"breeze",
"release-management",
"prepare-airflow-package",
- "--use-container-for-assets-compilation",
"--package-format",
"both",
],
@@ -356,7 +355,7 @@ def publish_release_candidate(version, previous_version, github_token):
# Create the artifacts
if confirm_action("Use breeze to create artifacts?"):
create_artifacts_with_breeze()
- elif confirm_action("Use setup.py to create artifacts?"):
+ elif confirm_action("Use hatch to create artifacts?"):
create_artifacts_with_sdist()
# Sign the release
sign_the_release(airflow_repo_root)
diff --git a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
index f943025205..39211c9658 100644
--- a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
@@ -19,6 +19,7 @@ from __future__ import annotations
import glob
import operator
import os
+import random
import re
import shutil
import sys
@@ -129,7 +130,6 @@ from airflow_breeze.utils.parallel import (
)
from airflow_breeze.utils.path_utils import (
AIRFLOW_SOURCES_ROOT,
- AIRFLOW_WWW_DIR,
CONSTRAINTS_CACHE_DIR,
DIST_DIR,
GENERATED_PROVIDER_PACKAGES_DIR,
@@ -143,9 +143,7 @@ from airflow_breeze.utils.provider_dependencies import (
)
from airflow_breeze.utils.python_versions import get_python_version_list
from airflow_breeze.utils.run_utils import (
- clean_www_assets,
run_command,
- run_compile_www_assets,
)
from airflow_breeze.utils.shared_options import get_dry_run, get_verbose
from airflow_breeze.utils.versions import is_pre_release
@@ -209,50 +207,74 @@ AIRFLOW_PIP_VERSION = "23.3.2"
WHEEL_VERSION = "0.36.2"
GITPYTHON_VERSION = "3.1.40"
RICH_VERSION = "13.7.0"
+NODE_VERSION = "21.2.0"
+PRE_COMMIT_VERSION = "3.5.0"
AIRFLOW_BUILD_DOCKERFILE = f"""
FROM python:{DEFAULT_PYTHON_MAJOR_MINOR_VERSION}-slim-{ALLOWED_DEBIAN_VERSIONS[0]}
RUN apt-get update && apt-get install -y --no-install-recommends git
-RUN pip install pip=={AIRFLOW_PIP_VERSION} wheel=={WHEEL_VERSION} \\
- gitpython=={GITPYTHON_VERSION} rich=={RICH_VERSION}
+RUN pip install pip=={AIRFLOW_PIP_VERSION} hatch==1.9.1 \
+ gitpython=={GITPYTHON_VERSION} rich=={RICH_VERSION} pre-commit=={PRE_COMMIT_VERSION}
+COPY . /opt/airflow
"""
-AIRFLOW_BUILD_IMAGE_TAG = "apache/airflow:local-build-image"
-NODE_BUILD_IMAGE_TAG = "node:21.2.0-bookworm-slim"
-
+AIRFLOW_BUILD_DOCKERIGNORE = """
+# Git version is dynamically generated
+airflow/git_version
+# Exclude mode_modules pulled by "yarn" for compilation of www files generated by NPM
+airflow/www/node_modules
+
+# Exclude link to docs
+airflow/www/static/docs
+
+# Exclude python generated files
+**/__pycache__/
+**/*.py[cod]
+**/*$py.class
+**/.pytest_cache/
+**/env/
+**/build/
+**/develop-eggs/
+/dist/
+**/downloads/
+**/eggs/
+**/.eggs/
+**/lib/
+**/lib64/
+**/parts/
+**/sdist/
+**/var/
+**/wheels/
+**/*.egg-info/
+**/.installed.cfg
+**/*.egg
+
+# Exclude temporary vi files
+**/*~
+
+# Exclude output files
+**/*.out
+**/hive_scratch_dir/
+
+# Exclude auto-generated Finder files on Mac OS
+**/.DS_Store
+**/Thumbs.db
+
+# Exclude docs generated files
+docs/_build/
+docs/_api/
+docs/_doctrees/
+
+# files generated by memray
+*.py.*.html
+*.py.*.bin
+"""
-def _compile_assets_in_docker():
- clean_www_assets()
- get_console().print("[info]Compiling assets in docker container\n")
- result = run_command(
- [
- "docker",
- "run",
- "-t",
- "-v",
- f"{AIRFLOW_WWW_DIR}:/opt/airflow/airflow/www/",
- "-e",
- "FORCE_COLOR=true",
- NODE_BUILD_IMAGE_TAG,
- "bash",
- "-c",
- "cd /opt/airflow/airflow/www && yarn install --frozen-lockfile && yarn run build",
- ],
- text=True,
- capture_output=not get_verbose(),
- check=False,
- )
- if result.returncode != 0:
- get_console().print("[error]Error compiling assets[/]")
- get_console().print(result.stdout)
- get_console().print(result.stderr)
- fix_ownership_using_docker()
- sys.exit(result.returncode)
+AIRFLOW_BUILD_IMAGE_TAG = "apache/airflow:local-build-image"
+NODE_BUILD_IMAGE_TAG = f"node:{NODE_VERSION}-bookworm-slim"
- get_console().print("[success]compiled assets in docker container\n")
- get_console().print("[info]Fixing ownership of compiled assets\n")
- fix_ownership_using_docker()
- get_console().print("[success]Fixing ownership of compiled assets\n")
+AIRFLOW_BUILD_DOCKERFILE_PATH = AIRFLOW_SOURCES_ROOT / "airflow-build-dockerfile"
+AIRFLOW_BUILD_DOCKERFILE_IGNORE_PATH = AIRFLOW_SOURCES_ROOT / "airflow-build-dockerfile.dockerignore"
@release_management.command(
@@ -260,63 +282,74 @@ def _compile_assets_in_docker():
help="Prepare sdist/whl package of Airflow.",
)
@option_package_format
-@click.option(
- "--use-container-for-assets-compilation",
- is_flag=True,
- help="If set, the assets are compiled in docker container. On MacOS, asset compilation in containers "
- "is slower, due to slow mounted filesystem and number of node_module files so by default asset "
- "compilation is done locally. This option is useful for officially building packages by release "
- "manager on MacOS to make sure it is a reproducible build.",
-)
@option_version_suffix_for_pypi
@option_verbose
@option_dry_run
def prepare_airflow_packages(
package_format: str,
version_suffix_for_pypi: str,
- use_container_for_assets_compilation: bool,
):
perform_environment_checks()
fix_ownership_using_docker()
cleanup_python_generated_files()
- get_console().print("[info]Compiling assets\n")
- from sys import platform
-
- if platform == "darwin" and not use_container_for_assets_compilation:
- run_compile_www_assets(dev=False, run_in_background=False, force_clean=True)
- else:
- _compile_assets_in_docker()
- get_console().print("[success]Assets compiled successfully[/]")
+ # This is security feature.
+ #
+ # Building the image needed to build airflow package including .git directory
+ # In isolated environment, to not allow the in-docker code to override local code
+ # The image used to build airflow package is built from scratch and contains
+ # Full Airflow code including Airflow repository is added to the image, but locally build node_modules
+ # are not added to the context of that image
+ AIRFLOW_BUILD_DOCKERFILE_PATH.write_text(AIRFLOW_BUILD_DOCKERFILE.strip())
+ AIRFLOW_BUILD_DOCKERFILE_IGNORE_PATH.write_text(AIRFLOW_BUILD_DOCKERIGNORE.strip())
run_command(
- ["docker", "build", "--tag", AIRFLOW_BUILD_IMAGE_TAG, "-"],
- input=AIRFLOW_BUILD_DOCKERFILE,
+ [
+ "docker",
+ "buildx",
+ "build",
+ ".",
+ "-f",
+ "airflow-build-dockerfile",
+ "--tag",
+ AIRFLOW_BUILD_IMAGE_TAG,
+ ],
text=True,
check=True,
+ cwd=AIRFLOW_SOURCES_ROOT,
env={"DOCKER_CLI_HINTS": "false"},
)
- run_command(
+ container_id = f"airflow-build-{random.getrandbits(64):08x}"
+ result = run_command(
cmd=[
"docker",
"run",
+ "--name",
+ container_id,
"-t",
- "-v",
- f"{AIRFLOW_SOURCES_ROOT}:/opt/airflow:cached",
"-e",
f"VERSION_SUFFIX_FOR_PYPI={version_suffix_for_pypi}",
"-e",
+ "HOME=/opt/airflow/files/home",
+ "-e",
"GITHUB_ACTIONS",
"-e",
f"PACKAGE_FORMAT={package_format}",
+ "-w",
+ "/opt/airflow",
AIRFLOW_BUILD_IMAGE_TAG,
"python",
"/opt/airflow/scripts/in_container/run_prepare_airflow_packages.py",
],
- check=True,
+ check=False,
)
+ if result.returncode != 0:
+ get_console().print("[error]Error preparing Airflow package[/]")
+ fix_ownership_using_docker()
+ sys.exit(result.returncode)
+ DIST_DIR.mkdir(parents=True, exist_ok=True)
+ # Copy all files in the dist directory in container to the host dist directory (note '/.' in SRC)
+ run_command(["docker", "cp", f"{container_id}:/opt/airflow/dist/.", "./dist"], check=True)
+ run_command(["docker", "rm", "--force", container_id], check=True)
get_console().print("[success]Successfully prepared Airflow package!\n\n")
- get_console().print("\n[info]Cleaning ownership of generated files\n")
- fix_ownership_using_docker()
- get_console().print("\n[success]Cleaned ownership of generated files\n")
def provider_action_summary(description: str, message_type: MessageType, packages: list[str]):
@@ -699,7 +732,7 @@ def run_generate_constraints_in_parallel(
@release_management.command(
name="generate-constraints",
- help="Generates pinned constraint files with all extras from setup.py in parallel.",
+ help="Generates pinned constraint files with all extras from pyproject.toml in parallel.",
)
@option_python
@option_run_in_parallel
diff --git a/dev/breeze/src/airflow_breeze/commands/release_management_commands_config.py b/dev/breeze/src/airflow_breeze/commands/release_management_commands_config.py
index 77fc83650c..e6ddec7e16 100644
--- a/dev/breeze/src/airflow_breeze/commands/release_management_commands_config.py
+++ b/dev/breeze/src/airflow_breeze/commands/release_management_commands_config.py
@@ -55,7 +55,6 @@ RELEASE_MANAGEMENT_PARAMETERS: dict[str, list[dict[str, str | list[str]]]] = {
"name": "Package flags",
"options": [
"--package-format",
- "--use-container-for-assets-compilation",
"--version-suffix-for-pypi",
],
}
diff --git a/dev/breeze/src/airflow_breeze/global_constants.py b/dev/breeze/src/airflow_breeze/global_constants.py
index 15e6bbec94..ded03520b8 100644
--- a/dev/breeze/src/airflow_breeze/global_constants.py
+++ b/dev/breeze/src/airflow_breeze/global_constants.py
@@ -387,8 +387,7 @@ with Path(AIRFLOW_SOURCES_ROOT, "generated", "provider_dependencies.json").open(
# Initialize files for rebuild check
FILES_FOR_REBUILD_CHECK = [
- "setup.py",
- "setup.cfg",
+ "pyproject.toml",
"Dockerfile.ci",
".dockerignore",
"generated/provider_dependencies.json",
@@ -430,19 +429,19 @@ DEFAULT_EXTRAS = [
"amazon",
"async",
"celery",
- "cncf.kubernetes",
- "common.io",
+ "cncf-kubernetes",
+ "common-io",
"docker",
"elasticsearch",
"ftp",
"google",
- "google_auth",
+ "google-auth",
"graphviz",
"grpc",
"hashicorp",
"http",
"ldap",
- "microsoft.azure",
+ "microsoft-azure",
"mysql",
"odbc",
"openlineage",
diff --git a/dev/breeze/src/airflow_breeze/params/build_ci_params.py b/dev/breeze/src/airflow_breeze/params/build_ci_params.py
index 954f405f9c..2883eded8c 100644
--- a/dev/breeze/src/airflow_breeze/params/build_ci_params.py
+++ b/dev/breeze/src/airflow_breeze/params/build_ci_params.py
@@ -33,7 +33,7 @@ class BuildCiParams(CommonBuildParams):
airflow_constraints_mode: str = "constraints-source-providers"
airflow_constraints_reference: str = DEFAULT_AIRFLOW_CONSTRAINTS_BRANCH
- airflow_extras: str = "devel_ci"
+ airflow_extras: str = "devel-ci"
airflow_pre_cached_pip_packages: bool = True
force_build: bool = False
upgrade_to_newer_dependencies: bool = False
diff --git a/dev/breeze/src/airflow_breeze/params/build_prod_params.py b/dev/breeze/src/airflow_breeze/params/build_prod_params.py
index ed0cec9a8c..334edaab48 100644
--- a/dev/breeze/src/airflow_breeze/params/build_prod_params.py
+++ b/dev/breeze/src/airflow_breeze/params/build_prod_params.py
@@ -212,7 +212,6 @@ class BuildProdParams(CommonBuildParams):
self._req_arg("DOCKER_CONTEXT_FILES", self.docker_context_files)
self._req_arg("INSTALL_PACKAGES_FROM_CONTEXT", self.install_packages_from_context)
self._req_arg("INSTALL_POSTGRES_CLIENT", self.install_postgres_client)
- self._req_arg("INSTALL_PROVIDERS_FROM_SOURCES", self.install_providers_from_sources)
self._req_arg("PYTHON_BASE_IMAGE", self.python_base_image)
# optional build args
self._opt_arg("AIRFLOW_CONSTRAINTS_LOCATION", self.airflow_constraints_location)
diff --git a/dev/breeze/src/airflow_breeze/params/common_build_params.py b/dev/breeze/src/airflow_breeze/params/common_build_params.py
index e1c77990e9..421bcc7c3a 100644
--- a/dev/breeze/src/airflow_breeze/params/common_build_params.py
+++ b/dev/breeze/src/airflow_breeze/params/common_build_params.py
@@ -63,7 +63,6 @@ class CommonBuildParams:
github_repository: str = APACHE_AIRFLOW_GITHUB_REPOSITORY
github_token: str = os.environ.get("GITHUB_TOKEN", "")
image_tag: str | None = None
- install_providers_from_sources: bool = False
install_mysql_client_type: str = ALLOWED_INSTALL_MYSQL_CLIENT_TYPES[0]
platform: str = DOCKER_DEFAULT_PLATFORM
prepare_buildx_cache: bool = False
diff --git a/dev/breeze/src/airflow_breeze/params/shell_params.py b/dev/breeze/src/airflow_breeze/params/shell_params.py
index e90d348046..85fbecd6ca 100644
--- a/dev/breeze/src/airflow_breeze/params/shell_params.py
+++ b/dev/breeze/src/airflow_breeze/params/shell_params.py
@@ -162,7 +162,6 @@ class ShellParams:
image_tag: str | None = None
include_mypy_volume: bool = False
install_airflow_version: str = ""
- install_providers_from_sources: bool = True
install_selected_providers: str | None = None
integration: tuple[str, ...] = ()
issue_id: str = ""
@@ -526,7 +525,6 @@ class ShellParams:
_set_var(_env, "HOST_USER_ID", self.host_user_id)
_set_var(_env, "INIT_SCRIPT_FILE", None, "init.sh")
_set_var(_env, "INSTALL_AIRFLOW_VERSION", self.install_airflow_version)
- _set_var(_env, "INSTALL_PROVIDERS_FROM_SOURCES", self.install_providers_from_sources)
_set_var(_env, "INSTALL_SELECTED_PROVIDERS", self.install_selected_providers)
_set_var(_env, "ISSUE_ID", self.issue_id)
_set_var(_env, "LOAD_DEFAULT_CONNECTIONS", self.load_default_connections)
diff --git a/dev/breeze/src/airflow_breeze/pre_commit_ids.py b/dev/breeze/src/airflow_breeze/pre_commit_ids.py
index 1b35b2c5bc..e5ad08afa1 100644
--- a/dev/breeze/src/airflow_breeze/pre_commit_ids.py
+++ b/dev/breeze/src/airflow_breeze/pre_commit_ids.py
@@ -71,9 +71,9 @@ PRE_COMMIT_LIST = [
"check-providers-init-file-missing",
"check-providers-subpackages-init-file-exist",
"check-pydevd-left-in-code",
+ "check-pyproject-toml-order",
"check-revision-heads-map",
"check-safe-filter-usage-in-html",
- "check-setup-order",
"check-start-date-not-used-in-defaults",
"check-system-tests-present",
"check-system-tests-tocs",
@@ -133,6 +133,5 @@ PRE_COMMIT_LIST = [
"update-supported-versions",
"update-vendored-in-k8s-json-schema",
"update-version",
- "validate-pyproject",
"yamllint",
]
diff --git a/dev/breeze/src/airflow_breeze/prepare_providers/provider_documentation.py b/dev/breeze/src/airflow_breeze/prepare_providers/provider_documentation.py
index 391edb6056..00bb2203a0 100644
--- a/dev/breeze/src/airflow_breeze/prepare_providers/provider_documentation.py
+++ b/dev/breeze/src/airflow_breeze/prepare_providers/provider_documentation.py
@@ -656,7 +656,7 @@ def update_release_notes(
) -> tuple[bool, bool]:
"""Updates generated files.
- This includes the readme, changes, and/or setup.cfg/setup.py/manifest.in/provider_info.
+ This includes the readme, changes, and provider.yaml files.
:param provider_package_id: id of the package
:param reapply_templates_only: regenerate already released documentation only - without updating versions
diff --git a/dev/breeze/src/airflow_breeze/prepare_providers/provider_packages.py b/dev/breeze/src/airflow_breeze/prepare_providers/provider_packages.py
index a158334f7d..fa3a1b6845 100644
--- a/dev/breeze/src/airflow_breeze/prepare_providers/provider_packages.py
+++ b/dev/breeze/src/airflow_breeze/prepare_providers/provider_packages.py
@@ -260,7 +260,8 @@ def get_packages_list_to_act_on(
return [
package.strip()
for package in package_list_file.readlines()
- if (package.strip() not in removed_provider_ids or include_removed)
+ if not package.strip().startswith("#")
+ and (package.strip() not in removed_provider_ids or include_removed)
and (package.strip() not in not_ready_provider_ids or include_not_ready)
]
elif provider_packages:
diff --git a/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py b/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
index 5c8e98056d..4e35cabe3b 100644
--- a/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
+++ b/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
@@ -65,7 +65,7 @@ if TYPE_CHECKING:
# Those are volumes that are mounted when MOUNT_SELECTED is chosen (which is the default when
# entering Breeze. MOUNT_SELECTED prevents to mount the files that you can have accidentally added
-# in your sources (or they were added automatically by setup.py etc.) to be mounted to container.
+# in your sources (or they were added automatically by pyproject.toml) to be mounted to container.
# This is important to get a "clean" environment for different python versions and to avoid
# unnecessary slow-downs when you are mounting files on MacOS (which has very slow filesystem)
# Any time you add a top-level folder in airflow that should also be added to container you should
@@ -80,7 +80,6 @@ VOLUMES_FOR_SELECTED_MOUNTS = [
(".rat-excludes", "/opt/airflow/.rat-excludes"),
("BREEZE.rst", "/opt/airflow/BREEZE.rst"),
("LICENSE", "/opt/airflow/LICENSE"),
- ("MANIFEST.in", "/opt/airflow/MANIFEST.in"),
("NOTICE", "/opt/airflow/NOTICE"),
("RELEASE_NOTES.rst", "/opt/airflow/RELEASE_NOTES.rst"),
("airflow", "/opt/airflow/airflow"),
@@ -95,8 +94,6 @@ VOLUMES_FOR_SELECTED_MOUNTS = [
("pyproject.toml", "/opt/airflow/pyproject.toml"),
("scripts", "/opt/airflow/scripts"),
("scripts/docker/entrypoint_ci.sh", "/entrypoint"),
- ("setup.cfg", "/opt/airflow/setup.cfg"),
- ("setup.py", "/opt/airflow/setup.py"),
("tests", "/opt/airflow/tests"),
("helm_tests", "/opt/airflow/helm_tests"),
("kubernetes_tests", "/opt/airflow/kubernetes_tests"),
diff --git a/dev/breeze/src/airflow_breeze/utils/kubernetes_utils.py b/dev/breeze/src/airflow_breeze/utils/kubernetes_utils.py
index 8a8db025d5..dcbb585db1 100644
--- a/dev/breeze/src/airflow_breeze/utils/kubernetes_utils.py
+++ b/dev/breeze/src/airflow_breeze/utils/kubernetes_utils.py
@@ -299,7 +299,6 @@ def _install_packages_in_k8s_virtualenv():
str(K8S_REQUIREMENTS.resolve()),
]
env = os.environ.copy()
- env["INSTALL_PROVIDERS_FROM_SOURCES"] = "true"
capture_output = True
if get_verbose():
capture_output = False
diff --git a/dev/breeze/src/airflow_breeze/utils/packages.py b/dev/breeze/src/airflow_breeze/utils/packages.py
index 49626a4668..a6eb386148 100644
--- a/dev/breeze/src/airflow_breeze/utils/packages.py
+++ b/dev/breeze/src/airflow_breeze/utils/packages.py
@@ -53,16 +53,6 @@ HTTPS_REMOTE = "apache-https-for-providers"
LONG_PROVIDERS_PREFIX = "apache-airflow-providers-"
-# TODO: use single source of truth for those
-# for now we need to keep them in sync with the ones in setup.py
-PREINSTALLED_PROVIDERS = [
- "common.sql",
- "ftp",
- "http",
- "imap",
- "sqlite",
-]
-
class EntityType(Enum):
Operators = "Operators"
@@ -259,7 +249,7 @@ def get_available_packages(
if include_not_ready:
valid_states.add("not-ready")
if include_regular:
- valid_states.add("ready")
+ valid_states.update({"ready", "pre-release"})
if include_suspended:
valid_states.add("suspended")
if include_removed:
diff --git a/dev/breeze/src/airflow_breeze/utils/selective_checks.py b/dev/breeze/src/airflow_breeze/utils/selective_checks.py
index 4db2972356..3d23647392 100644
--- a/dev/breeze/src/airflow_breeze/utils/selective_checks.py
+++ b/dev/breeze/src/airflow_breeze/utils/selective_checks.py
@@ -113,13 +113,12 @@ CI_FILE_GROUP_MATCHES = HashableDict(
r"^dev/.*\.py$",
r"^Dockerfile",
r"^scripts",
- r"^setup.py",
- r"^setup.cfg",
+ r"^pyproject.toml",
r"^generated/provider_dependencies.json$",
],
FileGroupForCi.PYTHON_PRODUCTION_FILES: [
r"^airflow/.*\.py",
- r"^setup.py",
+ r"^pyproject.toml",
],
FileGroupForCi.JAVASCRIPT_PRODUCTION_FILES: [
r"^airflow/.*\.[jt]sx?",
@@ -141,8 +140,6 @@ CI_FILE_GROUP_MATCHES = HashableDict(
],
FileGroupForCi.SETUP_FILES: [
r"^pyproject.toml",
- r"^setup.cfg",
- r"^setup.py",
r"^generated/provider_dependencies.json$",
],
FileGroupForCi.DOC_FILES: [
diff --git a/dev/breeze/tests/test_selective_checks.py b/dev/breeze/tests/test_selective_checks.py
index 56e2190628..2db8f657ae 100644
--- a/dev/breeze/tests/test_selective_checks.py
+++ b/dev/breeze/tests/test_selective_checks.py
@@ -426,7 +426,7 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
),
(
pytest.param(
- ("setup.py",),
+ ("pyproject.toml",),
{
"affected-providers-list-as-string": ALL_PROVIDERS_AFFECTED,
"all-python-versions": "['3.8', '3.9', '3.10', '3.11']",
@@ -447,7 +447,7 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"mypy-packages": "['airflow', 'airflow/providers', 'docs', 'dev']",
},
id="Everything should run - including all providers and upgrading to "
- "newer requirements as setup.py changed and all Python versions",
+ "newer requirements as pyproject.toml changed and all Python versions",
)
),
(
@@ -1168,20 +1168,12 @@ def test_no_commit_provided_trigger_full_build_for_any_event_type(github_event):
id="Regular source changed",
),
pytest.param(
- ("setup.py",),
+ ("pyproject.toml",),
{
"upgrade-to-newer-dependencies": "true",
},
(),
- id="Setup.py changed",
- ),
- pytest.param(
- ("setup.cfg",),
- {
- "upgrade-to-newer-dependencies": "true",
- },
- (),
- id="Setup.cfg changed",
+ id="pyproject.toml changed",
),
pytest.param(
("airflow/providers/microsoft/azure/provider.yaml",),
diff --git a/dev/hatch_build.py b/dev/hatch_build.py
new file mode 100644
index 0000000000..afbf35df63
--- /dev/null
+++ b/dev/hatch_build.py
@@ -0,0 +1,176 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import json
+import logging
+import os
+from pathlib import Path
+from subprocess import run
+from typing import Any, Callable, Iterable
+
+from hatchling.builders.config import BuilderConfig
+from hatchling.builders.hooks.plugin.interface import BuildHookInterface
+from hatchling.builders.plugin.interface import BuilderInterface
+from hatchling.plugin.manager import PluginManager
+
+log = logging.getLogger(__name__)
+log_level = logging.getLevelName(os.getenv("CUSTOM_AIRFLOW_BUILD_LOG_LEVEL", "INFO"))
+log.setLevel(log_level)
+
+AIRFLOW_ROOT_PATH = Path(__file__).parent.parent.resolve()
+GENERATED_PROVIDERS_DEPENDENCIES_FILE = AIRFLOW_ROOT_PATH / "generated" / "provider_dependencies.json"
+DEV_DIR_PATH = AIRFLOW_ROOT_PATH / "dev"
+PREINSTALLED_PROVIDERS_FILE = DEV_DIR_PATH / "airflow_pre_installed_providers.txt"
+DEPENDENCIES = json.loads(GENERATED_PROVIDERS_DEPENDENCIES_FILE.read_text())
+PREINSTALLED_PROVIDER_IDS = [
+ package.strip()
+ for package in PREINSTALLED_PROVIDERS_FILE.read_text().splitlines()
+ if not package.strip().startswith("#")
+]
+
+# if providers are ready, we can preinstall them
+PREINSTALLED_PROVIDERS = [
+ f"apache-airflow-providers-{provider_id.replace('.','-')}"
+ for provider_id in PREINSTALLED_PROVIDER_IDS
+ if DEPENDENCIES[provider_id]["state"] == "ready"
+]
+# if provider is in not-ready or pre-release, we need to install its dependencies
+# however we need to skip apache-airflow itself and potentially any providers that are
+PREINSTALLED_NOT_READY_DEPS = []
+for provider_id in PREINSTALLED_PROVIDER_IDS:
+ if DEPENDENCIES[provider_id]["state"] not in ["ready", "suspended", "removed"]:
+ for dependency in DEPENDENCIES[provider_id]["deps"]:
+ if dependency.startswith("apache-airflow-providers"):
+ raise Exception(
+ f"The provider {provider_id} is pre-installed and it has as dependency "
+ f"to another provider {dependency}. This is not allowed. Pre-installed"
+ f"providers should only have 'apache-airflow' and regular dependencies."
+ )
+ if not dependency.startswith("apache-airflow"):
+ PREINSTALLED_NOT_READY_DEPS.append(dependency)
+
+
+class CustomBuild(BuilderInterface[BuilderConfig, PluginManager]):
+ """Custom build class for Airflow assets and git version."""
+
+ # Note that this name of the plugin MUST be `custom` - as long as we use it from custom
+ # hatch_build.py file and not from external plugin. See note in the:
+ # https://hatch.pypa.io/latest/plugins/build-hook/custom/#example
+ #
+ PLUGIN_NAME = "custom"
+
+ def clean(self, directory: str, versions: Iterable[str]) -> None:
+ work_dir = Path(self.root)
+ commands = [
+ ["rm -rf airflow/www/static/dist"],
+ ["rm -rf airflow/www/node_modules"],
+ ]
+ for cmd in commands:
+ run(cmd, cwd=work_dir.as_posix(), check=True, shell=True)
+
+ def get_version_api(self) -> dict[str, Callable[..., str]]:
+ """Custom build target for standard package preparation."""
+ return {"standard": self.build_standard}
+
+ def build_standard(self, directory: str, artifacts: Any, **build_data: Any) -> str:
+ self.write_git_version()
+ work_dir = Path(self.root)
+ commands = [
+ ["pre-commit run --hook-stage manual compile-www-assets --all-files"],
+ ]
+ for cmd in commands:
+ run(cmd, cwd=work_dir.as_posix(), check=True, shell=True)
+ dist_path = work_dir / "airflow" / "www" / "static" / "dist"
+ return dist_path.resolve().as_posix()
+
+ def get_git_version(self) -> str:
+ """
+ Return a version to identify the state of the underlying git repo.
+
+ The version will indicate whether the head of the current git-backed working directory
+ is tied to a release tag or not. It will indicate the former with a 'release:{version}'
+ prefix and the latter with a '.dev0' suffix. Following the prefix will be a sha of the
+ current branch head. Finally, a "dirty" suffix is appended to indicate that uncommitted
+ changes are present.
+
+ Example pre-release version: ".dev0+2f635dc265e78db6708f59f68e8009abb92c1e65".
+ Example release version: ".release+2f635dc265e78db6708f59f68e8009abb92c1e65".
+ Example modified release version: ".release+2f635dc265e78db6708f59f68e8009abb92c1e65".dirty
+
+ :return: Found Airflow version in Git repo.
+ """
+ try:
+ import git
+
+ try:
+ repo = git.Repo(str(Path(self.root) / ".git"))
+ except git.NoSuchPathError:
+ log.warning(".git directory not found: Cannot compute the git version")
+ return ""
+ except git.InvalidGitRepositoryError:
+ log.warning("Invalid .git directory not found: Cannot compute the git version")
+ return ""
+ except ImportError:
+ log.warning("gitpython not found: Cannot compute the git version.")
+ return ""
+ if repo:
+ sha = repo.head.commit.hexsha
+ if repo.is_dirty():
+ return f".dev0+{sha}.dirty"
+ # commit is clean
+ return f".release:{sha}"
+ return "no_git_version"
+
+ def write_git_version(self) -> None:
+ """Write git version to git_version file."""
+ version = self.get_git_version()
+ git_version_file = Path(self.root) / "airflow" / "git_version"
+ self.app.display(f"Writing version {version} to {git_version_file}")
+ git_version_file.write_text(version)
+
+
+class CustomBuildHook(BuildHookInterface[BuilderConfig]):
+ """Custom build hook for Airflow - remove devel extras and adds preinstalled providers."""
+
+ def initialize(self, version: str, build_data: dict[str, Any]) -> None:
+ """
+ This occurs immediately before each build.
+
+ Any modifications to the build data will be seen by the build target.
+ """
+ if version == "standard":
+ # remove devel dependencies from optional dependencies for standard packages
+ self.metadata.core._optional_dependencies = {
+ key: value
+ for (key, value) in self.metadata.core.optional_dependencies.items()
+ if not key.startswith("devel") and key not in ["doc", "doc-gen"]
+ }
+ # Replace editable dependencies with provider dependencies for provider packages
+ for dependency_id in DEPENDENCIES.keys():
+ if DEPENDENCIES[dependency_id]["state"] != "ready":
+ continue
+ normalized_dependency_id = dependency_id.replace(".", "-")
+ self.metadata.core._optional_dependencies[normalized_dependency_id] = [
+ f"apache-airflow-providers-{normalized_dependency_id}"
+ ]
+ # Inject preinstalled providers into the dependencies for standard packages
+ if self.metadata.core._dependencies:
+ for provider in PREINSTALLED_PROVIDERS:
+ self.metadata.core._dependencies.append(provider)
+ for dependency in PREINSTALLED_NOT_READY_DEPS:
+ self.metadata.core._dependencies.append(dependency)
diff --git a/airflow/providers/installed_providers.txt b/dev/prod_image_installed_providers.txt
similarity index 74%
rename from airflow/providers/installed_providers.txt
rename to dev/prod_image_installed_providers.txt
index d416a8a47b..3aadc9d0b1 100644
--- a/airflow/providers/installed_providers.txt
+++ b/dev/prod_image_installed_providers.txt
@@ -1,3 +1,4 @@
+# List of all the providers installed in regular airflow PROD image
amazon
celery
cncf.kubernetes
diff --git a/dev/refresh_images.sh b/dev/refresh_images.sh
index 2b11229aa0..a6d4e9a06f 100755
--- a/dev/refresh_images.sh
+++ b/dev/refresh_images.sh
@@ -36,7 +36,7 @@ breeze ci-image build \
rm -fv ./dist/* ./docker-context-files/*
breeze release-management prepare-provider-packages \
- --package-list-file ./airflow/providers/installed_providers.txt \
+ --package-list-file ./dev/prod_image_installed_providers.txt \
--package-format wheel \
--version-suffix-for-pypi dev0
diff --git a/dev/sign.sh b/dev/sign.sh
index 178ba3ccb7..0d4c2861a5 100755
--- a/dev/sign.sh
+++ b/dev/sign.sh
@@ -17,9 +17,7 @@
# under the License.
set -euo pipefail
-# Use this to sign the tar balls generated from
-# python setup.py sdist --formats=gztar
-# ie. sign.sh <my_tar_ball>
+# Use this to sign the tar balls generated via hatch
# you will still be required to type in your signing key password
# or it needs to be available in your keychain
diff --git a/docker_tests/docker_tests_utils.py b/docker_tests/docker_tests_utils.py
index 8d95805f64..7eea98e9bd 100644
--- a/docker_tests/docker_tests_utils.py
+++ b/docker_tests/docker_tests_utils.py
@@ -73,7 +73,8 @@ in the image.
It can mean one of those:
1) The main is currently broken (other PRs will fail with the same error)
-2) You changed some dependencies in setup.py or setup.cfg and they are conflicting.
+2) You changed some dependencies in pyproject.toml (either manually or automatically by pre-commit)
+ and they are conflicting.
diff --git a/docker_tests/test_prod_image.py b/docker_tests/test_prod_image.py
index e54de129d8..4f6a93de42 100644
--- a/docker_tests/test_prod_image.py
+++ b/docker_tests/test_prod_image.py
@@ -20,10 +20,10 @@ import json
import os
import subprocess
from importlib.util import find_spec
+from pathlib import Path
import pytest
-# isort:off (needed to workaround isort bug)
from docker_tests.command_utils import run_command
from docker_tests.constants import SOURCE_ROOT
from docker_tests.docker_tests_utils import (
@@ -33,10 +33,20 @@ from docker_tests.docker_tests_utils import (
run_python_in_docker,
)
-# isort:on (needed to workaround isort bug)
-from setup import PREINSTALLED_PROVIDERS
-
-INSTALLED_PROVIDER_PATH = SOURCE_ROOT / "airflow" / "providers" / "installed_providers.txt"
+DEV_DIR_PATH = SOURCE_ROOT / "dev"
+AIRFLOW_PRE_INSTALLED_PROVIDERS_FILE_PATH = DEV_DIR_PATH / "airflow_pre_installed_providers.txt"
+PROD_IMAGE_PROVIDERS_FILE_PATH = DEV_DIR_PATH / "prod_image_installed_providers.txt"
+AIRFLOW_ROOT_PATH = Path(__file__).parents[2].resolve()
+SLIM_IMAGE_PROVIDERS = [
+ f"apache-airflow-providers-{provider_id.replace('.','-')}"
+ for provider_id in AIRFLOW_PRE_INSTALLED_PROVIDERS_FILE_PATH.read_text().splitlines()
+ if not provider_id.startswith("#")
+]
+REGULAR_IMAGE_PROVIDERS = [
+ f"apache-airflow-providers-{provider_id.replace('.','-')}"
+ for provider_id in PROD_IMAGE_PROVIDERS_FILE_PATH.read_text().splitlines()
+ if not provider_id.startswith("#")
+]
class TestCommands:
@@ -80,23 +90,22 @@ class TestCommands:
class TestPythonPackages:
def test_required_providers_are_installed(self):
if os.environ.get("TEST_SLIM_IMAGE"):
- lines = PREINSTALLED_PROVIDERS
+ packages_to_install = set(SLIM_IMAGE_PROVIDERS)
+ package_file = AIRFLOW_PRE_INSTALLED_PROVIDERS_FILE_PATH
else:
- lines = (d.strip() for d in INSTALLED_PROVIDER_PATH.read_text().splitlines())
- packages_to_install = {f"apache-airflow-providers-{d.replace('.', '-')}" for d in lines}
+ packages_to_install = set(REGULAR_IMAGE_PROVIDERS)
+ package_file = PROD_IMAGE_PROVIDERS_FILE_PATH
assert len(packages_to_install) != 0
-
output = run_bash_in_docker(
"airflow providers list --output json", stderr=subprocess.DEVNULL, return_output=True
)
providers = json.loads(output)
- packages_installed = {d["package_name"] for d in providers}
+ packages_installed = set(d["package_name"] for d in providers)
assert len(packages_installed) != 0
- assert packages_to_install == packages_installed, (
- f"List of expected installed packages and image content mismatch. "
- f"Check {INSTALLED_PROVIDER_PATH} file."
- )
+ assert (
+ packages_to_install == packages_installed
+ ), f"List of expected installed packages and image content mismatch. Check {package_file} file."
def test_pip_dependencies_conflict(self):
try:
diff --git a/docs/apache-airflow-providers-apache-hive/index.rst b/docs/apache-airflow-providers-apache-hive/index.rst
index 63d3096112..8aca48be98 100644
--- a/docs/apache-airflow-providers-apache-hive/index.rst
+++ b/docs/apache-airflow-providers-apache-hive/index.rst
@@ -106,7 +106,7 @@ PIP package Version required
``apache-airflow-providers-common-sql`` ``>=1.3.1``
``hmsclient`` ``>=0.1.0``
``pandas`` ``>=0.17.1``
-``pyhive[hive_pure_sasl]`` ``>=0.7.0``
+``pyhive[hive-pure-sasl]`` ``>=0.7.0``
``thrift`` ``>=0.9.2``
======================================= ==================
diff --git a/docs/apache-airflow-providers/howto/create-custom-providers.rst b/docs/apache-airflow-providers/howto/create-custom-providers.rst
index 7b2a2fcf4c..0e868ac253 100644
--- a/docs/apache-airflow-providers/howto/create-custom-providers.rst
+++ b/docs/apache-airflow-providers/howto/create-custom-providers.rst
@@ -196,8 +196,8 @@ names, so preferably choose package that is in your "domain".
You need to do the following to turn an existing Python package into a provider (see below for examples):
-* Add the ``apache_airflow_provider`` entry point in the ``setup.cfg`` - this tells airflow where to get
- the required provider metadata
+* Add the ``apache_airflow_provider`` entry point in the ``pyproject.toml`` file - this tells airflow
+ where to get the required provider metadata
* Create the function that you refer to in the first step as part of your package: this functions returns a
dictionary that contains all meta-data about your provider package
* If you want Airflow to link to documentation of your Provider in the providers page, make sure
@@ -211,29 +211,27 @@ You need to do the following to turn an existing Python package into a provider
.. exampleinclude:: /../../airflow/provider_info.schema.json
:language: json
-Example ``setup.cfg``:
+Example ``pyproject.toml``:
-.. code-block:: cfg
+.. code-block:: toml
- [options.entry_points]
- # the function get_provider_info is defined in myproviderpackage.somemodule
- apache_airflow_provider=
- provider_info=myproviderpackage.somemodule:get_provider_info
+ [project.entry-points."apache_airflow_provider"]
+ provider_info = "airflow.providers.myproviderpackage.get_provider_info:get_provider_info"
-Example ``myproviderpackage/somemodule.py``:
-.. code-block:: Python
+Example ``myproviderpackage/get_provider_info.py``:
- def get_provider_info():
- return {
- "package-name": "my-package-name",
- "name": "name",
- "description": "a description",
- "hook-class-names": [
- "myproviderpackage.hooks.source.SourceHook",
- ],
- }
+.. code-block:: Python
+ def get_provider_info():
+ return {
+ "package-name": "my-package-name",
+ "name": "name",
+ "description": "a description",
+ "hook-class-names": [
+ "myproviderpackage.hooks.source.SourceHook",
+ ],
+ }
**Is there a convention for a connection id and type?**
diff --git a/docs/apache-airflow/administration-and-deployment/modules_management.rst b/docs/apache-airflow/administration-and-deployment/modules_management.rst
index 44b360fbc8..a619d2a06e 100644
--- a/docs/apache-airflow/administration-and-deployment/modules_management.rst
+++ b/docs/apache-airflow/administration-and-deployment/modules_management.rst
@@ -331,20 +331,12 @@ packages, so learning how to build your package is handy.
Here is how to create your package:
-1. Before starting, install the following packages:
+1. Before starting, choose and install the build/packaging tool that you will use, ideally it should be
+PEP-621 compliant to be able to switch to a different tool easily.
+The popular choices are setuptools, poetry, hatch, flit.
-``setuptools``: setuptools is a package development process library designed
-for creating and distributing Python packages.
-
-``wheel``: The wheel package provides a bdist_wheel command for setuptools. It
-creates .whl file which is directly installable through the ``pip install``
-command. We can then upload the same file to `PyPI <https://pypi.org>`_.
-
-.. code-block:: bash
-
- pip install --upgrade pip setuptools wheel
-
-2. Create the package directory - in our case, we will call it ``airflow_operators``.
+2. Decide when you create your own package. create the package directory - in our case,
+ we will call it ``airflow_operators``.
.. code-block:: bash
@@ -358,42 +350,16 @@ command. We can then upload the same file to `PyPI <https://pypi.org>`_.
When we import this package, it should print the above message.
-4. Create ``setup.py``:
-
-.. code-block:: python
-
- import setuptools
-
- setuptools.setup(
- name="airflow_operators",
- packages=setuptools.find_packages(),
- )
-
-5. Build the wheel:
-
-.. code-block:: bash
-
- python setup.py bdist_wheel
+4. Create ``pyproject.toml`` and fill it with build tool configuration of your choice
+See `The pyproject.toml specification <https://packaging.python.org/en/latest/specifications/pyproject-toml/#pyproject-toml-spec>`__
-This will create a few directories in the project and the overall structure will
-look like following:
+5. Build your project using the tool of your choice. For example for hatch it can be:
.. code-block:: bash
- .
- ├── airflow_operators
- │ ├── __init__.py
- ├── airflow_operators.egg-info
- │ ├── PKG-INFO
- │ ├── SOURCES.txt
- │ ├── dependency_links.txt
- │ └── top_level.txt
- ├── build
- │ └── bdist.macosx-10.15-x86_64
- ├── dist
- │ └── airflow_operators-0.0.0-py3-none-any.whl
- └── setup.py
+ hatch build -t wheel
+This will create .whl file in your ``dist`` folder
6. Install the .whl file using pip:
diff --git a/docs/apache-airflow/extra-packages-ref.rst b/docs/apache-airflow/extra-packages-ref.rst
index 0b6a51fd32..ffbe922524 100644
--- a/docs/apache-airflow/extra-packages-ref.rst
+++ b/docs/apache-airflow/extra-packages-ref.rst
@@ -18,17 +18,22 @@
Reference for package extras
''''''''''''''''''''''''''''
-Here's the list of all the extra dependencies of Apache Airflow.
+Airflow has a number of optional "extras" that you can use to add features to your installation when you
+are installing Airflow. Those extras are a good way for the users to manage their installation, but also
+they are useful for contributors to airflow when they want to contribute some of the features - including
+optional integrations of Airflow - via providers.
-The entries with ``*`` in the ``Preinstalled`` column indicate that those extras (providers) are always
-pre-installed when Airflow is installed.
+.. warning::
+
+ Traditionally in Airflow some of the extras used `.` and `_` to separate the parts of the extra name.
+ This was not PEP-685 normalized name and we opted to change it to to `-` for all our extras, Expecting that
+ PEP-685 will be implemented in full by `pip` and other tools we change all our extras to use `-` as
+ separator even if in some cases it will introduce warnings (the warnings are harmless). This is future
+ proof approach. It's also fully backwards-compatible if you use `_` or `.` in your extras, but we
+ recommend using `-` as separator in the future.
-.. note::
- You can disable automated installation of the providers with extras when installing Airflow. You need to
- have ``INSTALL_PROVIDERS_FROM_SOURCES`` environment variable to ``true`` before running ``pip install``
- command. Contributors need to set it, if they are installing Airflow locally, and want to develop
- providers directly via Airflow sources. This variable is automatically set in ``Breeze``
- development environment. Setting this variable is not needed in editable mode (``pip install -e``).
+
+Here's the list of all the extra dependencies of Apache Airflow.
Core Airflow extras
-------------------
@@ -46,11 +51,13 @@ python dependencies for the provided package.
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| cgroups | ``pip install 'apache-airflow[cgroups]'`` | Needed To use CgroupTaskRunner |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
-| deprecated_api | ``pip install 'apache-airflow[deprecated_api]'`` | Deprecated, experimental API that is replaced with the new REST API |
+| deprecated-api | ``pip install 'apache-airflow[deprecated-api]'`` | Deprecated, experimental API that is replaced with the new REST API |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
-| github_enterprise | ``pip install 'apache-airflow[github_enterprise]'`` | GitHub Enterprise auth backend |
+| github-enterprise | ``pip install 'apache-airflow[github-enterprise]'`` | GitHub Enterprise auth backend |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
-| google_auth | ``pip install 'apache-airflow[google_auth]'`` | Google auth backend |
+| google-auth | ``pip install 'apache-airflow[google-auth]'`` | Google auth backend |
++---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
+| graphviz | ``pip install 'apache-airflow[graphvis]'`` | Enables exporting DAGs to .dot graphical output |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| graphviz | ``pip install 'apache-airflow[graphviz]'`` | Graphviz renderer for converting DAG to graphical output |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
@@ -99,7 +106,7 @@ with a consistent set of dependencies based on constraint files provided by Airf
.. code-block:: bash
:substitutions:
- pip install apache-airflow[google,amazon,apache.spark]==|version| \
+ pip install apache-airflow[google,amazon,apache-spark]==|version| \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
Note, that this will install providers in the versions that were released at the time of Airflow |version| release. You can later
@@ -116,40 +123,39 @@ custom bash/python providers).
+---------------------+-----------------------------------------------------+------------------------------------------------+
| extra | install command | enables |
+=====================+=====================================================+================================================+
-| apache.atlas | ``pip install 'apache-airflow[apache.atlas]'`` | Apache Atlas |
+| apache-atlas | ``pip install 'apache-airflow[apache-atlas]'`` | Apache Atlas |
+---------------------+-----------------------------------------------------+------------------------------------------------+
-| apache.beam | ``pip install 'apache-airflow[apache.beam]'`` | Apache Beam operators & hooks |
+| apache-beam | ``pip install 'apache-airflow[apache-beam]'`` | Apache Beam operators & hooks |
+---------------------+-----------------------------------------------------+------------------------------------------------+
-| apache.cassandra | ``pip install 'apache-airflow[apache.cassandra]'`` | Cassandra related operators & hooks |
+| apache-cassandra | ``pip install 'apache-airflow[apache-cassandra]'`` | Cassandra related operators & hooks |
+---------------------+-----------------------------------------------------+------------------------------------------------+
-| apache.drill | ``pip install 'apache-airflow[apache.drill]'`` | Drill related operators & hooks |
+| apache-drill | ``pip install 'apache-airflow[apache-drill]'`` | Drill related operators & hooks |
+---------------------+-----------------------------------------------------+------------------------------------------------+
-| apache.druid | ``pip install 'apache-airflow[apache.druid]'`` | Druid related operators & hooks |
+| apache-druid | ``pip install 'apache-airflow[apache-druid]'`` | Druid related operators & hooks |
+---------------------+-----------------------------------------------------+------------------------------------------------+
-| apache.flink | ``pip install 'apache-airflow[apache.flink]'`` | Flink related operators & hooks |
+| apache-flink | ``pip install 'apache-airflow[apache-flink]'`` | Flink related operators & hooks |
+---------------------+-----------------------------------------------------+------------------------------------------------+
-| apache.hdfs | ``pip install 'apache-airflow[apache.hdfs]'`` | HDFS hooks and operators |
+| apache-hdfs | ``pip install 'apache-airflow[apache-hdfs]'`` | HDFS hooks and operators |
+---------------------+-----------------------------------------------------+------------------------------------------------+
-| apache.hive | ``pip install 'apache-airflow[apache.hive]'`` | All Hive related operators |
+| apache-hive | ``pip install 'apache-airflow[apache-hive]'`` | All Hive related operators |
+---------------------+-----------------------------------------------------+------------------------------------------------+
-| apache.impala | ``pip install 'apache-airflow[apache.impala]'`` | All Impala related operators & hooks |
+| apache-impala | ``pip install 'apache-airflow[apache-impala]'`` | All Impala related operators & hooks |
+---------------------+-----------------------------------------------------+------------------------------------------------+
-| apache.kafka | ``pip install 'apache-airflow[apache.kafka]'`` | All Kafka related operators & hooks |
+| apache-kafka | ``pip install 'apache-airflow[apache-kafka]'`` | All Kafka related operators & hooks |
+---------------------+-----------------------------------------------------+------------------------------------------------+
-| apache.kylin | ``pip install 'apache-airflow[apache.kylin]'`` | All Kylin related operators & hooks |
+| apache-kylin | ``pip install 'apache-airflow[apache-kylin]'`` | All Kylin related operators & hooks |
+---------------------+-----------------------------------------------------+------------------------------------------------+
-| apache.livy | ``pip install 'apache-airflow[apache.livy]'`` | All Livy related operators, hooks & sensors |
+| apache-livy | ``pip install 'apache-airflow[apache-livy]'`` | All Livy related operators, hooks & sensors |
+---------------------+-----------------------------------------------------+------------------------------------------------+
-| apache.pig | ``pip install 'apache-airflow[apache.pig]'`` | All Pig related operators & hooks |
+| apache-pig | ``pip install 'apache-airflow[apache-pig]'`` | All Pig related operators & hooks |
+---------------------+-----------------------------------------------------+------------------------------------------------+
-| apache.pinot | ``pip install 'apache-airflow[apache.pinot]'`` | All Pinot related hooks |
+| apache-pinot | ``pip install 'apache-airflow[apache-pinot]'`` | All Pinot related hooks |
+---------------------+-----------------------------------------------------+------------------------------------------------+
-| apache.spark | ``pip install 'apache-airflow[apache.spark]'`` | All Spark related operators & hooks |
+| apache-spark | ``pip install 'apache-airflow[apache-spark]'`` | All Spark related operators & hooks |
+---------------------+-----------------------------------------------------+------------------------------------------------+
-| apache.webhdfs | ``pip install 'apache-airflow[apache.webhdfs]'`` | HDFS hooks and operators |
+| apache-webhdfs | ``pip install 'apache-airflow[apache-webhdfs]'`` | HDFS hooks and operators |
+---------------------+-----------------------------------------------------+------------------------------------------------+
-
External Services extras
========================
@@ -168,9 +174,9 @@ These are extras that add dependencies needed for integration with external serv
+---------------------+-----------------------------------------------------+-----------------------------------------------------+
| asana | ``pip install 'apache-airflow[asana]'`` | Asana hooks and operators |
+---------------------+-----------------------------------------------------+-----------------------------------------------------+
-| atlassian.jira | ``pip install 'apache-airflow[atlassian.jira]'`` | Jira hooks and operators |
+| atlassian-jira | ``pip install 'apache-airflow[atlassian-jira]'`` | Jira hooks and operators |
+---------------------+-----------------------------------------------------+-----------------------------------------------------+
-| azure | ``pip install 'apache-airflow[microsoft.azure]'`` | Microsoft Azure |
+| microsoft-azure | ``pip install 'apache-airflow[microsoft-azure]'`` | Microsoft Azure |
+---------------------+-----------------------------------------------------+-----------------------------------------------------+
| cloudant | ``pip install 'apache-airflow[cloudant]'`` | Cloudant hook |
+---------------------+-----------------------------------------------------+-----------------------------------------------------+
@@ -180,7 +186,7 @@ These are extras that add dependencies needed for integration with external serv
+---------------------+-----------------------------------------------------+-----------------------------------------------------+
| datadog | ``pip install 'apache-airflow[datadog]'`` | Datadog hooks and sensors |
+---------------------+-----------------------------------------------------+-----------------------------------------------------+
-| dbt.cloud | ``pip install 'apache-airflow[dbt.cloud]'`` | dbt Cloud hooks and operators |
+| dbt-cloud | ``pip install 'apache-airflow[dbt-cloud]'`` | dbt Cloud hooks and operators |
+---------------------+-----------------------------------------------------+-----------------------------------------------------+
| dingding | ``pip install 'apache-airflow[dingding]'`` | Dingding hooks and sensors |
+---------------------+-----------------------------------------------------+-----------------------------------------------------+
@@ -243,7 +249,7 @@ Some of those enable Airflow to use executors to run tasks with them - other tha
+---------------------+-----------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------+
| celery | ``pip install 'apache-airflow[celery]'`` | Celery dependencies and sensor | CeleryExecutor, CeleryKubernetesExecutor |
+---------------------+-----------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------+
-| cncf.kubernetes | ``pip install 'apache-airflow[cncf.kubernetes]'`` | Kubernetes client libraries, KubernetesPodOperator & friends | KubernetesExecutor, LocalKubernetesExecutor |
+| cncf-kubernetes | ``pip install 'apache-airflow[cncf-kubernetes]'`` | Kubernetes client libraries, KubernetesPodOperator & friends | KubernetesExecutor, LocalKubernetesExecutor |
+---------------------+-----------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------+
| docker | ``pip install 'apache-airflow[docker]'`` | Docker hooks and operators | |
+---------------------+-----------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------+
@@ -259,7 +265,7 @@ Some of those enable Airflow to use executors to run tasks with them - other tha
+---------------------+-----------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------+
| mongo | ``pip install 'apache-airflow[mongo]'`` | Mongo hooks and operators | |
+---------------------+-----------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------+
-| microsoft.mssql | ``pip install 'apache-airflow[microsoft.mssql]'`` | Microsoft SQL Server operators and hook. | |
+| microsoft-mssql | ``pip install 'apache-airflow[microsoft-mssql]'`` | Microsoft SQL Server operators and hook. | |
+---------------------+-----------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------+
| mysql | ``pip install 'apache-airflow[mysql]'`` | MySQL operators and hook | |
+---------------------+-----------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------+
@@ -290,12 +296,16 @@ Other extras
These are extras that provide support for integration with external systems via some - usually - standard protocols.
+The entries with ``*`` in the ``Preinstalled`` column indicate that those extras (providers) are always
+pre-installed when Airflow is installed.
+
+
+---------------------+-----------------------------------------------------+--------------------------------------+--------------+
| extra | install command | enables | Preinstalled |
+=====================+=====================================================+======================================+==============+
-| common.io | ``pip install 'apache-airflow[common.io]'`` | Core IO Operators | |
+| common-io | ``pip install 'apache-airflow[common-io]'`` | Core IO Operators | |
+---------------------+-----------------------------------------------------+--------------------------------------+--------------+
-| common.sql | ``pip install 'apache-airflow[common.sql]'`` | Core SQL Operators | * |
+| common-sql | ``pip install 'apache-airflow[common-sql]'`` | Core SQL Operators | * |
+---------------------+-----------------------------------------------------+--------------------------------------+--------------+
| ftp | ``pip install 'apache-airflow[ftp]'`` | FTP hooks and operators | * |
+---------------------+-----------------------------------------------------+--------------------------------------+--------------+
@@ -307,9 +317,9 @@ These are extras that provide support for integration with external systems via
+---------------------+-----------------------------------------------------+--------------------------------------+--------------+
| jdbc | ``pip install 'apache-airflow[jdbc]'`` | JDBC hooks and operators | |
+---------------------+-----------------------------------------------------+--------------------------------------+--------------+
-| microsoft.psrp | ``pip install 'apache-airflow[microsoft.psrp]'`` | PSRP hooks and operators | |
+| microsoft-psrp | ``pip install 'apache-airflow[microsoft-psrp]'`` | PSRP hooks and operators | |
+---------------------+-----------------------------------------------------+--------------------------------------+--------------+
-| microsoft.winrm | ``pip install 'apache-airflow[microsoft.winrm]'`` | WinRM hooks and operators | |
+| microsoft-winrm | ``pip install 'apache-airflow[microsoft-winrm]'`` | WinRM hooks and operators | |
+---------------------+-----------------------------------------------------+--------------------------------------+--------------+
| openlineage | ``pip install 'apache-airflow[openlineage]'`` | Sending OpenLineage events | |
+---------------------+-----------------------------------------------------+--------------------------------------+--------------+
@@ -326,44 +336,90 @@ These are extras that provide support for integration with external systems via
| ssh | ``pip install 'apache-airflow[ssh]'`` | SSH hooks and operators | |
+---------------------+-----------------------------------------------------+--------------------------------------+--------------+
-Bundle extras
--------------
+Production Bundle extras
+-------------------------
-These are extras that install one or more extras as a bundle. Note that these extras should only be used for "development" version
-of Airflow - i.e. when Airflow is installed from sources. Because of the way how bundle extras are constructed they might not
-work when airflow is installed from 'PyPI`.
-
-If you want to install Airflow from PyPI with "all" extras (which should basically be never needed - you almost never need all extras from Airflow),
-you need to list explicitly all the non-bundle extras that you want to install.
+These are extras that install one or more extras as a bundle.
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
| extra | install command | enables |
+=====================+=====================================================+========================================================================+
| all | ``pip install 'apache-airflow[all]'`` | All Airflow user facing features (no devel and doc requirements) |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
-| all_dbs | ``pip install 'apache-airflow[all_dbs]'`` | All database integrations |
+| all-core | ``pip install 'apache-airflow[all-core]'`` | All core airflow features that do not require installing providers |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
-| devel | ``pip install 'apache-airflow[devel]'`` | Minimum development dependencies (without Hadoop, Kerberos, providers) |
+| all-dbs | ``pip install 'apache-airflow[all-dbs]'`` | All database integrations |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
-| devel_hadoop | ``pip install 'apache-airflow[devel_hadoop]'`` | Adds Hadoop stack libraries to ``devel`` dependencies |
+
+Development extras
+------------------
+
+The ``devel`` extras only make sense in editable mode. Users of Airflow should not be using them, unless they
+start contributing back and install airflow from sources. Those extras are only available in Airflow when
+it is installed in editable mode from sources (``pip install -e .[devel,EXTRAS]``).
+
+Devel extras
+============
+
+The devel extras do not install dependencies for features of Airflow, but add functionality that is needed to
+develop Airflow, such as running tests, static checks.
+
++---------------------+-----------------------------------------+------------------------------------------------------+
+| extra | install command | enables |
++=====================+=========================================+======================================================+
+| devel-debuggers | pip install -e '.[devel-debuggers]' | Adds all test libraries needed to test debuggers |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel-devscripts | pip install -e '.[devel-devscripts]' | Adds all test libraries needed to test devel scripts |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel-duckdb | pip install -e '.[devel-duckdb]' | Adds all test libraries needed to test duckdb |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel-iceberg | pip install -e '.[devel-iceberg]' | Adds all test libraries needed to test iceberg |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel-mypy | pip install -e '.[devel-mypy]' | Adds all test libraries needed to test mypy |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel-sentry | pip install -e '.[devel-sentry]' | Adds all test libraries needed to test sentry |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel-static-checks | pip install -e '.[devel-static-checks]' | Adds all test libraries needed to test static_checks |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel-tests | pip install -e '.[devel-tests]' | Adds all test libraries needed to test tests |
++---------------------+-----------------------------------------+------------------------------------------------------+
+
+Bundle devel extras
+===================
+
+Those are extras that bundle devel, editable and doc extras together to make it easy to install them together in a single installation. Some of the
+extras are more difficult to install on certain systems (such as ARM MacBooks) because they require system level dependencies to be installed.
+
+Note that ``pip install -e ".[devel]"`` should be run at least once, the first time you initialize the editable environment in order
+to get minimal, complete test environment with usual tools and dependencies needed for unit testing.
+
++---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
+| extra | install command | enables |
++=====================+=====================================================+========================================================================+
+| devel | ``pip install -e '.[devel]'`` | Minimum development dependencies - minimal, complete test environment |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
-| devel_all | ``pip install 'apache-airflow[devel_all]'`` | Everything needed for development including Hadoop and providers |
+| devel-hadoop | ``pip install -e '.[devel-hadoop]'`` | Adds Hadoop stack libraries ``devel`` dependencies |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
-| devel_ci | ``pip install 'apache-airflow[devel_ci]'`` | All dependencies required for CI tests (same as ``devel_all``) |
+| devel-all-dbs | ``pip install -e '.[devel-all-dbs]'`` | Adds all libraries needed to test database providers |
++---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
+| devel-all | ``pip install -e '.[devel-all]'`` | Everything needed for development including Hadoop, all devel extras, |
+| | | all doc extras. Generally: all possible dependencies |
++---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
+| devel-ci | ``pip install -e '.[devel-ci]'`` | All dependencies required for CI tests (same as ``devel-all``) |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
Doc extras
-----------
+==========
Those are the extras that are needed to generated documentation for Airflow. This is used for development time only
-+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| extra | install command | enables |
-+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| doc | ``pip install 'apache-airflow[doc]'`` | Packages needed to build docs (included in ``devel``) |
-+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| doc_gen | ``pip install 'apache-airflow[doc_gen]'`` | Packages needed to generate er diagrams (included in ``devel_all``) |
-+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
++---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
+| extra | install command | enables |
++=====================+=====================================================+========================================================================+
+| doc | ``pip install -e '.[doc]'`` | Packages needed to build docs (included in ``devel``) |
++---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
+| doc-gen | ``pip install -e '.[doc-gen]'`` | Packages needed to generate er diagrams (included in ``devel-all``) |
++---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
Deprecated 1.10 extras
@@ -378,37 +434,37 @@ so there is no replacement for ``crypto`` extra.
+---------------------+-----------------------------+
| Deprecated extra | Extra to be used instead |
+=====================+=============================+
-| atlas | apache.atlas |
+| atlas | apache-atlas |
+---------------------+-----------------------------+
| aws | amazon |
+---------------------+-----------------------------+
-| azure | microsoft.azure |
+| azure | microsoft-azure |
+---------------------+-----------------------------+
-| cassandra | apache.cassandra |
+| cassandra | apache-cassandra |
+---------------------+-----------------------------+
| crypto | |
+---------------------+-----------------------------+
-| druid | apache.druid |
+| druid | apache-druid |
+---------------------+-----------------------------+
| gcp | google |
+---------------------+-----------------------------+
| gcp_api | google |
+---------------------+-----------------------------+
-| hdfs | apache.hdfs |
+| hdfs | apache-hdfs |
+---------------------+-----------------------------+
-| hive | apache.hive |
+| hive | apache-hive |
+---------------------+-----------------------------+
-| kubernetes | cncf.kubernetes |
+| kubernetes | cncf-kubernetes |
+---------------------+-----------------------------+
-| mssql | microsoft.mssql |
+| mssql | microsoft-mssql |
+---------------------+-----------------------------+
-| pinot | apache.pinot |
+| pinot | apache-pinot |
+---------------------+-----------------------------+
| s3 | amazon |
+---------------------+-----------------------------+
-| spark | apache.spark |
+| spark | apache-spark |
+---------------------+-----------------------------+
-| webhdfs | apache.webhdfs |
+| webhdfs | apache-webhdfs |
+---------------------+-----------------------------+
-| winrm | microsoft.winrm |
+| winrm | microsoft-winrm |
+---------------------+-----------------------------+
diff --git a/docs/apache-airflow/installation/installing-from-pypi.rst b/docs/apache-airflow/installation/installing-from-pypi.rst
index ed345f0cc1..6c379f6639 100644
--- a/docs/apache-airflow/installation/installing-from-pypi.rst
+++ b/docs/apache-airflow/installation/installing-from-pypi.rst
@@ -84,9 +84,9 @@ Airflow™ installation can be tricky because Airflow is both a library and an a
Libraries usually keep their dependencies open and applications usually pin them, but we should do neither
and both at the same time. We decided to keep our dependencies as open as possible
-(in ``setup.cfg`` and ``setup.py``) so users can install different
-version of libraries if needed. This means that from time to time plain ``pip install apache-airflow`` will
-not work or will produce an unusable Airflow installation.
+(in ``pyproject.toml``) so users can install different version of libraries if needed. This means that
+from time to time plain ``pip install apache-airflow`` will not work or will produce an unusable
+Airflow installation.
Reproducible Airflow installation
=================================
diff --git a/docs/docker-stack/build-arg-ref.rst b/docs/docker-stack/build-arg-ref.rst
index 73c30a3892..4ea605a6de 100644
--- a/docs/docker-stack/build-arg-ref.rst
+++ b/docs/docker-stack/build-arg-ref.rst
@@ -84,19 +84,19 @@ List of default extras in the production Dockerfile:
* amazon
* async
* celery
-* cncf.kubernetes
-* common.io
+* cncf-kubernetes
+* common-io
* docker
* elasticsearch
* ftp
* google
-* google_auth
+* google-auth
* graphviz
* grpc
* hashicorp
* http
* ldap
-* microsoft.azure
+* microsoft-azure
* mysql
* odbc
* openlineage
@@ -225,12 +225,6 @@ You can see some examples of those in:
| | | for Airflow version installation - for |
| | | example ``<2.0.2`` for automated builds. |
+------------------------------------+------------------------------------------+------------------------------------------+
-| ``INSTALL_PROVIDERS_FROM_SOURCES`` | ``false`` | If set to ``true`` and image is built |
-| | | from sources, all provider packages are |
-| | | installed from sources rather than from |
-| | | packages. It has no effect when |
-| | | installing from PyPI or GitHub repo. |
-+------------------------------------+------------------------------------------+------------------------------------------+
| ``AIRFLOW_CONSTRAINTS_LOCATION`` | | If not empty, it will override the |
| | | source of the constraints with the |
| | | specified URL or file. Note that the |
@@ -260,7 +254,7 @@ Pre-caching PIP dependencies
............................
When image is build from PIP, by default pre-caching of PIP dependencies is used. This is in order to speed-up incremental
-builds during development. When pre-cached PIP dependencies are used and ``setup.py`` or ``setup.cfg`` changes, the
+builds during development. When pre-cached PIP dependencies are used and ``pyproject.toml`` changes, the
PIP dependencies are already pre-installed, thus resulting in much faster image rebuild. This is purely an optimization
of time needed to build the images and should be disabled if you want to install Airflow from
Docker context files.
diff --git a/docs/spelling_wordlist.txt b/docs/spelling_wordlist.txt
index e7b4b509b9..ef260e0d2b 100644
--- a/docs/spelling_wordlist.txt
+++ b/docs/spelling_wordlist.txt
@@ -458,6 +458,7 @@ dest
dev
devel
DevOps
+devscripts
devtools
df
dicts
@@ -514,6 +515,7 @@ dsn
dttm
dtypes
du
+duckdb
durations
dylib
Dynamodb
diff --git a/generated/PYPI_README.md b/generated/PYPI_README.md
index d11eb11cc5..30ad2e2f41 100644
--- a/generated/PYPI_README.md
+++ b/generated/PYPI_README.md
@@ -105,7 +105,7 @@ Documentation for dependent projects like provider packages, Docker image, Helm
We publish Apache Airflow as `apache-airflow` package in PyPI. Installing it however might be sometimes tricky
because Airflow is a bit of both a library and application. Libraries usually keep their dependencies open, and
applications usually pin them, but we should do neither and both simultaneously. We decided to keep
-our dependencies as open as possible (in `setup.py`) so users can install different versions of libraries
+our dependencies as open as possible (in `pyproject.toml`) so users can install different versions of libraries
if needed. This means that `pip install apache-airflow` will not work from time to time or will
produce unusable Airflow installation.
diff --git a/generated/provider_dependencies.json b/generated/provider_dependencies.json
index d42ae80ee4..e62c8521da 100644
--- a/generated/provider_dependencies.json
+++ b/generated/provider_dependencies.json
@@ -4,6 +4,7 @@
"apache-airflow-providers-http",
"apache-airflow>=2.6.0"
],
+ "devel-deps": [],
"cross-providers-deps": [
"http"
],
@@ -17,6 +18,7 @@
"apache-airflow>=2.6.0",
"oss2>=2.14.0"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -35,6 +37,16 @@
"sqlalchemy_redshift>=0.8.6",
"watchtower>=2.0.1,<4"
],
+ "devel-deps": [
+ "aiobotocore>=2.7.0",
+ "aws_xray_sdk>=2.12.0",
+ "moto[cloudformation,glue]>=4.2.12",
+ "mypy-boto3-appflow>=1.33.0",
+ "mypy-boto3-rds>=1.33.0",
+ "mypy-boto3-redshift-data>=1.33.0",
+ "mypy-boto3-s3>=1.33.0",
+ "s3fs>=2023.10.0"
+ ],
"cross-providers-deps": [
"apache.hive",
"cncf.kubernetes",
@@ -59,6 +71,7 @@
"apache-beam>=2.53.0",
"pyarrow>=14.0.1"
],
+ "devel-deps": [],
"cross-providers-deps": [
"google"
],
@@ -70,6 +83,7 @@
"apache-airflow>=2.6.0",
"cassandra-driver>=3.13.0"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -80,6 +94,7 @@
"apache-airflow>=2.6.0",
"sqlalchemy-drill>=1.1.0"
],
+ "devel-deps": [],
"cross-providers-deps": [
"common.sql"
],
@@ -92,6 +107,7 @@
"apache-airflow>=2.6.0",
"pydruid>=0.4.1"
],
+ "devel-deps": [],
"cross-providers-deps": [
"apache.hive",
"common.sql"
@@ -105,6 +121,7 @@
"apache-airflow>=2.6.0",
"cryptography>=2.0.0"
],
+ "devel-deps": [],
"cross-providers-deps": [
"cncf.kubernetes"
],
@@ -116,6 +133,7 @@
"apache-airflow>=2.6.0",
"hdfs[avro,dataframe,kerberos]>=2.0.4"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -126,9 +144,10 @@
"apache-airflow>=2.6.0",
"hmsclient>=0.1.0",
"pandas>=1.2.5",
- "pyhive[hive_pure_sasl]>=0.7.0",
+ "pyhive[hive-pure-sasl]>=0.7.0",
"thrift>=0.9.2"
],
+ "devel-deps": [],
"cross-providers-deps": [
"amazon",
"common.sql",
@@ -146,6 +165,7 @@
"apache-airflow>=2.6.0",
"impyla>=0.18.0,<1.0"
],
+ "devel-deps": [],
"cross-providers-deps": [
"common.sql"
],
@@ -158,6 +178,7 @@
"asgiref",
"confluent-kafka>=1.8.2"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -167,6 +188,7 @@
"apache-airflow>=2.6.0",
"kylinpy>=2.6"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -178,6 +200,7 @@
"apache-airflow>=2.6.0",
"asgiref"
],
+ "devel-deps": [],
"cross-providers-deps": [
"http"
],
@@ -188,6 +211,7 @@
"deps": [
"apache-airflow>=2.6.0"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -198,6 +222,7 @@
"apache-airflow>=2.6.0",
"pinotdb>0.4.7"
],
+ "devel-deps": [],
"cross-providers-deps": [
"common.sql"
],
@@ -210,6 +235,7 @@
"grpcio-status>=1.59.0",
"pyspark"
],
+ "devel-deps": [],
"cross-providers-deps": [
"cncf.kubernetes"
],
@@ -220,6 +246,7 @@
"deps": [
"apache-airflow>=2.6.0"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "suspended"
@@ -229,6 +256,7 @@
"apache-airflow>=2.6.0",
"apprise"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -238,6 +266,7 @@
"apache-airflow>=2.6.0",
"python-arango>=7.3.2"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -247,6 +276,7 @@
"apache-airflow>=2.6.0",
"asana>=0.10,<4.0.0"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -256,6 +286,7 @@
"apache-airflow>=2.6.0",
"atlassian-python-api>=1.14.2"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -267,6 +298,7 @@
"flower>=1.0.0",
"google-re2>=1.0"
],
+ "devel-deps": [],
"cross-providers-deps": [
"cncf.kubernetes"
],
@@ -278,6 +310,7 @@
"apache-airflow>=2.6.0",
"cloudant>=2.0"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -292,6 +325,7 @@
"kubernetes>=21.7.0,<24",
"kubernetes_asyncio>=18.20.1,<25"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -301,6 +335,7 @@
"apache-airflow>=2.6.0",
"cohere>=4.37"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -309,6 +344,7 @@
"deps": [
"apache-airflow>=2.8.0"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -318,6 +354,7 @@
"apache-airflow>=2.6.0",
"sqlparse>=0.4.2"
],
+ "devel-deps": [],
"cross-providers-deps": [
"openlineage"
],
@@ -331,6 +368,7 @@
"dask>=2.9.0,!=2022.10.1,!=2023.5.0",
"distributed>=2.11.1,!=2023.5.0"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "suspended"
@@ -343,6 +381,9 @@
"databricks-sql-connector>=2.0.0, <3.0.0, !=2.9.0",
"requests>=2.27,<3"
],
+ "devel-deps": [
+ "deltalake>=0.12.0"
+ ],
"cross-providers-deps": [
"common.sql"
],
@@ -354,6 +395,7 @@
"apache-airflow>=2.6.0",
"datadog>=0.14.0"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -365,6 +407,7 @@
"apache-airflow>=2.6.0",
"asgiref"
],
+ "devel-deps": [],
"cross-providers-deps": [
"http",
"openlineage"
@@ -377,6 +420,7 @@
"apache-airflow-providers-http",
"apache-airflow>=2.6.0"
],
+ "devel-deps": [],
"cross-providers-deps": [
"http"
],
@@ -388,6 +432,7 @@
"apache-airflow-providers-http",
"apache-airflow>=2.6.0"
],
+ "devel-deps": [],
"cross-providers-deps": [
"http"
],
@@ -400,6 +445,7 @@
"docker>=5.0.3",
"python-dotenv>=0.21.0"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -410,6 +456,7 @@
"apache-airflow>=2.6.0",
"elasticsearch>=8.10,<9"
],
+ "devel-deps": [],
"cross-providers-deps": [
"common.sql"
],
@@ -423,6 +470,7 @@
"pandas>=1.2.5",
"pyexasol>=0.5.1"
],
+ "devel-deps": [],
"cross-providers-deps": [
"common.sql"
],
@@ -434,6 +482,7 @@
"apache-airflow>=2.6.0",
"facebook-business>=6.0.2"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -442,6 +491,7 @@
"deps": [
"apache-airflow>=2.6.0"
],
+ "devel-deps": [],
"cross-providers-deps": [
"openlineage"
],
@@ -453,6 +503,7 @@
"PyGithub!=1.58",
"apache-airflow>=2.6.0"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -519,6 +570,7 @@
"sqlalchemy-bigquery>=1.2.1",
"sqlalchemy-spanner>=1.6.2"
],
+ "devel-deps": [],
"cross-providers-deps": [
"amazon",
"apache.beam",
@@ -548,6 +600,7 @@
"google-auth>=1.0.0, <3.0.0",
"grpcio>=1.15.0"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -557,6 +610,7 @@
"apache-airflow>=2.6.0",
"hvac>=1.1.0"
],
+ "devel-deps": [],
"cross-providers-deps": [
"google"
],
@@ -571,6 +625,7 @@
"requests>=2.26.0",
"requests_toolbelt"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -579,6 +634,7 @@
"deps": [
"apache-airflow>=2.6.0"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -589,6 +645,7 @@
"influxdb-client>=1.19.0",
"requests>=2.26.0"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -599,6 +656,7 @@
"apache-airflow>=2.6.0",
"jaydebeapi>=1.1.1"
],
+ "devel-deps": [],
"cross-providers-deps": [
"common.sql"
],
@@ -610,6 +668,7 @@
"apache-airflow>=2.6.0",
"python-jenkins>=1.0.0"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -639,6 +698,9 @@
"azure-synapse-artifacts>=0.17.0",
"azure-synapse-spark"
],
+ "devel-deps": [
+ "pywinrm"
+ ],
"cross-providers-deps": [
"google",
"oracle",
@@ -653,6 +715,7 @@
"apache-airflow>=2.6.0",
"pymssql>=2.1.8"
],
+ "devel-deps": [],
"cross-providers-deps": [
"common.sql"
],
@@ -664,6 +727,7 @@
"apache-airflow>=2.6.0",
"pypsrp>=0.8.0"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -673,6 +737,7 @@
"apache-airflow>=2.6.0",
"pywinrm>=0.4"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -683,6 +748,9 @@
"dnspython>=1.13.0",
"pymongo>=3.6.0"
],
+ "devel-deps": [
+ "mongomock"
+ ],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -694,6 +762,7 @@
"mysql-connector-python>=8.0.29",
"mysqlclient>=1.3.6"
],
+ "devel-deps": [],
"cross-providers-deps": [
"amazon",
"common.sql",
@@ -710,6 +779,7 @@
"apache-airflow>=2.6.0",
"neo4j>=4.2.1"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -720,6 +790,7 @@
"apache-airflow>=2.6.0",
"pyodbc"
],
+ "devel-deps": [],
"cross-providers-deps": [
"common.sql"
],
@@ -731,6 +802,7 @@
"apache-airflow>=2.6.0",
"openai[datalib]>=1.0"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -739,6 +811,7 @@
"deps": [
"apache-airflow>=2.6.0"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -751,6 +824,7 @@
"openlineage-integration-common>=0.28.0",
"openlineage-python>=0.28.0"
],
+ "devel-deps": [],
"cross-providers-deps": [
"common.sql"
],
@@ -762,6 +836,7 @@
"apache-airflow>=2.6.0",
"opensearch-py>=2.2.0"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -771,6 +846,7 @@
"apache-airflow>=2.6.0",
"opsgenie-sdk>=2.1.5"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -781,6 +857,7 @@
"apache-airflow>=2.6.0",
"oracledb>=1.0.0"
],
+ "devel-deps": [],
"cross-providers-deps": [
"common.sql"
],
@@ -792,6 +869,7 @@
"apache-airflow>=2.6.0",
"pdpyras>=4.1.2"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -803,6 +881,7 @@
"papermill[all]>=2.4.0",
"scrapbook[all]"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -813,6 +892,7 @@
"apache-airflow>=2.6.0",
"pgvector>=0.2.3"
],
+ "devel-deps": [],
"cross-providers-deps": [
"common.sql",
"postgres"
@@ -825,6 +905,7 @@
"apache-airflow>=2.6.0",
"pinecone-client>=2.2.4"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -834,6 +915,7 @@
"apache-airflow>=2.6.0",
"arrow>=0.16.0"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "suspended"
@@ -844,6 +926,7 @@
"apache-airflow>=2.6.0",
"psycopg2-binary>=2.8.0"
],
+ "devel-deps": [],
"cross-providers-deps": [
"amazon",
"common.sql",
@@ -859,6 +942,7 @@
"pandas>=1.2.5",
"presto-python-client>=0.8.4"
],
+ "devel-deps": [],
"cross-providers-deps": [
"common.sql",
"google"
@@ -871,6 +955,7 @@
"apache-airflow>=2.6.0",
"redis>=4.5.2,<5.0.0,!=4.5.5"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -881,6 +966,7 @@
"pandas>=1.2.5",
"simple-salesforce>=1.0.0"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -890,6 +976,7 @@
"apache-airflow>=2.6.0",
"smbprotocol>=1.5.0"
],
+ "devel-deps": [],
"cross-providers-deps": [
"google"
],
@@ -901,6 +988,7 @@
"analytics-python>=1.2.9",
"apache-airflow>=2.6.0"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -910,6 +998,7 @@
"apache-airflow>=2.6.0",
"sendgrid>=6.0.0"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -920,6 +1009,7 @@
"apache-airflow>=2.6.0",
"paramiko>=2.8.0"
],
+ "devel-deps": [],
"cross-providers-deps": [
"openlineage",
"ssh"
@@ -932,6 +1022,7 @@
"apache-airflow>=2.6.0",
"spython>=0.0.56"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -942,6 +1033,7 @@
"apache-airflow>=2.6.0",
"slack_sdk>=3.0.0"
],
+ "devel-deps": [],
"cross-providers-deps": [
"common.sql"
],
@@ -952,6 +1044,7 @@
"deps": [
"apache-airflow>=2.6.0"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -963,6 +1056,7 @@
"snowflake-connector-python>=2.7.8",
"snowflake-sqlalchemy>=1.1.0"
],
+ "devel-deps": [],
"cross-providers-deps": [
"common.sql",
"openlineage"
@@ -975,6 +1069,7 @@
"apache-airflow-providers-common-sql>=1.3.1",
"apache-airflow>=2.6.0"
],
+ "devel-deps": [],
"cross-providers-deps": [
"common.sql"
],
@@ -987,6 +1082,7 @@
"paramiko>=2.6.0",
"sshtunnel>=0.3.2"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -996,6 +1092,7 @@
"apache-airflow>=2.6.0",
"tableauserverclient"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -1004,6 +1101,9 @@
"deps": [
"apache-airflow>=2.6.0"
],
+ "devel-deps": [
+ "pyiceberg>=0.5.0"
+ ],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -1013,6 +1113,7 @@
"apache-airflow>=2.6.0",
"python-telegram-bot>=20.2"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -1024,6 +1125,7 @@
"pandas>=1.2.5",
"trino>=0.318.0"
],
+ "devel-deps": [],
"cross-providers-deps": [
"common.sql",
"google",
@@ -1038,6 +1140,7 @@
"apache-airflow>=2.6.0",
"vertica-python>=0.5.1"
],
+ "devel-deps": [],
"cross-providers-deps": [
"common.sql"
],
@@ -1050,6 +1153,7 @@
"pandas>=1.2.5",
"weaviate-client>=3.24.2"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -1059,6 +1163,7 @@
"apache-airflow>=2.6.0",
"yandexcloud>=0.228.0"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
@@ -1068,6 +1173,7 @@
"apache-airflow>=2.6.0",
"zenpy>=2.0.24"
],
+ "devel-deps": [],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "ready"
diff --git a/images/breeze/output_ci-image_build.svg b/images/breeze/output_ci-image_build.svg
index 70960aa673..b6d63b0465 100644
--- a/images/breeze/output_ci-image_build.svg
+++ b/images/breeze/output_ci-image_build.svg
@@ -1,4 +1,4 @@
-<svg class="rich-terminal" viewBox="0 0 1482 2270.4" xmlns="http://www.w3.org/2000/svg">
+<svg class="rich-terminal" viewBox="0 0 1482 2246.0" xmlns="http://www.w3.org/2000/svg">
<!-- Generated with Rich https://www.textualize.io -->
<style>
@@ -43,7 +43,7 @@
<defs>
<clipPath id="breeze-ci-image-build-clip-terminal">
- <rect x="0" y="0" width="1463.0" height="2219.4" />
+ <rect x="0" y="0" width="1463.0" height="2195.0" />
</clipPath>
<clipPath id="breeze-ci-image-build-line-0">
<rect x="0" y="1.5" width="1464" height="24.65"/>
@@ -312,12 +312,9 @@
<clipPath id="breeze-ci-image-build-line-88">
<rect x="0" y="2148.7" width="1464" height="24.65"/>
</clipPath>
-<clipPath id="breeze-ci-image-build-line-89">
- <rect x="0" y="2173.1" width="1464" height="24.65"/>
- </clipPath>
</defs>
- <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="2268.4" rx="8"/><text class="breeze-ci-image-build-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: ci-image build</text>
+ <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="2244" rx="8"/><text class="breeze-ci-image-build-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: ci-image build</text>
<g transform="translate(26,22)">
<circle cx="0" cy="0" r="7" fill="#ff5f57"/>
<circle cx="22" cy="0" r="7" fill="#febc2e"/>
@@ -363,61 +360,60 @@
</text><text class="breeze-ci-image-build-r5" x="0" y="825.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-33)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="825.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-33)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="825.2" textLength="61" clip-path="url(#breeze-ci-image-build-line-33)">-skip</text><text class="breeze-ci-image-build-r4" x="97.6" y="825.2" textLength="97.6" clip-path="url [...]
</text><text class="breeze-ci-image-build-r5" x="0" y="849.6" textLength="1464" clip-path="url(#breeze-ci-image-build-line-34)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-build-r1" x="1464" y="849.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-34)">
</text><text class="breeze-ci-image-build-r5" x="0" y="874" textLength="24.4" clip-path="url(#breeze-ci-image-build-line-35)">╭─</text><text class="breeze-ci-image-build-r5" x="24.4" y="874" textLength="512.4" clip-path="url(#breeze-ci-image-build-line-35)"> Advanced build options (for power users) </text><text class="breeze-ci-image-build-r5" x="536.8" y="874" textLength="902.8" clip-path="url(#breeze-ci-image-build-line-35)">────────────────────────── [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="898.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-36)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="898.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-36)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="898.4" textLength="134.2" clip-path="url(#breeze-ci-image-build-line-36)">-additional</text><text class="breeze-ci-image-build-r4" x="170.8" y="898.4" textLength="219.6" cli [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="922.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-37)">│</text><text class="breeze-ci-image-build-r1" x="463.6" y="922.8" textLength="976" clip-path="url(#breeze-ci-image-build-line-37)">itself).                                     &# [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="947.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-38)">│</text><text class="breeze-ci-image-build-r7" x="463.6" y="947.2" textLength="976" clip-path="url(#breeze-ci-image-build-line-38)">(TEXT)                                      [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="971.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-39)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="971.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-39)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="971.6" textLength="85.4" clip-path="url(#breeze-ci-image-build-line-39)">-commit</text><text class="breeze-ci-image-build-r4" x="122" y="971.6" textLength="48.8" clip-path=" [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="996" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-40)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="996" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-40)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="996" textLength="85.4" clip-path="url(#breeze-ci-image-build-line-40)">-debian</text><text class="breeze-ci-image-build-r4" x="122" y="996" textLength="97.6" clip-path="url(#bre [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1020.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-41)">│</text><text class="breeze-ci-image-build-r7" x="463.6" y="1020.4" textLength="793" clip-path="url(#breeze-ci-image-build-line-41)">(bookworm | bullseye)                                 & [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1044.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-42)">│</text><text class="breeze-ci-image-build-r5" x="463.6" y="1044.8" textLength="793" clip-path="url(#breeze-ci-image-build-line-42)">[default: bookworm]                                  &# [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1069.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-43)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1069.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-43)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1069.2" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-43)">-install</text><text class="breeze-ci-image-build-r4" x="134.2" y="1069.2" textLength="219.6" cli [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1093.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-44)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1093.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-44)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1093.6" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-44)">-install</text><text class="breeze-ci-image-build-r4" x="134.2" y="1093.6" textLength="280.6" cli [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1118" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-45)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1118" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-45)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1118" textLength="85.4" clip-path="url(#breeze-ci-image-build-line-45)">-python</text><text class="breeze-ci-image-build-r4" x="122" y="1118" textLength="73.2" clip-path="url( [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1142.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-46)">│</text><text class="breeze-ci-image-build-r1" x="463.6" y="1142.4" textLength="976" clip-path="url(#breeze-ci-image-build-line-46)">something like: python:VERSION-slim-bookworm.                             & [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1166.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-47)">│</text><text class="breeze-ci-image-build-r7" x="463.6" y="1166.8" textLength="976" clip-path="url(#breeze-ci-image-build-line-47)">(TEXT)                                     &# [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1191.2" textLength="1464" clip-path="url(#breeze-ci-image-build-line-48)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-build-r1" x="1464" y="1191.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-48)">
-</text><text class="breeze-ci-image-build-r5" x="0" y="1215.6" textLength="24.4" clip-path="url(#breeze-ci-image-build-line-49)">╭─</text><text class="breeze-ci-image-build-r5" x="24.4" y="1215.6" textLength="597.8" clip-path="url(#breeze-ci-image-build-line-49)"> Selecting constraint location (for power users) </text><text class="breeze-ci-image-build-r5" x="622.2" y="1215.6" textLength="817.4" clip-path="url(#breeze-ci-image-build-line-49)">────────── [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1240" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-50)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1240" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-50)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1240" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-50)">-airflow</text><text class="breeze-ci-image-build-r4" x="134.2" y="1240" textLength="256.2" clip-path=" [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1264.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-51)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1264.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-51)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1264.4" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-51)">-airflow</text><text class="breeze-ci-image-build-r4" x="134.2" y="1264.4" textLength="207.4" cli [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1288.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-52)">│</text><text class="breeze-ci-image-build-r7" x="451.4" y="1288.8" textLength="866.2" clip-path="url(#breeze-ci-image-build-line-52)">(constraints-source-providers | constraints | constraints-no-providers)</text><text class="breeze-ci-image-build-r5" x="1451.8" y="1288.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-52)">│</ [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1313.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-53)">│</text><text class="breeze-ci-image-build-r5" x="451.4" y="1313.2" textLength="866.2" clip-path="url(#breeze-ci-image-build-line-53)">[default: constraints-source-providers]                               [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1337.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-54)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1337.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-54)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1337.6" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-54)">-airflow</text><text class="breeze-ci-image-build-r4" x="134.2" y="1337.6" textLength="268.4" cli [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1362" textLength="1464" clip-path="url(#breeze-ci-image-build-line-55)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-build-r1" x="1464" y="1362" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-55)">
-</text><text class="breeze-ci-image-build-r5" x="0" y="1386.4" textLength="24.4" clip-path="url(#breeze-ci-image-build-line-56)">╭─</text><text class="breeze-ci-image-build-r5" x="24.4" y="1386.4" textLength="634.4" clip-path="url(#breeze-ci-image-build-line-56)"> Choosing dependencies and extras (for power users) </text><text class="breeze-ci-image-build-r5" x="658.8" y="1386.4" textLength="780.8" clip-path="url(#breeze-ci-image-build-line-56)">── [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1410.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-57)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1410.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-57)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1410.8" textLength="134.2" clip-path="url(#breeze-ci-image-build-line-57)">-additional</text><text class="breeze-ci-image-build-r4" x="170.8" y="1410.8" textLength="183" c [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1435.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-58)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1435.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-58)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1435.2" textLength="134.2" clip-path="url(#breeze-ci-image-build-line-58)">-additional</text><text class="breeze-ci-image-build-r4" x="170.8" y="1435.2" textLength="146.4" [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1459.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-59)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1459.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-59)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1459.6" textLength="48.8" clip-path="url(#breeze-ci-image-build-line-59)">-dev</text><text class="breeze-ci-image-build-r4" x="85.4" y="1459.6" textLength="109.8" clip-pat [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1484" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-60)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1484" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-60)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1484" textLength="134.2" clip-path="url(#breeze-ci-image-build-line-60)">-additional</text><text class="breeze-ci-image-build-r4" x="170.8" y="1484" textLength="158.6" clip-pa [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1508.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-61)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1508.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-61)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1508.4" textLength="48.8" clip-path="url(#breeze-ci-image-build-line-61)">-dev</text><text class="breeze-ci-image-build-r4" x="85.4" y="1508.4" textLength="146.4" clip-pat [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1532.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-62)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1532.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-62)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1532.8" textLength="134.2" clip-path="url(#breeze-ci-image-build-line-62)">-additional</text><text class="breeze-ci-image-build-r4" x="170.8" y="1532.8" textLength="195.2" [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1557.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-63)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1557.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-63)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1557.2" textLength="134.2" clip-path="url(#breeze-ci-image-build-line-63)">-additional</text><text class="breeze-ci-image-build-r4" x="170.8" y="1557.2" textLength="146.4" [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1581.6" textLength="1464" clip-path="url(#breeze-ci-image-build-line-64)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-build-r1" x="1464" y="1581.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-64)">
-</text><text class="breeze-ci-image-build-r5" x="0" y="1606" textLength="24.4" clip-path="url(#breeze-ci-image-build-line-65)">╭─</text><text class="breeze-ci-image-build-r5" x="24.4" y="1606" textLength="268.4" clip-path="url(#breeze-ci-image-build-line-65)"> Backtracking options </text><text class="breeze-ci-image-build-r5" x="292.8" y="1606" textLength="1146.8" clip-path="url(#breeze-ci-image-build-line-65)">────────────────────────────────────────────────────────────── [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1630.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-66)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1630.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-66)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1630.4" textLength="73.2" clip-path="url(#breeze-ci-image-build-line-66)">-build</text><text class="breeze-ci-image-build-r4" x="109.8" y="1630.4" textLength="195.2" clip- [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1654.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-67)">│</text><text class="breeze-ci-image-build-r1" x="549" y="1654.8" textLength="890.6" clip-path="url(#breeze-ci-image-build-line-67)">backtracking problems.                                   [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1679.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-68)">│</text><text class="breeze-ci-image-build-r7" x="549" y="1679.2" textLength="890.6" clip-path="url(#breeze-ci-image-build-line-68)">(INTEGER)                                      [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1703.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-69)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1703.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-69)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1703.6" textLength="73.2" clip-path="url(#breeze-ci-image-build-line-69)">-eager</text><text class="breeze-ci-image-build-r4" x="109.8" y="1703.6" textLength="390.4" clip- [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1728" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-70)">│</text><text class="breeze-ci-image-build-r1" x="549" y="1728" textLength="890.6" clip-path="url(#breeze-ci-image-build-line-70)">(see `breeze ci find-backtracking-candidates`).                          </text><text cla [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1752.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-71)">│</text><text class="breeze-ci-image-build-r7" x="549" y="1752.4" textLength="890.6" clip-path="url(#breeze-ci-image-build-line-71)">(TEXT)                                     &# [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1776.8" textLength="1464" clip-path="url(#breeze-ci-image-build-line-72)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-build-r1" x="1464" y="1776.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-72)">
-</text><text class="breeze-ci-image-build-r5" x="0" y="1801.2" textLength="24.4" clip-path="url(#breeze-ci-image-build-line-73)">╭─</text><text class="breeze-ci-image-build-r5" x="24.4" y="1801.2" textLength="622.2" clip-path="url(#breeze-ci-image-build-line-73)"> Preparing cache and push (for maintainers and CI) </text><text class="breeze-ci-image-build-r5" x="646.6" y="1801.2" textLength="793" clip-path="url(#breeze-ci-image-build-line-73)"> [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1825.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-74)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1825.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-74)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1825.6" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-74)">-builder</text><text class="breeze-ci-image-build-r1" x="341.6" y="1825.6" textLength="756.4" cli [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1850" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-75)">│</text><text class="breeze-ci-image-build-r5" x="341.6" y="1850" textLength="756.4" clip-path="url(#breeze-ci-image-build-line-75)">[default: autodetect]                                  &# [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1874.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-76)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1874.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-76)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1874.4" textLength="109.8" clip-path="url(#breeze-ci-image-build-line-76)">-platform</text><text class="breeze-ci-image-build-r1" x="341.6" y="1874.4" textLength="329.4" c [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1898.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-77)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1898.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-77)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1898.8" textLength="61" clip-path="url(#breeze-ci-image-build-line-77)">-push</text><text class="breeze-ci-image-build-r1" x="341.6" y="1898.8" textLength="353.8" clip-pat [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1923.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-78)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1923.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-78)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1923.2" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-78)">-prepare</text><text class="breeze-ci-image-build-r4" x="134.2" y="1923.2" textLength="158.6" cli [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1947.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-79)">│</text><text class="breeze-ci-image-build-r1" x="341.6" y="1947.6" textLength="1098" clip-path="url(#breeze-ci-image-build-line-79)">image).                                      [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1972" textLength="1464" clip-path="url(#breeze-ci-image-build-line-80)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-build-r1" x="1464" y="1972" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-80)">
-</text><text class="breeze-ci-image-build-r5" x="0" y="1996.4" textLength="24.4" clip-path="url(#breeze-ci-image-build-line-81)">╭─</text><text class="breeze-ci-image-build-r5" x="24.4" y="1996.4" textLength="280.6" clip-path="url(#breeze-ci-image-build-line-81)"> Github authentication </text><text class="breeze-ci-image-build-r5" x="305" y="1996.4" textLength="1134.6" clip-path="url(#breeze-ci-image-build-line-81)">───────────────────────────────────────────────────────── [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="2020.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-82)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="2020.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-82)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="2020.8" textLength="85.4" clip-path="url(#breeze-ci-image-build-line-82)">-github</text><text class="breeze-ci-image-build-r4" x="122" y="2020.8" textLength="134.2" clip-p [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="2045.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-83)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="2045.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-83)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="2045.2" textLength="85.4" clip-path="url(#breeze-ci-image-build-line-83)">-github</text><text class="breeze-ci-image-build-r4" x="122" y="2045.2" textLength="73.2" clip-pa [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="2069.6" textLength="1464" clip-path="url(#breeze-ci-image-build-line-84)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-build-r1" x="1464" y="2069.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-84)">
-</text><text class="breeze-ci-image-build-r5" x="0" y="2094" textLength="24.4" clip-path="url(#breeze-ci-image-build-line-85)">╭─</text><text class="breeze-ci-image-build-r5" x="24.4" y="2094" textLength="195.2" clip-path="url(#breeze-ci-image-build-line-85)"> Common options </text><text class="breeze-ci-image-build-r5" x="219.6" y="2094" textLength="1220" clip-path="url(#breeze-ci-image-build-line-85)">────────────────────────────────────────────────────────────────────── [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="2118.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-86)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="2118.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-86)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="2118.4" textLength="85.4" clip-path="url(#breeze-ci-image-build-line-86)">-answer</text><text class="breeze-ci-image-build-r6" x="158.6" y="2118.4" textLength="24.4" clip- [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="2142.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-87)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="2142.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-87)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="2142.8" textLength="48.8" clip-path="url(#breeze-ci-image-build-line-87)">-dry</text><text class="breeze-ci-image-build-r4" x="85.4" y="2142.8" textLength="48.8" clip-path [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="2167.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-88)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="2167.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-88)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="2167.2" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-88)">-verbose</text><text class="breeze-ci-image-build-r6" x="158.6" y="2167.2" textLength="24.4" clip [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="2191.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-89)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="2191.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-89)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="2191.6" textLength="61" clip-path="url(#breeze-ci-image-build-line-89)">-help</text><text class="breeze-ci-image-build-r6" x="158.6" y="2191.6" textLength="24.4" clip-path [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="2216" textLength="1464" clip-path="url(#breeze-ci-image-build-line-90)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-build-r1" x="1464" y="2216" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-90)">
+</text><text class="breeze-ci-image-build-r5" x="0" y="898.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-36)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="898.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-36)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="898.4" textLength="134.2" clip-path="url(#breeze-ci-image-build-line-36)">-additional</text><text class="breeze-ci-image-build-r4" x="170.8" y="898.4" textLength="219.6" cli [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="922.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-37)">│</text><text class="breeze-ci-image-build-r1" x="439.2" y="922.8" textLength="1000.4" clip-path="url(#breeze-ci-image-build-line-37)">itself).                                      [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="947.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-38)">│</text><text class="breeze-ci-image-build-r7" x="439.2" y="947.2" textLength="1000.4" clip-path="url(#breeze-ci-image-build-line-38)">(TEXT)                                     & [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="971.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-39)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="971.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-39)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="971.6" textLength="85.4" clip-path="url(#breeze-ci-image-build-line-39)">-commit</text><text class="breeze-ci-image-build-r4" x="122" y="971.6" textLength="48.8" clip-path=" [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="996" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-40)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="996" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-40)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="996" textLength="85.4" clip-path="url(#breeze-ci-image-build-line-40)">-debian</text><text class="breeze-ci-image-build-r4" x="122" y="996" textLength="97.6" clip-path="url(#bre [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1020.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-41)">│</text><text class="breeze-ci-image-build-r7" x="439.2" y="1020.4" textLength="793" clip-path="url(#breeze-ci-image-build-line-41)">(bookworm | bullseye)                                 & [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1044.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-42)">│</text><text class="breeze-ci-image-build-r5" x="439.2" y="1044.8" textLength="793" clip-path="url(#breeze-ci-image-build-line-42)">[default: bookworm]                                  &# [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1069.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-43)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1069.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-43)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1069.2" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-43)">-install</text><text class="breeze-ci-image-build-r4" x="134.2" y="1069.2" textLength="219.6" cli [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1093.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-44)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1093.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-44)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1093.6" textLength="85.4" clip-path="url(#breeze-ci-image-build-line-44)">-python</text><text class="breeze-ci-image-build-r4" x="122" y="1093.6" textLength="73.2" clip-pa [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1118" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-45)">│</text><text class="breeze-ci-image-build-r1" x="439.2" y="1118" textLength="1000.4" clip-path="url(#breeze-ci-image-build-line-45)">something like: python:VERSION-slim-bookworm.                             &# [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1142.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-46)">│</text><text class="breeze-ci-image-build-r7" x="439.2" y="1142.4" textLength="1000.4" clip-path="url(#breeze-ci-image-build-line-46)">(TEXT)                                      [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1166.8" textLength="1464" clip-path="url(#breeze-ci-image-build-line-47)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-build-r1" x="1464" y="1166.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-47)">
+</text><text class="breeze-ci-image-build-r5" x="0" y="1191.2" textLength="24.4" clip-path="url(#breeze-ci-image-build-line-48)">╭─</text><text class="breeze-ci-image-build-r5" x="24.4" y="1191.2" textLength="597.8" clip-path="url(#breeze-ci-image-build-line-48)"> Selecting constraint location (for power users) </text><text class="breeze-ci-image-build-r5" x="622.2" y="1191.2" textLength="817.4" clip-path="url(#breeze-ci-image-build-line-48)">────────── [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1215.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-49)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1215.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-49)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1215.6" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-49)">-airflow</text><text class="breeze-ci-image-build-r4" x="134.2" y="1215.6" textLength="256.2" cli [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1240" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-50)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1240" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-50)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1240" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-50)">-airflow</text><text class="breeze-ci-image-build-r4" x="134.2" y="1240" textLength="207.4" clip-path=" [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1264.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-51)">│</text><text class="breeze-ci-image-build-r7" x="451.4" y="1264.4" textLength="866.2" clip-path="url(#breeze-ci-image-build-line-51)">(constraints-source-providers | constraints | constraints-no-providers)</text><text class="breeze-ci-image-build-r5" x="1451.8" y="1264.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-51)">│</ [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1288.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-52)">│</text><text class="breeze-ci-image-build-r5" x="451.4" y="1288.8" textLength="866.2" clip-path="url(#breeze-ci-image-build-line-52)">[default: constraints-source-providers]                               [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1313.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-53)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1313.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-53)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1313.2" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-53)">-airflow</text><text class="breeze-ci-image-build-r4" x="134.2" y="1313.2" textLength="268.4" cli [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1337.6" textLength="1464" clip-path="url(#breeze-ci-image-build-line-54)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-build-r1" x="1464" y="1337.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-54)">
+</text><text class="breeze-ci-image-build-r5" x="0" y="1362" textLength="24.4" clip-path="url(#breeze-ci-image-build-line-55)">╭─</text><text class="breeze-ci-image-build-r5" x="24.4" y="1362" textLength="634.4" clip-path="url(#breeze-ci-image-build-line-55)"> Choosing dependencies and extras (for power users) </text><text class="breeze-ci-image-build-r5" x="658.8" y="1362" textLength="780.8" clip-path="url(#breeze-ci-image-build-line-55)">──────── [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1386.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-56)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1386.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-56)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1386.4" textLength="134.2" clip-path="url(#breeze-ci-image-build-line-56)">-additional</text><text class="breeze-ci-image-build-r4" x="170.8" y="1386.4" textLength="183" c [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1410.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-57)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1410.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-57)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1410.8" textLength="134.2" clip-path="url(#breeze-ci-image-build-line-57)">-additional</text><text class="breeze-ci-image-build-r4" x="170.8" y="1410.8" textLength="146.4" [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1435.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-58)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1435.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-58)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1435.2" textLength="48.8" clip-path="url(#breeze-ci-image-build-line-58)">-dev</text><text class="breeze-ci-image-build-r4" x="85.4" y="1435.2" textLength="109.8" clip-pat [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1459.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-59)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1459.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-59)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1459.6" textLength="134.2" clip-path="url(#breeze-ci-image-build-line-59)">-additional</text><text class="breeze-ci-image-build-r4" x="170.8" y="1459.6" textLength="158.6" [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1484" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-60)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1484" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-60)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1484" textLength="48.8" clip-path="url(#breeze-ci-image-build-line-60)">-dev</text><text class="breeze-ci-image-build-r4" x="85.4" y="1484" textLength="146.4" clip-path="url(# [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1508.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-61)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1508.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-61)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1508.4" textLength="134.2" clip-path="url(#breeze-ci-image-build-line-61)">-additional</text><text class="breeze-ci-image-build-r4" x="170.8" y="1508.4" textLength="195.2" [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1532.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-62)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1532.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-62)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1532.8" textLength="134.2" clip-path="url(#breeze-ci-image-build-line-62)">-additional</text><text class="breeze-ci-image-build-r4" x="170.8" y="1532.8" textLength="146.4" [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1557.2" textLength="1464" clip-path="url(#breeze-ci-image-build-line-63)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-build-r1" x="1464" y="1557.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-63)">
+</text><text class="breeze-ci-image-build-r5" x="0" y="1581.6" textLength="24.4" clip-path="url(#breeze-ci-image-build-line-64)">╭─</text><text class="breeze-ci-image-build-r5" x="24.4" y="1581.6" textLength="268.4" clip-path="url(#breeze-ci-image-build-line-64)"> Backtracking options </text><text class="breeze-ci-image-build-r5" x="292.8" y="1581.6" textLength="1146.8" clip-path="url(#breeze-ci-image-build-line-64)">──────────────────────────────────────────────────────── [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1606" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-65)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1606" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-65)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1606" textLength="73.2" clip-path="url(#breeze-ci-image-build-line-65)">-build</text><text class="breeze-ci-image-build-r4" x="109.8" y="1606" textLength="195.2" clip-path="ur [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1630.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-66)">│</text><text class="breeze-ci-image-build-r1" x="549" y="1630.4" textLength="890.6" clip-path="url(#breeze-ci-image-build-line-66)">backtracking problems.                                   [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1654.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-67)">│</text><text class="breeze-ci-image-build-r7" x="549" y="1654.8" textLength="890.6" clip-path="url(#breeze-ci-image-build-line-67)">(INTEGER)                                      [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1679.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-68)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1679.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-68)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1679.2" textLength="73.2" clip-path="url(#breeze-ci-image-build-line-68)">-eager</text><text class="breeze-ci-image-build-r4" x="109.8" y="1679.2" textLength="390.4" clip- [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1703.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-69)">│</text><text class="breeze-ci-image-build-r1" x="549" y="1703.6" textLength="890.6" clip-path="url(#breeze-ci-image-build-line-69)">(see `breeze ci find-backtracking-candidates`).                          </text><text [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1728" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-70)">│</text><text class="breeze-ci-image-build-r7" x="549" y="1728" textLength="890.6" clip-path="url(#breeze-ci-image-build-line-70)">(TEXT)                                       [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1752.4" textLength="1464" clip-path="url(#breeze-ci-image-build-line-71)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-build-r1" x="1464" y="1752.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-71)">
+</text><text class="breeze-ci-image-build-r5" x="0" y="1776.8" textLength="24.4" clip-path="url(#breeze-ci-image-build-line-72)">╭─</text><text class="breeze-ci-image-build-r5" x="24.4" y="1776.8" textLength="622.2" clip-path="url(#breeze-ci-image-build-line-72)"> Preparing cache and push (for maintainers and CI) </text><text class="breeze-ci-image-build-r5" x="646.6" y="1776.8" textLength="793" clip-path="url(#breeze-ci-image-build-line-72)"> [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1801.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-73)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1801.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-73)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1801.2" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-73)">-builder</text><text class="breeze-ci-image-build-r1" x="341.6" y="1801.2" textLength="756.4" cli [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1825.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-74)">│</text><text class="breeze-ci-image-build-r5" x="341.6" y="1825.6" textLength="756.4" clip-path="url(#breeze-ci-image-build-line-74)">[default: autodetect]                                  [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1850" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-75)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1850" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-75)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1850" textLength="109.8" clip-path="url(#breeze-ci-image-build-line-75)">-platform</text><text class="breeze-ci-image-build-r1" x="341.6" y="1850" textLength="329.4" clip-path [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1874.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-76)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1874.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-76)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1874.4" textLength="61" clip-path="url(#breeze-ci-image-build-line-76)">-push</text><text class="breeze-ci-image-build-r1" x="341.6" y="1874.4" textLength="353.8" clip-pat [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1898.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-77)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1898.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-77)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1898.8" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-77)">-prepare</text><text class="breeze-ci-image-build-r4" x="134.2" y="1898.8" textLength="158.6" cli [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1923.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-78)">│</text><text class="breeze-ci-image-build-r1" x="341.6" y="1923.2" textLength="1098" clip-path="url(#breeze-ci-image-build-line-78)">image).                                      [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1947.6" textLength="1464" clip-path="url(#breeze-ci-image-build-line-79)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-build-r1" x="1464" y="1947.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-79)">
+</text><text class="breeze-ci-image-build-r5" x="0" y="1972" textLength="24.4" clip-path="url(#breeze-ci-image-build-line-80)">╭─</text><text class="breeze-ci-image-build-r5" x="24.4" y="1972" textLength="280.6" clip-path="url(#breeze-ci-image-build-line-80)"> Github authentication </text><text class="breeze-ci-image-build-r5" x="305" y="1972" textLength="1134.6" clip-path="url(#breeze-ci-image-build-line-80)">─────────────────────────────────────────────────────────────── [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1996.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-81)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1996.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-81)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1996.4" textLength="85.4" clip-path="url(#breeze-ci-image-build-line-81)">-github</text><text class="breeze-ci-image-build-r4" x="122" y="1996.4" textLength="134.2" clip-p [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="2020.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-82)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="2020.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-82)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="2020.8" textLength="85.4" clip-path="url(#breeze-ci-image-build-line-82)">-github</text><text class="breeze-ci-image-build-r4" x="122" y="2020.8" textLength="73.2" clip-pa [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="2045.2" textLength="1464" clip-path="url(#breeze-ci-image-build-line-83)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-build-r1" x="1464" y="2045.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-83)">
+</text><text class="breeze-ci-image-build-r5" x="0" y="2069.6" textLength="24.4" clip-path="url(#breeze-ci-image-build-line-84)">╭─</text><text class="breeze-ci-image-build-r5" x="24.4" y="2069.6" textLength="195.2" clip-path="url(#breeze-ci-image-build-line-84)"> Common options </text><text class="breeze-ci-image-build-r5" x="219.6" y="2069.6" textLength="1220" clip-path="url(#breeze-ci-image-build-line-84)">──────────────────────────────────────────────────────────────── [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="2094" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-85)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="2094" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-85)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="2094" textLength="85.4" clip-path="url(#breeze-ci-image-build-line-85)">-answer</text><text class="breeze-ci-image-build-r6" x="158.6" y="2094" textLength="24.4" clip-path="ur [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="2118.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-86)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="2118.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-86)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="2118.4" textLength="48.8" clip-path="url(#breeze-ci-image-build-line-86)">-dry</text><text class="breeze-ci-image-build-r4" x="85.4" y="2118.4" textLength="48.8" clip-path [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="2142.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-87)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="2142.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-87)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="2142.8" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-87)">-verbose</text><text class="breeze-ci-image-build-r6" x="158.6" y="2142.8" textLength="24.4" clip [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="2167.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-88)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="2167.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-88)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="2167.2" textLength="61" clip-path="url(#breeze-ci-image-build-line-88)">-help</text><text class="breeze-ci-image-build-r6" x="158.6" y="2167.2" textLength="24.4" clip-path [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="2191.6" textLength="1464" clip-path="url(#breeze-ci-image-build-line-89)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-build-r1" x="1464" y="2191.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-89)">
</text>
</g>
</g>
diff --git a/images/breeze/output_ci-image_build.txt b/images/breeze/output_ci-image_build.txt
index efab651ef7..4d8baf4b90 100644
--- a/images/breeze/output_ci-image_build.txt
+++ b/images/breeze/output_ci-image_build.txt
@@ -1 +1 @@
-9bc9d889340bf7fc3b30ccad2c231257
+f282460baa522fbac2b43027973ea505
diff --git a/images/breeze/output_prod-image_build.svg b/images/breeze/output_prod-image_build.svg
index 973a763035..2e6599bb57 100644
--- a/images/breeze/output_prod-image_build.svg
+++ b/images/breeze/output_prod-image_build.svg
@@ -1,4 +1,4 @@
-<svg class="rich-terminal" viewBox="0 0 1482 2563.2" xmlns="http://www.w3.org/2000/svg">
+<svg class="rich-terminal" viewBox="0 0 1482 2538.7999999999997" xmlns="http://www.w3.org/2000/svg">
<!-- Generated with Rich https://www.textualize.io -->
<style>
@@ -43,7 +43,7 @@
<defs>
<clipPath id="breeze-prod-image-build-clip-terminal">
- <rect x="0" y="0" width="1463.0" height="2512.2" />
+ <rect x="0" y="0" width="1463.0" height="2487.7999999999997" />
</clipPath>
<clipPath id="breeze-prod-image-build-line-0">
<rect x="0" y="1.5" width="1464" height="24.65"/>
@@ -348,12 +348,9 @@
<clipPath id="breeze-prod-image-build-line-100">
<rect x="0" y="2441.5" width="1464" height="24.65"/>
</clipPath>
-<clipPath id="breeze-prod-image-build-line-101">
- <rect x="0" y="2465.9" width="1464" height="24.65"/>
- </clipPath>
</defs>
- <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="2561.2" rx="8"/><text class="breeze-prod-image-build-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: prod-image build</text>
+ <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="2536.8" rx="8"/><text class="breeze-prod-image-build-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: prod-image build</text>
<g transform="translate(26,22)">
<circle cx="0" cy="0" r="7" fill="#ff5f57"/>
<circle cx="22" cy="0" r="7" fill="#febc2e"/>
@@ -396,76 +393,75 @@
</text><text class="breeze-prod-image-build-r5" x="0" y="752" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-30)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="752" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-30)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="752" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-30)">-include</text><text class="breeze-prod-image-build-r4" x="134.2" y="752" textLength="195.2" c [...]
</text><text class="breeze-prod-image-build-r5" x="0" y="776.4" textLength="1464" clip-path="url(#breeze-prod-image-build-line-31)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-build-r1" x="1464" y="776.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-31)">
</text><text class="breeze-prod-image-build-r5" x="0" y="800.8" textLength="24.4" clip-path="url(#breeze-prod-image-build-line-32)">╭─</text><text class="breeze-prod-image-build-r5" x="24.4" y="800.8" textLength="512.4" clip-path="url(#breeze-prod-image-build-line-32)"> Advanced build options (for power users) </text><text class="breeze-prod-image-build-r5" x="536.8" y="800.8" textLength="902.8" clip-path="url(#breeze-prod-image-build-line-32)">──────── [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="825.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-33)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="825.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-33)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="825.2" textLength="85.4" clip-path="url(#breeze-prod-image-build-line-33)">-debian</text><text class="breeze-prod-image-build-r4" x="122" y="825.2" textLength="97. [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="849.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-34)">│</text><text class="breeze-prod-image-build-r7" x="463.6" y="849.6" textLength="793" clip-path="url(#breeze-prod-image-build-line-34)">(bookworm | bullseye)                                & [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="874" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-35)">│</text><text class="breeze-prod-image-build-r5" x="463.6" y="874" textLength="793" clip-path="url(#breeze-prod-image-build-line-35)">[default: bookworm]                                   [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="898.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-36)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="898.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-36)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="898.4" textLength="85.4" clip-path="url(#breeze-prod-image-build-line-36)">-python</text><text class="breeze-prod-image-build-r4" x="122" y="898.4" textLength="73. [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="922.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-37)">│</text><text class="breeze-prod-image-build-r1" x="463.6" y="922.8" textLength="976" clip-path="url(#breeze-prod-image-build-line-37)">something like: python:VERSION-slim-bookworm.                            & [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="947.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-38)">│</text><text class="breeze-prod-image-build-r7" x="463.6" y="947.2" textLength="976" clip-path="url(#breeze-prod-image-build-line-38)">(TEXT)                                    &# [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="971.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-39)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="971.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-39)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="971.6" textLength="85.4" clip-path="url(#breeze-prod-image-build-line-39)">-commit</text><text class="breeze-prod-image-build-r4" x="122" y="971.6" textLength="48. [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="996" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-40)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="996" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-40)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="996" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-40)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="996" textLength="219. [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1020.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-41)">│</text><text class="breeze-prod-image-build-r1" x="463.6" y="1020.4" textLength="976" clip-path="url(#breeze-prod-image-build-line-41)">itself).                                    [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1044.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-42)">│</text><text class="breeze-prod-image-build-r7" x="463.6" y="1044.8" textLength="976" clip-path="url(#breeze-prod-image-build-line-42)">(TEXT)                                     [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1069.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-43)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1069.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-43)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1069.2" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-43)">-install</text><text class="breeze-prod-image-build-r4" x="134.2" y="1069.2" textLeng [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1093.6" textLength="1464" clip-path="url(#breeze-prod-image-build-line-44)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-build-r1" x="1464" y="1093.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-44)">
-</text><text class="breeze-prod-image-build-r5" x="0" y="1118" textLength="24.4" clip-path="url(#breeze-prod-image-build-line-45)">╭─</text><text class="breeze-prod-image-build-r5" x="24.4" y="1118" textLength="597.8" clip-path="url(#breeze-prod-image-build-line-45)"> Selecting constraint location (for power users) </text><text class="breeze-prod-image-build-r5" x="622.2" y="1118" textLength="817.4" clip-path="url(#breeze-prod-image-build-line-45)">──── [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1142.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-46)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1142.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-46)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1142.4" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-46)">-airflow</text><text class="breeze-prod-image-build-r4" x="134.2" y="1142.4" textLeng [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1166.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-47)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1166.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-47)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1166.8" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-47)">-airflow</text><text class="breeze-prod-image-build-r4" x="134.2" y="1166.8" textLeng [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1191.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-48)">│</text><text class="breeze-prod-image-build-r7" x="451.4" y="1191.2" textLength="866.2" clip-path="url(#breeze-prod-image-build-line-48)">(constraints | constraints-no-providers | constraints-source-providers)</text><text class="breeze-prod-image-build-r5" x="1451.8" y="1191.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-l [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1215.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-49)">│</text><text class="breeze-prod-image-build-r5" x="451.4" y="1215.6" textLength="866.2" clip-path="url(#breeze-prod-image-build-line-49)">[default: constraints]                                & [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1240" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-50)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1240" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-50)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1240" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-50)">-airflow</text><text class="breeze-prod-image-build-r4" x="134.2" y="1240" textLength="268. [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1264.4" textLength="1464" clip-path="url(#breeze-prod-image-build-line-51)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-build-r1" x="1464" y="1264.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-51)">
-</text><text class="breeze-prod-image-build-r5" x="0" y="1288.8" textLength="24.4" clip-path="url(#breeze-prod-image-build-line-52)">╭─</text><text class="breeze-prod-image-build-r5" x="24.4" y="1288.8" textLength="634.4" clip-path="url(#breeze-prod-image-build-line-52)"> Choosing dependencies and extras (for power users) </text><text class="breeze-prod-image-build-r5" x="658.8" y="1288.8" textLength="780.8" clip-path="url(#breeze-prod-image-build- [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1313.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-53)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1313.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-53)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1313.2" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-53)">-airflow</text><text class="breeze-prod-image-build-r4" x="134.2" y="1313.2" textLeng [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1337.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-54)">│</text><text class="breeze-prod-image-build-r7" x="463.6" y="1337.6" textLength="976" clip-path="url(#breeze-prod-image-build-line-54)">(TEXT)                                     [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1362" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-55)">│</text><text class="breeze-prod-image-build-r5" x="463.6" y="1362" textLength="976" clip-path="url(#breeze-prod-image-build-line-55)">[default:                                    & [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1386.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-56)">│</text><text class="breeze-prod-image-build-r5" x="463.6" y="1386.4" textLength="976" clip-path="url(#breeze-prod-image-build-line-56)">aiobotocore,amazon,async,celery,cncf.kubernetes,common.io,docker,elasticsearch,…</text><text class="breeze-prod-image-build-r5" x="1451.8" y="1386.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-56)">│</t [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1410.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-57)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1410.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-57)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1410.8" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-57)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1410.8" text [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1435.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-58)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1435.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-58)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1435.2" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-58)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1435.2" text [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1459.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-59)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1459.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-59)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1459.6" textLength="48.8" clip-path="url(#breeze-prod-image-build-line-59)">-dev</text><text class="breeze-prod-image-build-r4" x="85.4" y="1459.6" textLength="1 [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1484" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-60)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1484" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-60)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1484" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-60)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1484" textLength=" [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1508.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-61)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1508.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-61)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1508.4" textLength="48.8" clip-path="url(#breeze-prod-image-build-line-61)">-dev</text><text class="breeze-prod-image-build-r4" x="85.4" y="1508.4" textLength="1 [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1532.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-62)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1532.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-62)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1532.8" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-62)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1532.8" text [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1557.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-63)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1557.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-63)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1557.2" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-63)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1557.2" text [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1581.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-64)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1581.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-64)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1581.6" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-64)">-runtime</text><text class="breeze-prod-image-build-r4" x="134.2" y="1581.6" textLeng [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1606" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-65)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1606" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-65)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1606" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-65)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1606" textLength=" [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1630.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-66)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1630.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-66)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1630.4" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-66)">-runtime</text><text class="breeze-prod-image-build-r4" x="134.2" y="1630.4" textLeng [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1654.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-67)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1654.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-67)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1654.8" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-67)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1654.8" text [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1679.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-68)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1679.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-68)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1679.2" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-68)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1679.2" text [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1703.6" textLength="1464" clip-path="url(#breeze-prod-image-build-line-69)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-build-r1" x="1464" y="1703.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-69)">
-</text><text class="breeze-prod-image-build-r5" x="0" y="1728" textLength="24.4" clip-path="url(#breeze-prod-image-build-line-70)">╭─</text><text class="breeze-prod-image-build-r5" x="24.4" y="1728" textLength="817.4" clip-path="url(#breeze-prod-image-build-line-70)"> Advanced customization options (for specific customization needs) </text><text class="breeze-prod-image-build-r5" x="841.8" y="1728" textLength="597.8" clip-path="url(#breeze-prod-ima [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1752.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-71)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1752.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-71)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1752.4" textLength="158.6" clip-path="url(#breeze-prod-image-build-line-71)">-installation</text><text class="breeze-prod-image-build-r4" x="195.2" y="1752.4" te [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1776.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-72)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1776.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-72)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1776.8" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-72)">-install</text><text class="breeze-prod-image-build-r4" x="134.2" y="1776.8" textLeng [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1801.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-73)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1801.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-73)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1801.2" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-73)">-install</text><text class="breeze-prod-image-build-r4" x="134.2" y="1801.2" textLeng [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1825.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-74)">│</text><text class="breeze-prod-image-build-r1" x="536.8" y="1825.6" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-74)">Implies </text><text class="breeze-prod-image-build-r4" x="634.4" y="1825.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-74)">-</text><text class="breeze-prod-image-build-r4" x="646.6" y="1825.6" t [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1850" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-75)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1850" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-75)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1850" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-75)">-install</text><text class="breeze-prod-image-build-r4" x="134.2" y="1850" textLength="219. [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1874.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-76)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1874.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-76)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1874.4" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-76)">-cleanup</text><text class="breeze-prod-image-build-r4" x="134.2" y="1874.4" textLeng [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1898.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-77)">│</text><text class="breeze-prod-image-build-r1" x="536.8" y="1898.8" textLength="170.8" clip-path="url(#breeze-prod-image-build-line-77)">together with </text><text class="breeze-prod-image-build-r4" x="707.6" y="1898.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-77)">-</text><text class="breeze-prod-image-build-r4" x="719.8" [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1923.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-78)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1923.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-78)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1923.2" textLength="48.8" clip-path="url(#breeze-prod-image-build-line-78)">-use</text><text class="breeze-prod-image-build-r4" x="85.4" y="1923.2" textLength="4 [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1947.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-79)">│</text><text class="breeze-prod-image-build-r1" x="536.8" y="1947.6" textLength="902.8" clip-path="url(#breeze-prod-image-build-line-79)">constraints store in docker-context-files or from github.                 </text><text class="breeze-prod [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1972" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-80)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1972" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-80)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1972" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-80)">-disable</text><text class="breeze-prod-image-build-r4" x="134.2" y="1972" textLength="231. [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1996.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-81)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1996.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-81)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1996.4" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-81)">-disable</text><text class="breeze-prod-image-build-r4" x="134.2" y="1996.4" textLeng [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2020.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-82)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2020.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-82)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2020.8" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-82)">-disable</text><text class="breeze-prod-image-build-r4" x="134.2" y="2020.8" textLeng [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2045.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-83)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2045.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-83)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2045.2" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-83)">-disable</text><text class="breeze-prod-image-build-r4" x="134.2" y="2045.2" textLeng [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2069.6" textLength="1464" clip-path="url(#breeze-prod-image-build-line-84)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-build-r1" x="1464" y="2069.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-84)">
-</text><text class="breeze-prod-image-build-r5" x="0" y="2094" textLength="24.4" clip-path="url(#breeze-prod-image-build-line-85)">╭─</text><text class="breeze-prod-image-build-r5" x="24.4" y="2094" textLength="622.2" clip-path="url(#breeze-prod-image-build-line-85)"> Preparing cache and push (for maintainers and CI) </text><text class="breeze-prod-image-build-r5" x="646.6" y="2094" textLength="793" clip-path="url(#breeze-prod-image-build-line [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2118.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-86)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2118.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-86)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2118.4" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-86)">-builder</text><text class="breeze-prod-image-build-r1" x="341.6" y="2118.4" textLeng [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2142.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-87)">│</text><text class="breeze-prod-image-build-r5" x="341.6" y="2142.8" textLength="756.4" clip-path="url(#breeze-prod-image-build-line-87)">[default: autodetect]                                &# [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2167.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-88)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2167.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-88)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2167.2" textLength="109.8" clip-path="url(#breeze-prod-image-build-line-88)">-platform</text><text class="breeze-prod-image-build-r1" x="341.6" y="2167.2" textLe [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2191.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-89)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2191.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-89)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2191.6" textLength="61" clip-path="url(#breeze-prod-image-build-line-89)">-push</text><text class="breeze-prod-image-build-r1" x="341.6" y="2191.6" textLength="3 [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2216" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-90)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2216" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-90)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2216" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-90)">-prepare</text><text class="breeze-prod-image-build-r4" x="134.2" y="2216" textLength="158. [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2240.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-91)">│</text><text class="breeze-prod-image-build-r1" x="341.6" y="2240.4" textLength="1098" clip-path="url(#breeze-prod-image-build-line-91)">image).                                    [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2264.8" textLength="1464" clip-path="url(#breeze-prod-image-build-line-92)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-build-r1" x="1464" y="2264.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-92)">
-</text><text class="breeze-prod-image-build-r5" x="0" y="2289.2" textLength="24.4" clip-path="url(#breeze-prod-image-build-line-93)">╭─</text><text class="breeze-prod-image-build-r5" x="24.4" y="2289.2" textLength="280.6" clip-path="url(#breeze-prod-image-build-line-93)"> Github authentication </text><text class="breeze-prod-image-build-r5" x="305" y="2289.2" textLength="1134.6" clip-path="url(#breeze-prod-image-build-line-93)">───────────────────────────────────────────── [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2313.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-94)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2313.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-94)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2313.6" textLength="85.4" clip-path="url(#breeze-prod-image-build-line-94)">-github</text><text class="breeze-prod-image-build-r4" x="122" y="2313.6" textLength= [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2338" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-95)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2338" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-95)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2338" textLength="85.4" clip-path="url(#breeze-prod-image-build-line-95)">-github</text><text class="breeze-prod-image-build-r4" x="122" y="2338" textLength="73.2" c [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2362.4" textLength="1464" clip-path="url(#breeze-prod-image-build-line-96)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-build-r1" x="1464" y="2362.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-96)">
-</text><text class="breeze-prod-image-build-r5" x="0" y="2386.8" textLength="24.4" clip-path="url(#breeze-prod-image-build-line-97)">╭─</text><text class="breeze-prod-image-build-r5" x="24.4" y="2386.8" textLength="195.2" clip-path="url(#breeze-prod-image-build-line-97)"> Common options </text><text class="breeze-prod-image-build-r5" x="219.6" y="2386.8" textLength="1220" clip-path="url(#breeze-prod-image-build-line-97)">──────────────────────────────────────────────────── [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2411.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-98)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2411.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-98)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2411.2" textLength="85.4" clip-path="url(#breeze-prod-image-build-line-98)">-answer</text><text class="breeze-prod-image-build-r6" x="158.6" y="2411.2" textLengt [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2435.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-99)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2435.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-99)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2435.6" textLength="48.8" clip-path="url(#breeze-prod-image-build-line-99)">-dry</text><text class="breeze-prod-image-build-r4" x="85.4" y="2435.6" textLength="4 [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2460" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-100)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2460" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-100)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2460" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-100)">-verbose</text><text class="breeze-prod-image-build-r6" x="158.6" y="2460" textLength="2 [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2484.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-101)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2484.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-101)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2484.4" textLength="61" clip-path="url(#breeze-prod-image-build-line-101)">-help</text><text class="breeze-prod-image-build-r6" x="158.6" y="2484.4" textLength [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2508.8" textLength="1464" clip-path="url(#breeze-prod-image-build-line-102)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-build-r1" x="1464" y="2508.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-102)">
+</text><text class="breeze-prod-image-build-r5" x="0" y="825.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-33)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="825.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-33)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="825.2" textLength="85.4" clip-path="url(#breeze-prod-image-build-line-33)">-debian</text><text class="breeze-prod-image-build-r4" x="122" y="825.2" textLength="97. [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="849.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-34)">│</text><text class="breeze-prod-image-build-r7" x="439.2" y="849.6" textLength="793" clip-path="url(#breeze-prod-image-build-line-34)">(bookworm | bullseye)                                & [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="874" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-35)">│</text><text class="breeze-prod-image-build-r5" x="439.2" y="874" textLength="793" clip-path="url(#breeze-prod-image-build-line-35)">[default: bookworm]                                   [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="898.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-36)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="898.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-36)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="898.4" textLength="85.4" clip-path="url(#breeze-prod-image-build-line-36)">-python</text><text class="breeze-prod-image-build-r4" x="122" y="898.4" textLength="73. [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="922.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-37)">│</text><text class="breeze-prod-image-build-r1" x="439.2" y="922.8" textLength="1000.4" clip-path="url(#breeze-prod-image-build-line-37)">something like: python:VERSION-slim-bookworm.                            [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="947.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-38)">│</text><text class="breeze-prod-image-build-r7" x="439.2" y="947.2" textLength="1000.4" clip-path="url(#breeze-prod-image-build-line-38)">(TEXT)                                     [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="971.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-39)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="971.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-39)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="971.6" textLength="85.4" clip-path="url(#breeze-prod-image-build-line-39)">-commit</text><text class="breeze-prod-image-build-r4" x="122" y="971.6" textLength="48. [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="996" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-40)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="996" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-40)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="996" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-40)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="996" textLength="219. [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1020.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-41)">│</text><text class="breeze-prod-image-build-r1" x="439.2" y="1020.4" textLength="1000.4" clip-path="url(#breeze-prod-image-build-line-41)">itself).                                   & [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1044.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-42)">│</text><text class="breeze-prod-image-build-r7" x="439.2" y="1044.8" textLength="1000.4" clip-path="url(#breeze-prod-image-build-line-42)">(TEXT)                                    [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1069.2" textLength="1464" clip-path="url(#breeze-prod-image-build-line-43)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-build-r1" x="1464" y="1069.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-43)">
+</text><text class="breeze-prod-image-build-r5" x="0" y="1093.6" textLength="24.4" clip-path="url(#breeze-prod-image-build-line-44)">╭─</text><text class="breeze-prod-image-build-r5" x="24.4" y="1093.6" textLength="597.8" clip-path="url(#breeze-prod-image-build-line-44)"> Selecting constraint location (for power users) </text><text class="breeze-prod-image-build-r5" x="622.2" y="1093.6" textLength="817.4" clip-path="url(#breeze-prod-image-build-line-44) [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1118" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-45)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1118" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-45)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1118" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-45)">-airflow</text><text class="breeze-prod-image-build-r4" x="134.2" y="1118" textLength="256. [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1142.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-46)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1142.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-46)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1142.4" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-46)">-airflow</text><text class="breeze-prod-image-build-r4" x="134.2" y="1142.4" textLeng [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1166.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-47)">│</text><text class="breeze-prod-image-build-r7" x="451.4" y="1166.8" textLength="866.2" clip-path="url(#breeze-prod-image-build-line-47)">(constraints | constraints-no-providers | constraints-source-providers)</text><text class="breeze-prod-image-build-r5" x="1451.8" y="1166.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-l [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1191.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-48)">│</text><text class="breeze-prod-image-build-r5" x="451.4" y="1191.2" textLength="866.2" clip-path="url(#breeze-prod-image-build-line-48)">[default: constraints]                                & [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1215.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-49)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1215.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-49)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1215.6" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-49)">-airflow</text><text class="breeze-prod-image-build-r4" x="134.2" y="1215.6" textLeng [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1240" textLength="1464" clip-path="url(#breeze-prod-image-build-line-50)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-build-r1" x="1464" y="1240" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-50)">
+</text><text class="breeze-prod-image-build-r5" x="0" y="1264.4" textLength="24.4" clip-path="url(#breeze-prod-image-build-line-51)">╭─</text><text class="breeze-prod-image-build-r5" x="24.4" y="1264.4" textLength="634.4" clip-path="url(#breeze-prod-image-build-line-51)"> Choosing dependencies and extras (for power users) </text><text class="breeze-prod-image-build-r5" x="658.8" y="1264.4" textLength="780.8" clip-path="url(#breeze-prod-image-build- [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1288.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-52)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1288.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-52)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1288.8" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-52)">-airflow</text><text class="breeze-prod-image-build-r4" x="134.2" y="1288.8" textLeng [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1313.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-53)">│</text><text class="breeze-prod-image-build-r7" x="463.6" y="1313.2" textLength="976" clip-path="url(#breeze-prod-image-build-line-53)">(TEXT)                                     [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1337.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-54)">│</text><text class="breeze-prod-image-build-r5" x="463.6" y="1337.6" textLength="976" clip-path="url(#breeze-prod-image-build-line-54)">[default:                                    [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1362" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-55)">│</text><text class="breeze-prod-image-build-r5" x="463.6" y="1362" textLength="976" clip-path="url(#breeze-prod-image-build-line-55)">aiobotocore,amazon,async,celery,cncf-kubernetes,common-io,docker,elasticsearch,…</text><text class="breeze-prod-image-build-r5" x="1451.8" y="1362" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-55)">│</text><t [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1386.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-56)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1386.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-56)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1386.4" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-56)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1386.4" text [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1410.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-57)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1410.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-57)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1410.8" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-57)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1410.8" text [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1435.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-58)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1435.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-58)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1435.2" textLength="48.8" clip-path="url(#breeze-prod-image-build-line-58)">-dev</text><text class="breeze-prod-image-build-r4" x="85.4" y="1435.2" textLength="1 [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1459.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-59)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1459.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-59)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1459.6" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-59)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1459.6" text [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1484" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-60)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1484" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-60)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1484" textLength="48.8" clip-path="url(#breeze-prod-image-build-line-60)">-dev</text><text class="breeze-prod-image-build-r4" x="85.4" y="1484" textLength="146.4" cl [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1508.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-61)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1508.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-61)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1508.4" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-61)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1508.4" text [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1532.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-62)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1532.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-62)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1532.8" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-62)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1532.8" text [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1557.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-63)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1557.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-63)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1557.2" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-63)">-runtime</text><text class="breeze-prod-image-build-r4" x="134.2" y="1557.2" textLeng [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1581.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-64)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1581.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-64)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1581.6" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-64)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1581.6" text [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1606" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-65)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1606" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-65)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1606" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-65)">-runtime</text><text class="breeze-prod-image-build-r4" x="134.2" y="1606" textLength="146. [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1630.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-66)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1630.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-66)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1630.4" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-66)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1630.4" text [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1654.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-67)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1654.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-67)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1654.8" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-67)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1654.8" text [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1679.2" textLength="1464" clip-path="url(#breeze-prod-image-build-line-68)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-build-r1" x="1464" y="1679.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-68)">
+</text><text class="breeze-prod-image-build-r5" x="0" y="1703.6" textLength="24.4" clip-path="url(#breeze-prod-image-build-line-69)">╭─</text><text class="breeze-prod-image-build-r5" x="24.4" y="1703.6" textLength="817.4" clip-path="url(#breeze-prod-image-build-line-69)"> Advanced customization options (for specific customization needs) </text><text class="breeze-prod-image-build-r5" x="841.8" y="1703.6" textLength="597.8" clip-path="url(#breeze-pr [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1728" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-70)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1728" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-70)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1728" textLength="158.6" clip-path="url(#breeze-prod-image-build-line-70)">-installation</text><text class="breeze-prod-image-build-r4" x="195.2" y="1728" textLength [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1752.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-71)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1752.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-71)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1752.4" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-71)">-install</text><text class="breeze-prod-image-build-r4" x="134.2" y="1752.4" textLeng [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1776.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-72)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1776.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-72)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1776.8" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-72)">-install</text><text class="breeze-prod-image-build-r4" x="134.2" y="1776.8" textLeng [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1801.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-73)">│</text><text class="breeze-prod-image-build-r1" x="536.8" y="1801.2" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-73)">Implies </text><text class="breeze-prod-image-build-r4" x="634.4" y="1801.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-73)">-</text><text class="breeze-prod-image-build-r4" x="646.6" y="1801.2" t [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1825.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-74)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1825.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-74)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1825.6" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-74)">-install</text><text class="breeze-prod-image-build-r4" x="134.2" y="1825.6" textLeng [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1850" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-75)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1850" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-75)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1850" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-75)">-cleanup</text><text class="breeze-prod-image-build-r4" x="134.2" y="1850" textLength="97.6 [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1874.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-76)">│</text><text class="breeze-prod-image-build-r1" x="536.8" y="1874.4" textLength="170.8" clip-path="url(#breeze-prod-image-build-line-76)">together with </text><text class="breeze-prod-image-build-r4" x="707.6" y="1874.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-76)">-</text><text class="breeze-prod-image-build-r4" x="719.8" [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1898.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-77)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1898.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-77)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1898.8" textLength="48.8" clip-path="url(#breeze-prod-image-build-line-77)">-use</text><text class="breeze-prod-image-build-r4" x="85.4" y="1898.8" textLength="4 [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1923.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-78)">│</text><text class="breeze-prod-image-build-r1" x="536.8" y="1923.2" textLength="902.8" clip-path="url(#breeze-prod-image-build-line-78)">constraints store in docker-context-files or from github.                 </text><text class="breeze-prod [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1947.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-79)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1947.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-79)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1947.6" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-79)">-disable</text><text class="breeze-prod-image-build-r4" x="134.2" y="1947.6" textLeng [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1972" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-80)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1972" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-80)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1972" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-80)">-disable</text><text class="breeze-prod-image-build-r4" x="134.2" y="1972" textLength="317. [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1996.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-81)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1996.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-81)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1996.4" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-81)">-disable</text><text class="breeze-prod-image-build-r4" x="134.2" y="1996.4" textLeng [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2020.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-82)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2020.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-82)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2020.8" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-82)">-disable</text><text class="breeze-prod-image-build-r4" x="134.2" y="2020.8" textLeng [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2045.2" textLength="1464" clip-path="url(#breeze-prod-image-build-line-83)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-build-r1" x="1464" y="2045.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-83)">
+</text><text class="breeze-prod-image-build-r5" x="0" y="2069.6" textLength="24.4" clip-path="url(#breeze-prod-image-build-line-84)">╭─</text><text class="breeze-prod-image-build-r5" x="24.4" y="2069.6" textLength="622.2" clip-path="url(#breeze-prod-image-build-line-84)"> Preparing cache and push (for maintainers and CI) </text><text class="breeze-prod-image-build-r5" x="646.6" y="2069.6" textLength="793" clip-path="url(#breeze-prod-image-buil [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2094" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-85)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2094" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-85)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2094" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-85)">-builder</text><text class="breeze-prod-image-build-r1" x="341.6" y="2094" textLength="756. [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2118.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-86)">│</text><text class="breeze-prod-image-build-r5" x="341.6" y="2118.4" textLength="756.4" clip-path="url(#breeze-prod-image-build-line-86)">[default: autodetect]                                &# [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2142.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-87)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2142.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-87)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2142.8" textLength="109.8" clip-path="url(#breeze-prod-image-build-line-87)">-platform</text><text class="breeze-prod-image-build-r1" x="341.6" y="2142.8" textLe [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2167.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-88)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2167.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-88)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2167.2" textLength="61" clip-path="url(#breeze-prod-image-build-line-88)">-push</text><text class="breeze-prod-image-build-r1" x="341.6" y="2167.2" textLength="3 [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2191.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-89)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2191.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-89)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2191.6" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-89)">-prepare</text><text class="breeze-prod-image-build-r4" x="134.2" y="2191.6" textLeng [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2216" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-90)">│</text><text class="breeze-prod-image-build-r1" x="341.6" y="2216" textLength="1098" clip-path="url(#breeze-prod-image-build-line-90)">image).                                    &# [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2240.4" textLength="1464" clip-path="url(#breeze-prod-image-build-line-91)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-build-r1" x="1464" y="2240.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-91)">
+</text><text class="breeze-prod-image-build-r5" x="0" y="2264.8" textLength="24.4" clip-path="url(#breeze-prod-image-build-line-92)">╭─</text><text class="breeze-prod-image-build-r5" x="24.4" y="2264.8" textLength="280.6" clip-path="url(#breeze-prod-image-build-line-92)"> Github authentication </text><text class="breeze-prod-image-build-r5" x="305" y="2264.8" textLength="1134.6" clip-path="url(#breeze-prod-image-build-line-92)">───────────────────────────────────────────── [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2289.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-93)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2289.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-93)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2289.2" textLength="85.4" clip-path="url(#breeze-prod-image-build-line-93)">-github</text><text class="breeze-prod-image-build-r4" x="122" y="2289.2" textLength= [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2313.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-94)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2313.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-94)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2313.6" textLength="85.4" clip-path="url(#breeze-prod-image-build-line-94)">-github</text><text class="breeze-prod-image-build-r4" x="122" y="2313.6" textLength= [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2338" textLength="1464" clip-path="url(#breeze-prod-image-build-line-95)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-build-r1" x="1464" y="2338" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-95)">
+</text><text class="breeze-prod-image-build-r5" x="0" y="2362.4" textLength="24.4" clip-path="url(#breeze-prod-image-build-line-96)">╭─</text><text class="breeze-prod-image-build-r5" x="24.4" y="2362.4" textLength="195.2" clip-path="url(#breeze-prod-image-build-line-96)"> Common options </text><text class="breeze-prod-image-build-r5" x="219.6" y="2362.4" textLength="1220" clip-path="url(#breeze-prod-image-build-line-96)">──────────────────────────────────────────────────── [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2386.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-97)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2386.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-97)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2386.8" textLength="85.4" clip-path="url(#breeze-prod-image-build-line-97)">-answer</text><text class="breeze-prod-image-build-r6" x="158.6" y="2386.8" textLengt [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2411.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-98)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2411.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-98)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2411.2" textLength="48.8" clip-path="url(#breeze-prod-image-build-line-98)">-dry</text><text class="breeze-prod-image-build-r4" x="85.4" y="2411.2" textLength="4 [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2435.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-99)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2435.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-99)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2435.6" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-99)">-verbose</text><text class="breeze-prod-image-build-r6" x="158.6" y="2435.6" textLeng [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2460" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-100)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2460" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-100)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2460" textLength="61" clip-path="url(#breeze-prod-image-build-line-100)">-help</text><text class="breeze-prod-image-build-r6" x="158.6" y="2460" textLength="24.4" [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2484.4" textLength="1464" clip-path="url(#breeze-prod-image-build-line-101)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-build-r1" x="1464" y="2484.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-101)">
</text>
</g>
</g>
diff --git a/images/breeze/output_prod-image_build.txt b/images/breeze/output_prod-image_build.txt
index 084f184b36..f3a8dbfed9 100644
--- a/images/breeze/output_prod-image_build.txt
+++ b/images/breeze/output_prod-image_build.txt
@@ -1 +1 @@
-ed924a9f4691700717cb77864ba9603a
+571a780417841263c788620fe5cf31da
diff --git a/images/breeze/output_release-management.svg b/images/breeze/output_release-management.svg
index 13ff72831f..f853a75b69 100644
--- a/images/breeze/output_release-management.svg
+++ b/images/breeze/output_release-management.svg
@@ -175,7 +175,7 @@
</text><text class="breeze-release-management-r5" x="0" y="581.2" textLength="1464" clip-path="url(#breeze-release-management-line-23)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-release-management-r1" x="1464" y="581.2" textLength="12.2" clip-path="url(#breeze-release-management-line-23)">
</text><text class="breeze-release-management-r5" x="0" y="605.6" textLength="24.4" clip-path="url(#breeze-release-management-line-24)">╭─</text><text class="breeze-release-management-r5" x="24.4" y="605.6" textLength="292.8" clip-path="url(#breeze-release-management-line-24)"> Other release commands </text><text class="breeze-release-management-r5" x="317.2" y="605.6" textLength="1122.4" clip-path="url(#breeze-release-management-line-24)">──────────────────────────── [...]
</text><text class="breeze-release-management-r5" x="0" y="630" textLength="12.2" clip-path="url(#breeze-release-management-line-25)">│</text><text class="breeze-release-management-r4" x="24.4" y="630" textLength="280.6" clip-path="url(#breeze-release-management-line-25)">publish-docs           </text><text class="breeze-release-management-r1" x="329.4" y="630" textLength="1110.2" clip-path="url(#breeze-release-management-line-25)">C [...]
-</text><text class="breeze-release-management-r5" x="0" y="654.4" textLength="12.2" clip-path="url(#breeze-release-management-line-26)">│</text><text class="breeze-release-management-r4" x="24.4" y="654.4" textLength="280.6" clip-path="url(#breeze-release-management-line-26)">generate-constraints   </text><text class="breeze-release-management-r1" x="329.4" y="654.4" textLength="1110.2" clip-path="url(#breeze-release-management-line-26)">Generates pinned constrai [...]
+</text><text class="breeze-release-management-r5" x="0" y="654.4" textLength="12.2" clip-path="url(#breeze-release-management-line-26)">│</text><text class="breeze-release-management-r4" x="24.4" y="654.4" textLength="280.6" clip-path="url(#breeze-release-management-line-26)">generate-constraints   </text><text class="breeze-release-management-r1" x="329.4" y="654.4" textLength="1110.2" clip-path="url(#breeze-release-management-line-26)">Generates pinned constrai [...]
</text><text class="breeze-release-management-r5" x="0" y="678.8" textLength="12.2" clip-path="url(#breeze-release-management-line-27)">│</text><text class="breeze-release-management-r4" x="24.4" y="678.8" textLength="280.6" clip-path="url(#breeze-release-management-line-27)">add-back-references    </text><text class="breeze-release-management-r1" x="329.4" y="678.8" textLength="1110.2" clip-path="url(#breeze-release-management-line-27)">Command to add  [...]
</text><text class="breeze-release-management-r5" x="0" y="703.2" textLength="1464" clip-path="url(#breeze-release-management-line-28)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-release-management-r1" x="1464" y="703.2" textLength="12.2" clip-path="url(#breeze-release-management-line-28)">
</text><text class="breeze-release-management-r5" x="0" y="727.6" textLength="24.4" clip-path="url(#breeze-release-management-line-29)">╭─</text><text class="breeze-release-management-r5" x="24.4" y="727.6" textLength="122" clip-path="url(#breeze-release-management-line-29)"> Commands </text><text class="breeze-release-management-r5" x="146.4" y="727.6" textLength="1293.2" clip-path="url(#breeze-release-management-line-29)">────────────────────────────────────────────────────── [...]
diff --git a/images/breeze/output_release-management.txt b/images/breeze/output_release-management.txt
index c446570e2e..a0e84e6bf9 100644
--- a/images/breeze/output_release-management.txt
+++ b/images/breeze/output_release-management.txt
@@ -1 +1 @@
-62b597858ea84bae695ddeba5e8c34f2
+f4edc0671c6ceb04d81e936f600ba8c4
diff --git a/images/breeze/output_release-management_generate-constraints.svg b/images/breeze/output_release-management_generate-constraints.svg
index b8bc2f4b31..e00db31f3f 100644
--- a/images/breeze/output_release-management_generate-constraints.svg
+++ b/images/breeze/output_release-management_generate-constraints.svg
@@ -165,7 +165,7 @@
<text class="breeze-release-management-generate-constraints-r1" x="1464" y="20" textLength="12.2" clip-path="url(#breeze-release-management-generate-constraints-line-0)">
</text><text class="breeze-release-management-generate-constraints-r2" x="12.2" y="44.4" textLength="73.2" clip-path="url(#breeze-release-management-generate-constraints-line-1)">Usage:</text><text class="breeze-release-management-generate-constraints-r3" x="97.6" y="44.4" textLength="561.2" clip-path="url(#breeze-release-management-generate-constraints-line-1)">breeze release-management generate-constraints</text><text class="breeze-release-management-generate-constraints-r1" [...]
</text><text class="breeze-release-management-generate-constraints-r1" x="1464" y="68.8" textLength="12.2" clip-path="url(#breeze-release-management-generate-constraints-line-2)">
-</text><text class="breeze-release-management-generate-constraints-r1" x="12.2" y="93.2" textLength="927.2" clip-path="url(#breeze-release-management-generate-constraints-line-3)">Generates pinned constraint files with all extras from setup.py in parallel.</text><text class="breeze-release-management-generate-constraints-r1" x="1464" y="93.2" textLength="12.2" clip-path="url(#breeze-release-management-generate-constraints-line-3)">
+</text><text class="breeze-release-management-generate-constraints-r1" x="12.2" y="93.2" textLength="1000.4" clip-path="url(#breeze-release-management-generate-constraints-line-3)">Generates pinned constraint files with all extras from pyproject.toml in parallel.</text><text class="breeze-release-management-generate-constraints-r1" x="1464" y="93.2" textLength="12.2" clip-path="url(#breeze-release-management-generate-constraints-line-3)">
</text><text class="breeze-release-management-generate-constraints-r1" x="1464" y="117.6" textLength="12.2" clip-path="url(#breeze-release-management-generate-constraints-line-4)">
</text><text class="breeze-release-management-generate-constraints-r5" x="0" y="142" textLength="24.4" clip-path="url(#breeze-release-management-generate-constraints-line-5)">╭─</text><text class="breeze-release-management-generate-constraints-r5" x="24.4" y="142" textLength="341.6" clip-path="url(#breeze-release-management-generate-constraints-line-5)"> Generate constraints flags </text><text class="breeze-release-management-generate-constraints-r5" x="366" y="142" t [...]
</text><text class="breeze-release-management-generate-constraints-r5" x="0" y="166.4" textLength="12.2" clip-path="url(#breeze-release-management-generate-constraints-line-6)">│</text><text class="breeze-release-management-generate-constraints-r4" x="24.4" y="166.4" textLength="12.2" clip-path="url(#breeze-release-management-generate-constraints-line-6)">-</text><text class="breeze-release-management-generate-constraints-r4" x="36.6" y="166.4" textLength="73.2" clip-path="url(#breeze-re [...]
diff --git a/images/breeze/output_release-management_generate-constraints.txt b/images/breeze/output_release-management_generate-constraints.txt
index 89a36cb1b1..37b035c8d5 100644
--- a/images/breeze/output_release-management_generate-constraints.txt
+++ b/images/breeze/output_release-management_generate-constraints.txt
@@ -1 +1 @@
-12f58e0c47b7533b91e547f33c79d4cb
+587c21578af75c3c8fb9b4629e3577e1
diff --git a/images/breeze/output_release-management_prepare-airflow-package.svg b/images/breeze/output_release-management_prepare-airflow-package.svg
index 4565c3e73d..c8490f5700 100644
--- a/images/breeze/output_release-management_prepare-airflow-package.svg
+++ b/images/breeze/output_release-management_prepare-airflow-package.svg
@@ -1,4 +1,4 @@
-<svg class="rich-terminal" viewBox="0 0 1482 513.5999999999999" xmlns="http://www.w3.org/2000/svg">
+<svg class="rich-terminal" viewBox="0 0 1482 391.59999999999997" xmlns="http://www.w3.org/2000/svg">
<!-- Generated with Rich https://www.textualize.io -->
<style>
@@ -43,7 +43,7 @@
<defs>
<clipPath id="breeze-release-management-prepare-airflow-package-clip-terminal">
- <rect x="0" y="0" width="1463.0" height="462.59999999999997" />
+ <rect x="0" y="0" width="1463.0" height="340.59999999999997" />
</clipPath>
<clipPath id="breeze-release-management-prepare-airflow-package-line-0">
<rect x="0" y="1.5" width="1464" height="24.65"/>
@@ -84,24 +84,9 @@
<clipPath id="breeze-release-management-prepare-airflow-package-line-12">
<rect x="0" y="294.3" width="1464" height="24.65"/>
</clipPath>
-<clipPath id="breeze-release-management-prepare-airflow-package-line-13">
- <rect x="0" y="318.7" width="1464" height="24.65"/>
- </clipPath>
-<clipPath id="breeze-release-management-prepare-airflow-package-line-14">
- <rect x="0" y="343.1" width="1464" height="24.65"/>
- </clipPath>
-<clipPath id="breeze-release-management-prepare-airflow-package-line-15">
- <rect x="0" y="367.5" width="1464" height="24.65"/>
- </clipPath>
-<clipPath id="breeze-release-management-prepare-airflow-package-line-16">
- <rect x="0" y="391.9" width="1464" height="24.65"/>
- </clipPath>
-<clipPath id="breeze-release-management-prepare-airflow-package-line-17">
- <rect x="0" y="416.3" width="1464" height="24.65"/>
- </clipPath>
</defs>
- <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="511.6" rx="8"/><text class="breeze-release-management-prepare-airflow-package-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: release-management prepare-airflow-package</text>
+ <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="389.6" rx="8"/><text class="breeze-release-management-prepare-airflow-package-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: release-management prepare-airflow-package</text>
<g transform="translate(26,22)">
<circle cx="0" cy="0" r="7" fill="#ff5f57"/>
<circle cx="22" cy="0" r="7" fill="#febc2e"/>
@@ -117,19 +102,14 @@
</text><text class="breeze-release-management-prepare-airflow-package-r1" x="12.2" y="93.2" textLength="451.4" clip-path="url(#breeze-release-management-prepare-airflow-package-line-3)">Prepare sdist/whl package of Airflow.</text><text class="breeze-release-management-prepare-airflow-package-r1" x="1464" y="93.2" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-3)">
</text><text class="breeze-release-management-prepare-airflow-package-r1" x="1464" y="117.6" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-4)">
</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="142" textLength="24.4" clip-path="url(#breeze-release-management-prepare-airflow-package-line-5)">╭─</text><text class="breeze-release-management-prepare-airflow-package-r5" x="24.4" y="142" textLength="183" clip-path="url(#breeze-release-management-prepare-airflow-package-line-5)"> Package flags </text><text class="breeze-release-management-prepare-airflow-package-r5" x="207.4" y="142" text [...]
-</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="166.4" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-6)">│</text><text class="breeze-release-management-prepare-airflow-package-r4" x="24.4" y="166.4" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-6)">-</text><text class="breeze-release-management-prepare-airflow-package-r4" x="36.6" y="166.4" textLength="97.6" clip-path= [...]
-</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="190.8" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-7)">│</text><text class="breeze-release-management-prepare-airflow-package-r4" x="24.4" y="190.8" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-7)">-</text><text class="breeze-release-management-prepare-airflow-package-r4" x="36.6" y="190.8" textLength="48.8" clip-path= [...]
-</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="215.2" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-8)">│</text><text class="breeze-release-management-prepare-airflow-package-r1" x="536.8" y="215.2" textLength="902.8" clip-path="url(#breeze-release-management-prepare-airflow-package-line-8)">compilation in containers is slower, due to slow mounted filesystem an [...]
-</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="239.6" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-9)">│</text><text class="breeze-release-management-prepare-airflow-package-r1" x="536.8" y="239.6" textLength="902.8" clip-path="url(#breeze-release-management-prepare-airflow-package-line-9)">number of node_module files so by default asset compilation is done [...]
-</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="264" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-10)">│</text><text class="breeze-release-management-prepare-airflow-package-r1" x="536.8" y="264" textLength="902.8" clip-path="url(#breeze-release-management-prepare-airflow-package-line-10)">locally. This option is useful for officially building packages by relea [...]
-</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-11)">│</text><text class="breeze-release-management-prepare-airflow-package-r1" x="536.8" y="288.4" textLength="902.8" clip-path="url(#breeze-release-management-prepare-airflow-package-line-11)">manager on MacOS to make sure it is a reproducible build.   [...]
-</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="312.8" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-12)">│</text><text class="breeze-release-management-prepare-airflow-package-r4" x="24.4" y="312.8" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-12)">-</text><text class="breeze-release-management-prepare-airflow-package-r4" x="36.6" y="312.8" textLength="97.6" clip-pat [...]
+</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="166.4" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-6)">│</text><text class="breeze-release-management-prepare-airflow-package-r4" x="24.4" y="166.4" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-6)">-</text><text class="breeze-release-management-prepare-airflow-package-r4" x="36.6" y="166.4" textLength="97.6" clip-path= [...]
+</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="190.8" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-7)">│</text><text class="breeze-release-management-prepare-airflow-package-r4" x="24.4" y="190.8" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-7)">-</text><text class="breeze-release-management-prepare-airflow-package-r4" x="36.6" y="190.8" textLength="97.6" clip-path= [...]
+</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="215.2" textLength="1464" clip-path="url(#breeze-release-management-prepare-airflow-package-line-8)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-release-management-prepare-airflow-package-r1" x="1464" y="215.2" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-8)">
+</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="239.6" textLength="24.4" clip-path="url(#breeze-release-management-prepare-airflow-package-line-9)">╭─</text><text class="breeze-release-management-prepare-airflow-package-r5" x="24.4" y="239.6" textLength="195.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-9)"> Common options </text><text class="breeze-release-management-prepare-airflow-package-r5" x="219.6" y="23 [...]
+</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="264" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-10)">│</text><text class="breeze-release-management-prepare-airflow-package-r4" x="24.4" y="264" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-10)">-</text><text class="breeze-release-management-prepare-airflow-package-r4" x="36.6" y="264" textLength="97.6" clip-path="url [...]
+</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-11)">│</text><text class="breeze-release-management-prepare-airflow-package-r4" x="24.4" y="288.4" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-11)">-</text><text class="breeze-release-management-prepare-airflow-package-r4" x="36.6" y="288.4" textLength="48.8" clip-pat [...]
+</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="312.8" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-12)">│</text><text class="breeze-release-management-prepare-airflow-package-r4" x="24.4" y="312.8" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-12)">-</text><text class="breeze-release-management-prepare-airflow-package-r4" x="36.6" y="312.8" textLength="61" clip-path= [...]
</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="337.2" textLength="1464" clip-path="url(#breeze-release-management-prepare-airflow-package-line-13)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-release-management-prepare-airflow-package-r1" x="1464" y="337.2" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-13)">
-</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="361.6" textLength="24.4" clip-path="url(#breeze-release-management-prepare-airflow-package-line-14)">╭─</text><text class="breeze-release-management-prepare-airflow-package-r5" x="24.4" y="361.6" textLength="195.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-14)"> Common options </text><text class="breeze-release-management-prepare-airflow-package-r5" x="219.6" y=" [...]
-</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="386" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-15)">│</text><text class="breeze-release-management-prepare-airflow-package-r4" x="24.4" y="386" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-15)">-</text><text class="breeze-release-management-prepare-airflow-package-r4" x="36.6" y="386" textLength="97.6" clip-path="url [...]
-</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="410.4" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-16)">│</text><text class="breeze-release-management-prepare-airflow-package-r4" x="24.4" y="410.4" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-16)">-</text><text class="breeze-release-management-prepare-airflow-package-r4" x="36.6" y="410.4" textLength="48.8" clip-pat [...]
-</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="434.8" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-17)">│</text><text class="breeze-release-management-prepare-airflow-package-r4" x="24.4" y="434.8" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-17)">-</text><text class="breeze-release-management-prepare-airflow-package-r4" x="36.6" y="434.8" textLength="61" clip-path= [...]
-</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="459.2" textLength="1464" clip-path="url(#breeze-release-management-prepare-airflow-package-line-18)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-release-management-prepare-airflow-package-r1" x="1464" y="459.2" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-18)">
</text>
</g>
</g>
diff --git a/images/breeze/output_release-management_prepare-airflow-package.txt b/images/breeze/output_release-management_prepare-airflow-package.txt
index 644026b1cf..f295947476 100644
--- a/images/breeze/output_release-management_prepare-airflow-package.txt
+++ b/images/breeze/output_release-management_prepare-airflow-package.txt
@@ -1 +1 @@
-5077bcfc4272c969f8b80ea4438a438a
+8f786828b9862f9ae63f1b691921979f
diff --git a/images/breeze/output_start-airflow.svg b/images/breeze/output_start-airflow.svg
index bfb5e80c5d..52bd238880 100644
--- a/images/breeze/output_start-airflow.svg
+++ b/images/breeze/output_start-airflow.svg
@@ -426,10 +426,10 @@
</text><text class="breeze-start-airflow-r5" x="0" y="1118" textLength="12.2" clip-path="url(#breeze-start-airflow-line-45)">│</text><text class="breeze-start-airflow-r4" x="24.4" y="1118" textLength="12.2" clip-path="url(#breeze-start-airflow-line-45)">-</text><text class="breeze-start-airflow-r4" x="36.6" y="1118" textLength="85.4" clip-path="url(#breeze-start-airflow-line-45)">-celery</text><text class="breeze-start-airflow-r4" x="122" y="1118" textLength="85.4" clip-path="url(#breeze [...]
</text><text class="breeze-start-airflow-r5" x="0" y="1142.4" textLength="1464" clip-path="url(#breeze-start-airflow-line-46)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-start-airflow-r1" x="1464" y="1142.4" textLength="12.2" clip-path="url(#breeze-start-airflow-line-46)">
</text><text class="breeze-start-airflow-r5" x="0" y="1166.8" textLength="24.4" clip-path="url(#breeze-start-airflow-line-47)">╭─</text><text class="breeze-start-airflow-r5" x="24.4" y="1166.8" textLength="329.4" clip-path="url(#breeze-start-airflow-line-47)"> Asset compilation options </text><text class="breeze-start-airflow-r5" x="353.8" y="1166.8" textLength="1085.8" clip-path="url(#breeze-start-airflow-line-47)">──────────────────────────────────────────────────── [...]
-</text><text class="breeze-start-airflow-r5" x="0" y="1191.2" textLength="12.2" clip-path="url(#breeze-start-airflow-line-48)">│</text><text class="breeze-start-airflow-r4" x="24.4" y="1191.2" textLength="12.2" clip-path="url(#breeze-start-airflow-line-48)">-</text><text class="breeze-start-airflow-r4" x="36.6" y="1191.2" textLength="61" clip-path="url(#breeze-start-airflow-line-48)">-skip</text><text class="breeze-start-airflow-r4" x="97.6" y="1191.2" textLength="219.6" clip-path="url(# [...]
-</text><text class="breeze-start-airflow-r5" x="0" y="1215.6" textLength="12.2" clip-path="url(#breeze-start-airflow-line-49)">│</text><text class="breeze-start-airflow-r1" x="366" y="1215.6" textLength="305" clip-path="url(#breeze-start-airflow-line-49)">(mutually exclusive with </text><text class="breeze-start-airflow-r4" x="671" y="1215.6" textLength="12.2" clip-path="url(#breeze-start-airflow-line-49)">-</text><text class="breeze-start-airflow-r4" x="683.2" y="1215.6" [...]
-</text><text class="breeze-start-airflow-r5" x="0" y="1240" textLength="12.2" clip-path="url(#breeze-start-airflow-line-50)">│</text><text class="breeze-start-airflow-r4" x="24.4" y="1240" textLength="12.2" clip-path="url(#breeze-start-airflow-line-50)">-</text><text class="breeze-start-airflow-r4" x="36.6" y="1240" textLength="48.8" clip-path="url(#breeze-start-airflow-line-50)">-dev</text><text class="breeze-start-airflow-r4" x="85.4" y="1240" textLength="61" clip-path="url(#breeze-sta [...]
-</text><text class="breeze-start-airflow-r5" x="0" y="1264.4" textLength="12.2" clip-path="url(#breeze-start-airflow-line-51)">│</text><text class="breeze-start-airflow-r1" x="366" y="1264.4" textLength="305" clip-path="url(#breeze-start-airflow-line-51)">(mutually exclusive with </text><text class="breeze-start-airflow-r4" x="671" y="1264.4" textLength="12.2" clip-path="url(#breeze-start-airflow-line-51)">-</text><text class="breeze-start-airflow-r4" x="683.2" y="1264.4" [...]
+</text><text class="breeze-start-airflow-r5" x="0" y="1191.2" textLength="12.2" clip-path="url(#breeze-start-airflow-line-48)">│</text><text class="breeze-start-airflow-r4" x="24.4" y="1191.2" textLength="12.2" clip-path="url(#breeze-start-airflow-line-48)">-</text><text class="breeze-start-airflow-r4" x="36.6" y="1191.2" textLength="61" clip-path="url(#breeze-start-airflow-line-48)">-skip</text><text class="breeze-start-airflow-r4" x="97.6" y="1191.2" textLength="231.8" clip-path="url(# [...]
+</text><text class="breeze-start-airflow-r5" x="0" y="1215.6" textLength="12.2" clip-path="url(#breeze-start-airflow-line-49)">│</text><text class="breeze-start-airflow-r1" x="378.2" y="1215.6" textLength="305" clip-path="url(#breeze-start-airflow-line-49)">(mutually exclusive with </text><text class="breeze-start-airflow-r4" x="683.2" y="1215.6" textLength="12.2" clip-path="url(#breeze-start-airflow-line-49)">-</text><text class="breeze-start-airflow-r4" x="695.4" y="1215 [...]
+</text><text class="breeze-start-airflow-r5" x="0" y="1240" textLength="12.2" clip-path="url(#breeze-start-airflow-line-50)">│</text><text class="breeze-start-airflow-r4" x="24.4" y="1240" textLength="12.2" clip-path="url(#breeze-start-airflow-line-50)">-</text><text class="breeze-start-airflow-r4" x="36.6" y="1240" textLength="48.8" clip-path="url(#breeze-start-airflow-line-50)">-dev</text><text class="breeze-start-airflow-r4" x="85.4" y="1240" textLength="61" clip-path="url(#breeze-sta [...]
+</text><text class="breeze-start-airflow-r5" x="0" y="1264.4" textLength="12.2" clip-path="url(#breeze-start-airflow-line-51)">│</text><text class="breeze-start-airflow-r1" x="378.2" y="1264.4" textLength="305" clip-path="url(#breeze-start-airflow-line-51)">(mutually exclusive with </text><text class="breeze-start-airflow-r4" x="683.2" y="1264.4" textLength="12.2" clip-path="url(#breeze-start-airflow-line-51)">-</text><text class="breeze-start-airflow-r4" x="695.4" y="1264 [...]
</text><text class="breeze-start-airflow-r5" x="0" y="1288.8" textLength="1464" clip-path="url(#breeze-start-airflow-line-52)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-start-airflow-r1" x="1464" y="1288.8" textLength="12.2" clip-path="url(#breeze-start-airflow-line-52)">
</text><text class="breeze-start-airflow-r5" x="0" y="1313.2" textLength="24.4" clip-path="url(#breeze-start-airflow-line-53)">╭─</text><text class="breeze-start-airflow-r5" x="24.4" y="1313.2" textLength="488" clip-path="url(#breeze-start-airflow-line-53)"> Build CI image (before entering shell) </text><text class="breeze-start-airflow-r5" x="512.4" y="1313.2" textLength="927.2" clip-path="url(#breeze-start-airflow-line-53)">─────────────────────────── [...]
</text><text class="breeze-start-airflow-r5" x="0" y="1337.6" textLength="12.2" clip-path="url(#breeze-start-airflow-line-54)">│</text><text class="breeze-start-airflow-r4" x="24.4" y="1337.6" textLength="12.2" clip-path="url(#breeze-start-airflow-line-54)">-</text><text class="breeze-start-airflow-r4" x="36.6" y="1337.6" textLength="73.2" clip-path="url(#breeze-start-airflow-line-54)">-force</text><text class="breeze-start-airflow-r4" x="109.8" y="1337.6" textLength="73.2" clip-path="ur [...]
diff --git a/images/breeze/output_start-airflow.txt b/images/breeze/output_start-airflow.txt
index 8d4f9b8b9a..cadcbb728d 100644
--- a/images/breeze/output_start-airflow.txt
+++ b/images/breeze/output_start-airflow.txt
@@ -1 +1 @@
-2a7ecb1274ddcab1c6839a224958565e
+3f1895d03b9f1b82431ca67c9269fe75
diff --git a/images/breeze/output_static-checks.svg b/images/breeze/output_static-checks.svg
index f5aa3bba12..4212e283ea 100644
--- a/images/breeze/output_static-checks.svg
+++ b/images/breeze/output_static-checks.svg
@@ -1,4 +1,4 @@
-<svg class="rich-terminal" viewBox="0 0 1482 2075.2" xmlns="http://www.w3.org/2000/svg">
+<svg class="rich-terminal" viewBox="0 0 1482 2050.8" xmlns="http://www.w3.org/2000/svg">
<!-- Generated with Rich https://www.textualize.io -->
<style>
@@ -43,7 +43,7 @@
<defs>
<clipPath id="breeze-static-checks-clip-terminal">
- <rect x="0" y="0" width="1463.0" height="2024.1999999999998" />
+ <rect x="0" y="0" width="1463.0" height="1999.8" />
</clipPath>
<clipPath id="breeze-static-checks-line-0">
<rect x="0" y="1.5" width="1464" height="24.65"/>
@@ -288,12 +288,9 @@
<clipPath id="breeze-static-checks-line-80">
<rect x="0" y="1953.5" width="1464" height="24.65"/>
</clipPath>
-<clipPath id="breeze-static-checks-line-81">
- <rect x="0" y="1977.9" width="1464" height="24.65"/>
- </clipPath>
</defs>
- <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="2073.2" rx="8"/><text class="breeze-static-checks-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: static-checks</text>
+ <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="2048.8" rx="8"/><text class="breeze-static-checks-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: static-checks</text>
<g transform="translate(26,22)">
<circle cx="0" cy="0" r="7" fill="#ff5f57"/>
<circle cx="22" cy="0" r="7" fill="#febc2e"/>
@@ -333,59 +330,58 @@
</text><text class="breeze-static-checks-r5" x="0" y="678.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-27)">│</text><text class="breeze-static-checks-r7" x="451.4" y="678.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-27)">check-pre-commit-information-consistent | check-provide-create-sessions-imports |</text><text class="breeze-static-checks-r5" x="1451.8" y="678.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-27)">│</text [...]
</text><text class="breeze-static-checks-r5" x="0" y="703.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-28)">│</text><text class="breeze-static-checks-r7" x="451.4" y="703.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-28)">check-provider-docs-valid | check-provider-yaml-valid |                          </text>< [...]
</text><text class="breeze-static-checks-r5" x="0" y="727.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-29)">│</text><text class="breeze-static-checks-r7" x="451.4" y="727.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-29)">check-providers-init-file-missing | check-providers-subpackages-init-file-exist |</text><text class="breeze-static-checks-r5" x="1451.8" y="727.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-29)">│</text [...]
-</text><text class="breeze-static-checks-r5" x="0" y="752" textLength="12.2" clip-path="url(#breeze-static-checks-line-30)">│</text><text class="breeze-static-checks-r7" x="451.4" y="752" textLength="988.2" clip-path="url(#breeze-static-checks-line-30)">check-pydevd-left-in-code | check-revision-heads-map |                           </text> [...]
-</text><text class="breeze-static-checks-r5" x="0" y="776.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-31)">│</text><text class="breeze-static-checks-r7" x="451.4" y="776.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-31)">check-safe-filter-usage-in-html | check-setup-order |                            [...]
-</text><text class="breeze-static-checks-r5" x="0" y="800.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-32)">│</text><text class="breeze-static-checks-r7" x="451.4" y="800.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-32)">check-start-date-not-used-in-defaults | check-system-tests-present |             </text><text class="breeze-static-checks-r5" x="1451.8" y="800.8" textLen [...]
-</text><text class="breeze-static-checks-r5" x="0" y="825.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-33)">│</text><text class="breeze-static-checks-r7" x="451.4" y="825.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-33)">check-system-tests-tocs | check-tests-unittest-testcase |                        </text><text class [...]
-</text><text class="breeze-static-checks-r5" x="0" y="849.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-34)">│</text><text class="breeze-static-checks-r7" x="451.4" y="849.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-34)">check-urlparse-usage-in-code | check-usage-of-re2-over-re | check-xml | codespell</text><text class="breeze-static-checks-r5" x="1451.8" y="849.6" textLength="12.2" clip-path="url(#breeze-static-checks-li [...]
-</text><text class="breeze-static-checks-r5" x="0" y="874" textLength="12.2" clip-path="url(#breeze-static-checks-line-35)">│</text><text class="breeze-static-checks-r7" x="451.4" y="874" textLength="988.2" clip-path="url(#breeze-static-checks-line-35)">| compile-www-assets | compile-www-assets-dev |                             [...]
-</text><text class="breeze-static-checks-r5" x="0" y="898.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-36)">│</text><text class="breeze-static-checks-r7" x="451.4" y="898.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-36)">create-missing-init-py-files-tests | debug-statements | detect-private-key |     </text><text class="breeze-static-checks-r5" x="1451.8" y="898.4" textLength="12.2" clip-path="url(#bre [...]
-</text><text class="breeze-static-checks-r5" x="0" y="922.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-37)">│</text><text class="breeze-static-checks-r7" x="451.4" y="922.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-37)">doctoc | end-of-file-fixer | fix-encoding-pragma | flynt |                       </t [...]
-</text><text class="breeze-static-checks-r5" x="0" y="947.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-38)">│</text><text class="breeze-static-checks-r7" x="451.4" y="947.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-38)">generate-airflow-diagrams | generate-pypi-readme | identity | insert-license |   </text><text class="breeze-static-checks-r5" x="1451.8" y="947.2" textLength="12.2" clip-path="url(#bre [...]
-</text><text class="breeze-static-checks-r5" x="0" y="971.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-39)">│</text><text class="breeze-static-checks-r7" x="451.4" y="971.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-39)">lint-chart-schema | lint-css | lint-dockerfile | lint-helm-chart |               </text><text class="breeze-static-checks-r5 [...]
-</text><text class="breeze-static-checks-r5" x="0" y="996" textLength="12.2" clip-path="url(#breeze-static-checks-line-40)">│</text><text class="breeze-static-checks-r7" x="451.4" y="996" textLength="988.2" clip-path="url(#breeze-static-checks-line-40)">lint-json-schema | lint-markdown | lint-openapi | mixed-line-ending | mypy |     </text><text class="breeze-static-checks-r5" x="1451.8" y="996" textLength="12.2" clip- [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1020.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-41)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1020.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-41)">mypy-core | mypy-dev | mypy-docs | mypy-providers | pretty-format-json |         </text><text class="breeze-static-checks-r5" x="1451.8" y="10 [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1044.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-42)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1044.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-42)">python-no-log-warn | replace-bad-characters | rst-backticks | ruff | ruff-format </text><text class="breeze-static-checks-r5" x="1451.8" y="1044.8" textLength="12.2" clip-path="url(#breez [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1069.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-43)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1069.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-43)">| shellcheck | trailing-whitespace | ts-compile-format-lint-www |                </text><text class="breeze-static-checks- [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1093.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-44)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1093.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-44)">update-black-version | update-breeze-cmd-output |                             [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1118" textLength="12.2" clip-path="url(#breeze-static-checks-line-45)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1118" textLength="988.2" clip-path="url(#breeze-static-checks-line-45)">update-breeze-readme-config-hash | update-common-sql-api-stubs |                 </text><text class="breeze-static-checks-r5" x="1451.8" [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1142.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-46)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1142.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-46)">update-er-diagram | update-extras | update-in-the-wild-to-be-sorted |            </text><text class="breeze-static-checks-r5" x="1451.8" y="1142.4" [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1166.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-47)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1166.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-47)">update-inlined-dockerfile-scripts | update-installed-providers-to-be-sorted |    </text><text class="breeze-static-checks-r5" x="1451.8" y="1166.8" textLength="12.2" clip-path="url(#breeze-static-c [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1191.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-48)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1191.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-48)">update-local-yml-file | update-migration-references |                           &# [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1215.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-49)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1215.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-49)">update-providers-dependencies | update-spelling-wordlist-to-be-sorted |          </text><text class="breeze-static-checks-r5" x="1451.8" y="1215.6" textLength="12.2" c [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1240" textLength="12.2" clip-path="url(#breeze-static-checks-line-50)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1240" textLength="988.2" clip-path="url(#breeze-static-checks-line-50)">update-supported-versions | update-vendored-in-k8s-json-schema | update-version |</text><text class="breeze-static-checks-r5" x="1451.8" y="1240" textLength="12.2" clip-path="url(#breeze-static-checks-line-50)"> [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1264.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-51)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1264.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-51)">validate-pyproject | yamllint)                                 [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1288.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-52)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1288.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-52)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1288.8" textLength="61" clip-path="url(#breeze-static-checks-line-52)">-show</text><text class="breeze-static-checks-r4" x="97.6" y="1288.8" textLength="195.2" clip-path="url(# [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1313.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-53)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1313.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-53)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1313.2" textLength="134.2" clip-path="url(#breeze-static-checks-line-53)">-initialize</text><text class="breeze-static-checks-r4" x="170.8" y="1313.2" textLength="146.4" clip-p [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1337.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-54)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1337.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-54)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1337.6" textLength="48.8" clip-path="url(#breeze-static-checks-line-54)">-max</text><text class="breeze-static-checks-r4" x="85.4" y="1337.6" textLength="292.8" clip-path="url( [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1362" textLength="12.2" clip-path="url(#breeze-static-checks-line-55)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1362" textLength="854" clip-path="url(#breeze-static-checks-line-55)">(INTEGER RANGE)                                    &# [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1386.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-56)">│</text><text class="breeze-static-checks-r5" x="451.4" y="1386.4" textLength="854" clip-path="url(#breeze-static-checks-line-56)">[default: 3; 1<=x<=10]                                 [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1410.8" textLength="1464" clip-path="url(#breeze-static-checks-line-57)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-static-checks-r1" x="1464" y="1410.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-57)">
-</text><text class="breeze-static-checks-r5" x="0" y="1435.2" textLength="24.4" clip-path="url(#breeze-static-checks-line-58)">╭─</text><text class="breeze-static-checks-r5" x="24.4" y="1435.2" textLength="463.6" clip-path="url(#breeze-static-checks-line-58)"> Selecting files to run the checks on </text><text class="breeze-static-checks-r5" x="488" y="1435.2" textLength="951.6" clip-path="url(#breeze-static-checks-line-58)">──────────────────────── [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1459.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-59)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1459.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-59)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1459.6" textLength="61" clip-path="url(#breeze-static-checks-line-59)">-file</text><text class="breeze-static-checks-r6" x="256.2" y="1459.6" textLength="24.4" clip-path="url(# [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1484" textLength="12.2" clip-path="url(#breeze-static-checks-line-60)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1484" textLength="12.2" clip-path="url(#breeze-static-checks-line-60)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1484" textLength="48.8" clip-path="url(#breeze-static-checks-line-60)">-all</text><text class="breeze-static-checks-r4" x="85.4" y="1484" textLength="73.2" clip-path="url(#breeze-s [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1508.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-61)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1508.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-61)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1508.4" textLength="85.4" clip-path="url(#breeze-static-checks-line-61)">-commit</text><text class="breeze-static-checks-r4" x="122" y="1508.4" textLength="48.8" clip-path="url [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1532.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-62)">│</text><text class="breeze-static-checks-r1" x="305" y="1532.8" textLength="183" clip-path="url(#breeze-static-checks-line-62)">exclusive with </text><text class="breeze-static-checks-r4" x="488" y="1532.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-62)">-</text><text class="breeze-static-checks-r4" x="500.2" y="1532.8" textLength="61" [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1557.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-63)">│</text><text class="breeze-static-checks-r7" x="305" y="1557.2" textLength="1134.6" clip-path="url(#breeze-static-checks-line-63)">(TEXT)                                       [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1581.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-64)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1581.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-64)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1581.6" textLength="61" clip-path="url(#breeze-static-checks-line-64)">-last</text><text class="breeze-static-checks-r4" x="97.6" y="1581.6" textLength="85.4" clip-path="url(#b [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1606" textLength="12.2" clip-path="url(#breeze-static-checks-line-65)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1606" textLength="12.2" clip-path="url(#breeze-static-checks-line-65)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1606" textLength="61" clip-path="url(#breeze-static-checks-line-65)">-only</text><text class="breeze-static-checks-r4" x="97.6" y="1606" textLength="134.2" clip-path="url(#breeze-s [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1630.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-66)">│</text><text class="breeze-static-checks-r1" x="305" y="1630.4" textLength="1134.6" clip-path="url(#breeze-static-checks-line-66)">branch and HEAD of your branch.                              [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1654.8" textLength="1464" clip-path="url(#breeze-static-checks-line-67)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-static-checks-r1" x="1464" y="1654.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-67)">
-</text><text class="breeze-static-checks-r5" x="0" y="1679.2" textLength="24.4" clip-path="url(#breeze-static-checks-line-68)">╭─</text><text class="breeze-static-checks-r5" x="24.4" y="1679.2" textLength="463.6" clip-path="url(#breeze-static-checks-line-68)"> Building image before running checks </text><text class="breeze-static-checks-r5" x="488" y="1679.2" textLength="951.6" clip-path="url(#breeze-static-checks-line-68)">────────────────────────────────── [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1703.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-69)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1703.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-69)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1703.6" textLength="61" clip-path="url(#breeze-static-checks-line-69)">-skip</text><text class="breeze-static-checks-r4" x="97.6" y="1703.6" textLength="244" clip-path="url(#br [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1728" textLength="12.2" clip-path="url(#breeze-static-checks-line-70)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1728" textLength="12.2" clip-path="url(#breeze-static-checks-line-70)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1728" textLength="73.2" clip-path="url(#breeze-static-checks-line-70)">-force</text><text class="breeze-static-checks-r4" x="109.8" y="1728" textLength="73.2" clip-path="url(#breez [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1752.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-71)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1752.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-71)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1752.4" textLength="73.2" clip-path="url(#breeze-static-checks-line-71)">-image</text><text class="breeze-static-checks-r4" x="109.8" y="1752.4" textLength="48.8" clip-path="ur [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1776.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-72)">│</text><text class="breeze-static-checks-r7" x="414.8" y="1776.8" textLength="963.8" clip-path="url(#breeze-static-checks-line-72)">(TEXT)                                      [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1801.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-73)">│</text><text class="breeze-static-checks-r5" x="414.8" y="1801.2" textLength="963.8" clip-path="url(#breeze-static-checks-line-73)">[default: latest]                                    [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1825.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-74)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1825.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-74)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1825.6" textLength="85.4" clip-path="url(#breeze-static-checks-line-74)">-github</text><text class="breeze-static-checks-r4" x="122" y="1825.6" textLength="134.2" clip-path="ur [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1850" textLength="12.2" clip-path="url(#breeze-static-checks-line-75)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1850" textLength="12.2" clip-path="url(#breeze-static-checks-line-75)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1850" textLength="97.6" clip-path="url(#breeze-static-checks-line-75)">-builder</text><text class="breeze-static-checks-r1" x="414.8" y="1850" textLength="756.4" clip-path="url(#br [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1874.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-76)">│</text><text class="breeze-static-checks-r5" x="414.8" y="1874.4" textLength="756.4" clip-path="url(#breeze-static-checks-line-76)">[default: autodetect]                                  &# [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1898.8" textLength="1464" clip-path="url(#breeze-static-checks-line-77)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-static-checks-r1" x="1464" y="1898.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-77)">
-</text><text class="breeze-static-checks-r5" x="0" y="1923.2" textLength="24.4" clip-path="url(#breeze-static-checks-line-78)">╭─</text><text class="breeze-static-checks-r5" x="24.4" y="1923.2" textLength="195.2" clip-path="url(#breeze-static-checks-line-78)"> Common options </text><text class="breeze-static-checks-r5" x="219.6" y="1923.2" textLength="1220" clip-path="url(#breeze-static-checks-line-78)">────────────────────────────────────────────────────────────────────── [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1947.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-79)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1947.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-79)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1947.6" textLength="48.8" clip-path="url(#breeze-static-checks-line-79)">-dry</text><text class="breeze-static-checks-r4" x="85.4" y="1947.6" textLength="48.8" clip-path="url(# [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1972" textLength="12.2" clip-path="url(#breeze-static-checks-line-80)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1972" textLength="12.2" clip-path="url(#breeze-static-checks-line-80)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1972" textLength="97.6" clip-path="url(#breeze-static-checks-line-80)">-verbose</text><text class="breeze-static-checks-r6" x="158.6" y="1972" textLength="24.4" clip-path="url(#bre [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1996.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-81)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1996.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-81)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1996.4" textLength="61" clip-path="url(#breeze-static-checks-line-81)">-help</text><text class="breeze-static-checks-r6" x="158.6" y="1996.4" textLength="24.4" clip-path="url(# [...]
-</text><text class="breeze-static-checks-r5" x="0" y="2020.8" textLength="1464" clip-path="url(#breeze-static-checks-line-82)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-static-checks-r1" x="1464" y="2020.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-82)">
+</text><text class="breeze-static-checks-r5" x="0" y="752" textLength="12.2" clip-path="url(#breeze-static-checks-line-30)">│</text><text class="breeze-static-checks-r7" x="451.4" y="752" textLength="988.2" clip-path="url(#breeze-static-checks-line-30)">check-pydevd-left-in-code | check-pyproject-toml-order | check-revision-heads-map</text><text class="breeze-static-checks-r5" x="1451.8" y="752" textLength="12.2" clip-path="url(#breeze-static-checks-line-30)">│</text> [...]
+</text><text class="breeze-static-checks-r5" x="0" y="776.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-31)">│</text><text class="breeze-static-checks-r7" x="451.4" y="776.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-31)">| check-safe-filter-usage-in-html | check-start-date-not-used-in-defaults |      </text><text class="breeze-static-checks-r5" x="1451.8" y="776.4" textLength="12.2" clip-path="url(#bre [...]
+</text><text class="breeze-static-checks-r5" x="0" y="800.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-32)">│</text><text class="breeze-static-checks-r7" x="451.4" y="800.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-32)">check-system-tests-present | check-system-tests-tocs |                           </t [...]
+</text><text class="breeze-static-checks-r5" x="0" y="825.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-33)">│</text><text class="breeze-static-checks-r7" x="451.4" y="825.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-33)">check-tests-unittest-testcase | check-urlparse-usage-in-code |                   </text><text class="breeze-static-checks-r5 [...]
+</text><text class="breeze-static-checks-r5" x="0" y="849.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-34)">│</text><text class="breeze-static-checks-r7" x="451.4" y="849.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-34)">check-usage-of-re2-over-re | check-xml | codespell | compile-www-assets |        </text><text class="breeze-static-checks-r5" x="1451.8" y="849.6" textLength=" [...]
+</text><text class="breeze-static-checks-r5" x="0" y="874" textLength="12.2" clip-path="url(#breeze-static-checks-line-35)">│</text><text class="breeze-static-checks-r7" x="451.4" y="874" textLength="988.2" clip-path="url(#breeze-static-checks-line-35)">compile-www-assets-dev | create-missing-init-py-files-tests | debug-statements | </text><text class="breeze-static-checks-r5" x="1451.8" y="874" textLength="12.2" clip-path="url(#breeze-static-checks-line-35) [...]
+</text><text class="breeze-static-checks-r5" x="0" y="898.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-36)">│</text><text class="breeze-static-checks-r7" x="451.4" y="898.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-36)">detect-private-key | doctoc | end-of-file-fixer | fix-encoding-pragma | flynt |  </text><text class="breeze-static-checks-r5" x="1451.8" y="898.4" textLength="12.2" clip-path="url [...]
+</text><text class="breeze-static-checks-r5" x="0" y="922.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-37)">│</text><text class="breeze-static-checks-r7" x="451.4" y="922.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-37)">generate-airflow-diagrams | generate-pypi-readme | identity | insert-license |   </text><text class="breeze-static-checks-r5" x="1451.8" y="922.8" textLength="12.2" clip-path="url(#bre [...]
+</text><text class="breeze-static-checks-r5" x="0" y="947.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-38)">│</text><text class="breeze-static-checks-r7" x="451.4" y="947.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-38)">lint-chart-schema | lint-css | lint-dockerfile | lint-helm-chart |               </text><text class="breeze-static-checks-r5 [...]
+</text><text class="breeze-static-checks-r5" x="0" y="971.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-39)">│</text><text class="breeze-static-checks-r7" x="451.4" y="971.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-39)">lint-json-schema | lint-markdown | lint-openapi | mixed-line-ending | mypy |     </text><text class="breeze-static-checks-r5" x="1451.8" y="971.6" textLength="12.2" [...]
+</text><text class="breeze-static-checks-r5" x="0" y="996" textLength="12.2" clip-path="url(#breeze-static-checks-line-40)">│</text><text class="breeze-static-checks-r7" x="451.4" y="996" textLength="988.2" clip-path="url(#breeze-static-checks-line-40)">mypy-core | mypy-dev | mypy-docs | mypy-providers | pretty-format-json |         </text><text class="breeze-static-checks-r5" x="1451.8" y="996" tex [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1020.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-41)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1020.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-41)">python-no-log-warn | replace-bad-characters | rst-backticks | ruff | ruff-format </text><text class="breeze-static-checks-r5" x="1451.8" y="1020.4" textLength="12.2" clip-path="url(#breez [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1044.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-42)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1044.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-42)">| shellcheck | trailing-whitespace | ts-compile-format-lint-www |                </text><text class="breeze-static-checks- [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1069.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-43)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1069.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-43)">update-black-version | update-breeze-cmd-output |                             [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1093.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-44)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1093.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-44)">update-breeze-readme-config-hash | update-common-sql-api-stubs |                 </text><text class="breeze-static-checks-r5" x="145 [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1118" textLength="12.2" clip-path="url(#breeze-static-checks-line-45)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1118" textLength="988.2" clip-path="url(#breeze-static-checks-line-45)">update-er-diagram | update-extras | update-in-the-wild-to-be-sorted |            </text><text class="breeze-static-checks-r5" x="1451.8" y="1118" textL [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1142.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-46)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1142.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-46)">update-inlined-dockerfile-scripts | update-installed-providers-to-be-sorted |    </text><text class="breeze-static-checks-r5" x="1451.8" y="1142.4" textLength="12.2" clip-path="url(#breeze-static-c [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1166.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-47)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1166.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-47)">update-local-yml-file | update-migration-references |                           &# [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1191.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-48)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1191.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-48)">update-providers-dependencies | update-spelling-wordlist-to-be-sorted |          </text><text class="breeze-static-checks-r5" x="1451.8" y="1191.2" textLength="12.2" c [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1215.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-49)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1215.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-49)">update-supported-versions | update-vendored-in-k8s-json-schema | update-version |</text><text class="breeze-static-checks-r5" x="1451.8" y="1215.6" textLength="12.2" clip-path="url(#breeze-static-checks-line [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1240" textLength="12.2" clip-path="url(#breeze-static-checks-line-50)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1240" textLength="988.2" clip-path="url(#breeze-static-checks-line-50)">yamllint)                                       [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1264.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-51)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1264.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-51)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1264.4" textLength="61" clip-path="url(#breeze-static-checks-line-51)">-show</text><text class="breeze-static-checks-r4" x="97.6" y="1264.4" textLength="195.2" clip-path="url(# [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1288.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-52)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1288.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-52)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1288.8" textLength="134.2" clip-path="url(#breeze-static-checks-line-52)">-initialize</text><text class="breeze-static-checks-r4" x="170.8" y="1288.8" textLength="146.4" clip-p [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1313.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-53)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1313.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-53)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1313.2" textLength="48.8" clip-path="url(#breeze-static-checks-line-53)">-max</text><text class="breeze-static-checks-r4" x="85.4" y="1313.2" textLength="292.8" clip-path="url( [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1337.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-54)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1337.6" textLength="854" clip-path="url(#breeze-static-checks-line-54)">(INTEGER RANGE)                                    [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1362" textLength="12.2" clip-path="url(#breeze-static-checks-line-55)">│</text><text class="breeze-static-checks-r5" x="451.4" y="1362" textLength="854" clip-path="url(#breeze-static-checks-line-55)">[default: 3; 1<=x<=10]                                 &# [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1386.4" textLength="1464" clip-path="url(#breeze-static-checks-line-56)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-static-checks-r1" x="1464" y="1386.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-56)">
+</text><text class="breeze-static-checks-r5" x="0" y="1410.8" textLength="24.4" clip-path="url(#breeze-static-checks-line-57)">╭─</text><text class="breeze-static-checks-r5" x="24.4" y="1410.8" textLength="463.6" clip-path="url(#breeze-static-checks-line-57)"> Selecting files to run the checks on </text><text class="breeze-static-checks-r5" x="488" y="1410.8" textLength="951.6" clip-path="url(#breeze-static-checks-line-57)">──────────────────────── [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1435.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-58)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1435.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-58)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1435.2" textLength="61" clip-path="url(#breeze-static-checks-line-58)">-file</text><text class="breeze-static-checks-r6" x="256.2" y="1435.2" textLength="24.4" clip-path="url(# [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1459.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-59)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1459.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-59)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1459.6" textLength="48.8" clip-path="url(#breeze-static-checks-line-59)">-all</text><text class="breeze-static-checks-r4" x="85.4" y="1459.6" textLength="73.2" clip-path="url(# [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1484" textLength="12.2" clip-path="url(#breeze-static-checks-line-60)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1484" textLength="12.2" clip-path="url(#breeze-static-checks-line-60)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1484" textLength="85.4" clip-path="url(#breeze-static-checks-line-60)">-commit</text><text class="breeze-static-checks-r4" x="122" y="1484" textLength="48.8" clip-path="url(#breeze [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1508.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-61)">│</text><text class="breeze-static-checks-r1" x="305" y="1508.4" textLength="183" clip-path="url(#breeze-static-checks-line-61)">exclusive with </text><text class="breeze-static-checks-r4" x="488" y="1508.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-61)">-</text><text class="breeze-static-checks-r4" x="500.2" y="1508.4" textLength="61" [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1532.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-62)">│</text><text class="breeze-static-checks-r7" x="305" y="1532.8" textLength="1134.6" clip-path="url(#breeze-static-checks-line-62)">(TEXT)                                       [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1557.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-63)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1557.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-63)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1557.2" textLength="61" clip-path="url(#breeze-static-checks-line-63)">-last</text><text class="breeze-static-checks-r4" x="97.6" y="1557.2" textLength="85.4" clip-path="url(#b [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1581.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-64)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1581.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-64)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1581.6" textLength="61" clip-path="url(#breeze-static-checks-line-64)">-only</text><text class="breeze-static-checks-r4" x="97.6" y="1581.6" textLength="134.2" clip-path="url(# [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1606" textLength="12.2" clip-path="url(#breeze-static-checks-line-65)">│</text><text class="breeze-static-checks-r1" x="305" y="1606" textLength="1134.6" clip-path="url(#breeze-static-checks-line-65)">branch and HEAD of your branch.                              & [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1630.4" textLength="1464" clip-path="url(#breeze-static-checks-line-66)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-static-checks-r1" x="1464" y="1630.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-66)">
+</text><text class="breeze-static-checks-r5" x="0" y="1654.8" textLength="24.4" clip-path="url(#breeze-static-checks-line-67)">╭─</text><text class="breeze-static-checks-r5" x="24.4" y="1654.8" textLength="463.6" clip-path="url(#breeze-static-checks-line-67)"> Building image before running checks </text><text class="breeze-static-checks-r5" x="488" y="1654.8" textLength="951.6" clip-path="url(#breeze-static-checks-line-67)">────────────────────────────────── [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1679.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-68)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1679.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-68)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1679.2" textLength="61" clip-path="url(#breeze-static-checks-line-68)">-skip</text><text class="breeze-static-checks-r4" x="97.6" y="1679.2" textLength="244" clip-path="url(#br [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1703.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-69)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1703.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-69)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1703.6" textLength="73.2" clip-path="url(#breeze-static-checks-line-69)">-force</text><text class="breeze-static-checks-r4" x="109.8" y="1703.6" textLength="73.2" clip-path="ur [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1728" textLength="12.2" clip-path="url(#breeze-static-checks-line-70)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1728" textLength="12.2" clip-path="url(#breeze-static-checks-line-70)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1728" textLength="73.2" clip-path="url(#breeze-static-checks-line-70)">-image</text><text class="breeze-static-checks-r4" x="109.8" y="1728" textLength="48.8" clip-path="url(#breez [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1752.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-71)">│</text><text class="breeze-static-checks-r7" x="414.8" y="1752.4" textLength="963.8" clip-path="url(#breeze-static-checks-line-71)">(TEXT)                                      [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1776.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-72)">│</text><text class="breeze-static-checks-r5" x="414.8" y="1776.8" textLength="963.8" clip-path="url(#breeze-static-checks-line-72)">[default: latest]                                    [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1801.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-73)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1801.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-73)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1801.2" textLength="85.4" clip-path="url(#breeze-static-checks-line-73)">-github</text><text class="breeze-static-checks-r4" x="122" y="1801.2" textLength="134.2" clip-path="ur [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1825.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-74)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1825.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-74)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1825.6" textLength="97.6" clip-path="url(#breeze-static-checks-line-74)">-builder</text><text class="breeze-static-checks-r1" x="414.8" y="1825.6" textLength="756.4" clip-path= [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1850" textLength="12.2" clip-path="url(#breeze-static-checks-line-75)">│</text><text class="breeze-static-checks-r5" x="414.8" y="1850" textLength="756.4" clip-path="url(#breeze-static-checks-line-75)">[default: autodetect]                                    [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1874.4" textLength="1464" clip-path="url(#breeze-static-checks-line-76)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-static-checks-r1" x="1464" y="1874.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-76)">
+</text><text class="breeze-static-checks-r5" x="0" y="1898.8" textLength="24.4" clip-path="url(#breeze-static-checks-line-77)">╭─</text><text class="breeze-static-checks-r5" x="24.4" y="1898.8" textLength="195.2" clip-path="url(#breeze-static-checks-line-77)"> Common options </text><text class="breeze-static-checks-r5" x="219.6" y="1898.8" textLength="1220" clip-path="url(#breeze-static-checks-line-77)">────────────────────────────────────────────────────────────────────── [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1923.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-78)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1923.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-78)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1923.2" textLength="48.8" clip-path="url(#breeze-static-checks-line-78)">-dry</text><text class="breeze-static-checks-r4" x="85.4" y="1923.2" textLength="48.8" clip-path="url(# [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1947.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-79)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1947.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-79)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1947.6" textLength="97.6" clip-path="url(#breeze-static-checks-line-79)">-verbose</text><text class="breeze-static-checks-r6" x="158.6" y="1947.6" textLength="24.4" clip-path=" [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1972" textLength="12.2" clip-path="url(#breeze-static-checks-line-80)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1972" textLength="12.2" clip-path="url(#breeze-static-checks-line-80)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1972" textLength="61" clip-path="url(#breeze-static-checks-line-80)">-help</text><text class="breeze-static-checks-r6" x="158.6" y="1972" textLength="24.4" clip-path="url(#breeze-s [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1996.4" textLength="1464" clip-path="url(#breeze-static-checks-line-81)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-static-checks-r1" x="1464" y="1996.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-81)">
</text>
</g>
</g>
diff --git a/images/breeze/output_static-checks.txt b/images/breeze/output_static-checks.txt
index d135c297d0..8f9982f8cf 100644
--- a/images/breeze/output_static-checks.txt
+++ b/images/breeze/output_static-checks.txt
@@ -1 +1 @@
-3131ead122c4ff23924ceef326f7ce04
+01342faaf5558fb2af9ce20c7ca0be8c
diff --git a/newsfragments/36537.significant.rst b/newsfragments/36537.significant.rst
new file mode 100644
index 0000000000..22a288b076
--- /dev/null
+++ b/newsfragments/36537.significant.rst
@@ -0,0 +1,41 @@
+Airflow packaging specification follows modern Python packaging standards.
+
+We standardized Airflow dependency configuration to follow latest development in Python packaging by
+using pyproject.toml. Airflow is now compliant with those accepted PEPs:
+
+* `PEP-440 Version Identification and Dependency Specification <https://www.python.org/dev/peps/pep-0440/>`__
+* `PEP-517 A build-system independent format for source trees <https://www.python.org/dev/peps/pep-0517/>`__
+* `PEP-518 Specifying Minimum Build System Requirements for Python Projects <https://www.python.org/dev/peps/pep-0518/>`__
+* `PEP-561 Distributing and Packaging Type Information <https://www.python.org/dev/peps/pep-0561/>`__
+* `PEP-621 Storing project metadata in pyproject.toml <https://www.python.org/dev/peps/pep-0621/>`__
+* `PEP-660 Editable installs for pyproject.toml based builds (wheel based) <https://www.python.org/dev/peps/pep-0660/>`__
+* `PEP-685 Comparison of extra names for optional distribution dependencies <https://www.python.org/dev/peps/pep-0685/>`__
+
+Also we implement multiple license files support coming from Draft, not yet accepted (but supported by hatchling) PEP:
+* `PEP 639 Improving License Clarity with Better Package Metadata <https://peps.python.org/pep-0639/>`__
+
+This has almost no noticeable impact on users if they are using modern Python packaging and development tools, generally
+speaking Airflow should behave as it did before when installing it from PyPI and it should be much easier to install
+it for development purposes using ``pip install -e ".[devel]"``.
+
+The differences from the user side are:
+
+* Airflow extras now get extras normalized to ``-`` (following PEP-685) instead of ``_`` and ``.``
+ (as it was before in some extras). When you install airflow with such extras (for example ``dbt.core`` or
+ ``all_dbs``) you should use ``-`` instead of ``_`` and ``.``.
+
+In most modern tools this will work in backwards-compatible way, but in some old version of those tools you might need to
+replace ``_`` and ``.`` with ``-``. You can also get warnings that the extra you are installing does not exist - but usually
+this warning is harmless and the extra is installed anyway. It is, however, recommended to change to use ``-`` in extras in your dependency
+specifications for all Airflow extras.
+
+* Released airflow package does not contain ``devel``, ``devel-*``, ``doc`` and ``doc-gen`` extras.
+ Those extras are only available when you install Airflow from sources in ``--editable`` mode. This is
+ because those extras are only used for development and documentation building purposes and are not needed
+ when you install Airflow for production use. Those dependencies had unspecified and varying behaviour for
+ released packages anyway and you were not supposed to use them in released packages.
+
+* The ``all`` and ``all-*`` extras were not always working correctly when installing Airflow using constraints
+ because they were also considered as development-only dependencies. With this change, those dependencies are
+ now properly handling constraints and they will install properly with constraints, pulling the right set
+ of providers and dependencies when constraints are used.
diff --git a/pyproject.toml b/pyproject.toml
index a8f4bbdeec..b1c40d4b40 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -14,20 +14,1245 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
+
+[build-system]
+# build dependencies should be fixed - including all transitive dependencies. This way we can ensure
+# reproducibility of the build and make sure that any future releases of any dependencies will not
+# break the build of released airflow sources in the future.
+# We should periodically refresh these build dependencies by following this procecedure:
+# 1. Create a new virtual environment
+# 2. Run `pip install hatchling gitpython
+# 3. Run `pip freeze`
+# 4. Copy versions you see here (note that tomli needs to have python_version < '3.11').
+requires = [
+ "editables==0.5",
+ "gitdb==4.0.11",
+ "gitpython==3.1.40",
+ "hatchling==1.21.0",
+ "packaging==23.2",
+ "pathspec==0.12.1",
+ "pluggy==1.3.0",
+ "smmap==5.0.1",
+ "tomli==2.0.1; python_version < '3.11'",
+ "trove-classifiers==2023.11.29",
+]
+build-backend = "hatchling.build"
+
+[project]
+name = "apache-airflow"
+dynamic = ["version"]
+
+description = "Programmatically author, schedule and monitor data pipelines"
+readme = { file = "generated/PYPI_README.md", content-type = "text/markdown" }
+license-files.globs = ["LICENSE", "3rd-party-licenses/*.txt"]
+# Mainly because of distutils deprecation and some packages not being compatible with it, we should
+# Limit airflow to < 3.12 until those dependencies are ready and until we can support Python 3.12
+requires-python = "~=3.8,<3.12"
+authors = [
+ { name = "Apache Software Foundation", email = "dev@airflow.apache.org" },
+]
+maintainers = [
+ { name = "Apache Software Foundation", email="dev@airflow.apache.org" },
+]
+keywords = [ "airflow", "orchestration", "workflow", "dag", "pipelines", "automation", "data" ]
+classifiers = [
+ "Development Status :: 5 - Production/Stable",
+ "Environment :: Console",
+ "Environment :: Web Environment",
+ "Framework :: Apache Airflow",
+ "Intended Audience :: Developers",
+ "Intended Audience :: System Administrators",
+ "License :: OSI Approved :: Apache Software License",
+ "Programming Language :: Python :: 3.8",
+ "Programming Language :: Python :: 3.9",
+ "Programming Language :: Python :: 3.10",
+ "Programming Language :: Python :: 3.11",
+ "Topic :: System :: Monitoring",
+]
+dependencies = [
+ # Alembic is important to handle our migrations in predictable and performant way. It is developed
+ # together with SQLAlchemy. Our experience with Alembic is that it very stable in minor version
+ "alembic>=1.6.3, <2.0",
+ "argcomplete>=1.10",
+ "asgiref",
+ "attrs>=22.1.0",
+ "blinker",
+ # Colorlog 6.x merges TTYColoredFormatter into ColoredFormatter, breaking backwards compatibility with 4.x
+ # Update CustomTTYColoredFormatter to remove
+ "colorlog>=4.0.2, <5.0",
+ "configupdater>=3.1.1",
+ # `airflow/www/extensions/init_views` imports `connexion.decorators.validation.RequestBodyValidator`
+ # connexion v3 has refactored the entire module to middleware, see: /spec-first/connexion/issues/1525
+ # Specifically, RequestBodyValidator was removed in: /spec-first/connexion/pull/1595
+ # The usage was added in #30596, seemingly only to override and improve the default error message.
+ # Either revert that change or find another way, preferably without using connexion internals.
+ # This limit can be removed after https://github.com/apache/airflow/issues/35234 is fixed
+ "connexion[flask]>=2.10.0,<3.0",
+ "cron-descriptor>=1.2.24",
+ "croniter>=0.3.17",
+ "cryptography>=0.9.3",
+ "deprecated>=1.2.13",
+ "dill>=0.2.2",
+ "flask-appbuilder==4.3.10",
+ "flask-caching>=1.5.0",
+ "flask-login>=0.6.2",
+ "flask-session>=0.4.0",
+ "flask-wtf>=0.15",
+ # Flask 2.3 is scheduled to introduce a number of deprecation removals - some of them might be breaking
+ # for our dependencies - notably `_app_ctx_stack` and `_request_ctx_stack` removals.
+ # We should remove the limitation after 2.3 is released and our dependencies are updated to handle it
+ "flask>=2.2,<2.3",
+ "fsspec>=2023.10.0",
+ "google-re2>=1.0",
+ "gunicorn>=20.1.0",
+ "httpx",
+ "importlib_metadata>=1.7;python_version<\"3.9\"",
+ "importlib_resources>=5.2;python_version<\"3.9\"",
+ "itsdangerous>=2.0",
+ "jinja2>=3.0.0",
+ "jsonschema>=4.18.0",
+ "lazy-object-proxy",
+ "linkify-it-py>=2.0.0",
+ "lockfile>=0.12.2",
+ "markdown-it-py>=2.1.0",
+ "markdown>=3.0",
+ "markupsafe>=1.1.1",
+ "marshmallow-oneofschema>=2.0.1",
+ "mdit-py-plugins>=0.3.0",
+ "opentelemetry-api>=1.15.0",
+ "opentelemetry-exporter-otlp",
+ "packaging>=14.0",
+ "pathspec>=0.9.0",
+ # When (if) pendulum 3 released it would introduce changes in module/objects imports,
+ # since we are tightly coupled with pendulum library internally it will breaks Airflow functionality.
+ "pendulum>=2.0,<3.0",
+ "pluggy>=1.0",
+ "psutil>=4.2.0",
+ "pydantic>=2.3.0",
+ "pygments>=2.0.1",
+ "pyjwt>=2.0.0",
+ "python-daemon>=3.0.0",
+ "python-dateutil>=2.3",
+ "python-nvd3>=0.15.0",
+ "python-slugify>=5.0",
+ "rfc3339-validator>=0.1.4",
+ "rich-argparse>=1.0.0",
+ "rich>=12.4.4",
+ "setproctitle>=1.1.8",
+ # We use some deprecated features of sqlalchemy 2.0 and we should replace them before we can upgrade
+ # See https://sqlalche.me/e/b8d9 for details of deprecated features
+ # you can set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings.
+ # The issue tracking it is https://github.com/apache/airflow/issues/28723
+ "sqlalchemy>=1.4.28,<2.0",
+ "sqlalchemy-jsonfield>=1.0",
+ "tabulate>=0.7.5",
+ "tenacity>=6.2.0,!=8.2.0",
+ "termcolor>=1.1.0",
+ # We should remove this dependency when Providers are limited to Airflow 2.7+
+ # as we replaced the usage of unicodecsv with csv in Airflow 2.7
+ # See https://github.com/apache/airflow/pull/31693
+ # We should also remove "licenses/LICENSE-unicodecsv.txt" file when we remove this dependency
+ "unicodecsv>=0.14.1",
+ "universal-pathlib>=0.1.4",
+ # Werkzug 3 breaks Flask-Login 0.6.2, also connexion needs to be updated to >= 3.0
+ # we should remove this limitation when FAB supports Flask 2.3 and we migrate connexion to 3+
+ "werkzeug>=2.0,<3",
+]
+
+[project.optional-dependencies]
+# Here manually managed extras start
+# Those extras are manually managed and should be updated when needed
+#
+# START OF core extras
+#
+# This required for AWS deferrable operators.
+# There is conflict between boto3 and aiobotocore dependency botocore.
+# TODO: We can remove it once boto3 and aiobotocore both have compatible botocore version or
+# boto3 have native aync support and we move away from aio aiobotocore
+#
+aiobotocore = [
+ "aiobotocore>=2.7.0",
+]
+async = [
+ "eventlet>=0.33.3",
+ "gevent>=0.13",
+ "greenlet>=0.4.9",
+]
+cgroups = [
+ # Cgroupspy 0.2.2 added Python 3.10 compatibility
+ "cgroupspy>=0.2.2",
+]
+deprecated-api = [
+ "requests>=2.26.0",
+]
+github-enterprise = [
+ "apache-airflow[fab]",
+ "authlib>=1.0.0",
+]
+google-auth = [
+ "apache-airflow[fab]",
+ "authlib>=1.0.0",
+]
+graphviz = [
+ "graphviz>=0.12",
+]
+kerberos = [
+ "pykerberos>=1.1.13",
+ "requests-kerberos>=0.10.0",
+ "thrift-sasl>=0.2.0",
+]
+ldap = [
+ "ldap3>=2.5.1",
+ "python-ldap",
+]
+leveldb = [
+ "plyvel",
+]
+otel = [
+ "opentelemetry-exporter-prometheus",
+]
+pandas = [
+ "pandas>=1.2.5",
+]
+password = [
+ "bcrypt>=2.0.0",
+ "flask-bcrypt>=0.7.1",
+]
+rabbitmq = [
+ "amqp",
+]
+s3fs = [
+ # This is required for support of S3 file system which uses aiobotocore
+ # which can have a conflict with boto3 as mentioned in aiobotocore extra
+ "s3fs>=2023.10.0",
+]
+saml = [
+ # This is required for support of SAML which might be used by some providers (e.g. Amazon)
+ "python3-saml>=1.16.0",
+]
+sentry = [
+ "blinker>=1.1",
+ # Sentry SDK 1.33 is broken when greenlets are installed and fails to import
+ # See https://github.com/getsentry/sentry-python/issues/2473
+ "sentry-sdk>=1.32.0,!=1.33.0",
+]
+statsd = [
+ "statsd>=3.3.0",
+]
+virtualenv = [
+ "virtualenv",
+]
+# END OF core extras
+# START OF Apache no provider extras
+apache-atlas = [
+ "atlasclient>=0.1.2",
+]
+apache-webhdfs = [
+ "hdfs[avro,dataframe,kerberos]>=2.0.4",
+]
+# END OF Apache no provider extras
+all-core = [
+ "apache-airflow[aiobotocore]",
+ "apache-airflow[apache-atlas]",
+ "apache-airflow[async]",
+ "apache-airflow[cgroups]",
+ "apache-airflow[deprecated-api]",
+ "apache-airflow[github-enterprise]",
+ "apache-airflow[google-auth]",
+ "apache-airflow[graphviz]",
+ "apache-airflow[kerberos]",
+ "apache-airflow[ldap]",
+ "apache-airflow[leveldb]",
+ "apache-airflow[otel]",
+ "apache-airflow[pandas]",
+ "apache-airflow[password]",
+ "apache-airflow[rabbitmq]",
+ "apache-airflow[s3fs]",
+ "apache-airflow[saml]",
+ "apache-airflow[sentry]",
+ "apache-airflow[statsd]",
+ "apache-airflow[apache-webhdfs]",
+ "apache-airflow[virtualenv]",
+]
+# START OF devel extras
+devel-debuggers = [
+ "ipdb>=0.13.13",
+]
+devel-devscripts = [
+ "click>=8.0",
+ "gitpython>=3.1.40",
+ "hatch>=1.9.1",
+ "pipdeptree>=2.13.1",
+ "pygithub>=2.1.1",
+ "restructuredtext-lint>=1.4.0",
+ "rich-click>=1.7.0",
+ "semver>=3.0.2",
+ "towncrier>=23.11.0",
+ "twine>=4.0.2",
+]
+devel-duckdb = [
+ "duckdb>=0.9.0",
+]
+# Mypy 0.900 and above ships only with stubs from stdlib so if we need other stubs, we need to install them
+# manually as `types-*`. See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
+# for details. We want to install them explicitly because we want to eventually move to
+# mypyd which does not support installing the types dynamically with --install-types
+devel-mypy = [
+ # TODO: upgrade to newer versions of MyPy continuously as they are released
+ # Make sure to upgrade the mypy version in update-common-sql-api-stubs in .pre-commit-config.yaml
+ # when you upgrade it here !!!!
+ "mypy==1.2.0",
+ "types-Deprecated",
+ "types-Markdown",
+ "types-PyMySQL",
+ "types-PyYAML",
+ "types-aiofiles",
+ "types-certifi",
+ "types-croniter",
+ "types-docutils",
+ "types-paramiko",
+ "types-protobuf",
+ "types-python-dateutil",
+ "types-python-slugify",
+ "types-pytz",
+ "types-redis",
+ "types-requests",
+ "types-setuptools",
+ "types-tabulate",
+ "types-termcolor",
+ "types-toml",
+]
+devel-sentry = [
+ "blinker>=1.7.0",
+]
+devel-static-checks = [
+ "black>=23.12.0",
+ "pre-commit>=3.5.0",
+ "ruff==0.1.11",
+ "yamllint>=1.33.0",
+]
+devel-tests = [
+ "aioresponses>=0.7.6",
+ "backports.zoneinfo>=0.2.1;python_version<'3.9'",
+ "beautifulsoup4>=4.7.1",
+ "coverage>=7.2",
+ "pytest-asyncio>=0.23.3",
+ "pytest-cov>=4.1.0",
+ "pytest-httpx>=0.21.3",
+ "pytest-icdiff>=0.9",
+ "pytest-instafail>=0.5.0",
+ "pytest-mock>=3.12.0",
+ "pytest-rerunfailures>=13.0",
+ "pytest-timeouts>=1.2.1",
+ "pytest-xdist>=3.5.0",
+ "pytest>=7.4.4",
+ "requests_mock>=1.11.0",
+ "time-machine>=2.13.0",
+]
+# END OF devel extras
+# START OF doc extras
+doc = [
+ "astroid>=2.12.3,<3.0",
+ "checksumdir>=1.2.0",
+ # click 8.1.4 and 8.1.5 generate mypy errors due to typing issue in the upstream package:
+ # https://github.com/pallets/click/issues/2558
+ "click>=8.0,!=8.1.4,!=8.1.5",
+ # Docutils 0.17.0 converts generated <div class="section"> into <section> and breaks our doc formatting
+ # By adding a lot of whitespace separation. This limit can be lifted when we update our doc to handle
+ # <section> tags for sections
+ "docutils<0.17,>=0.16",
+ "sphinx-airflow-theme>=0.0.12",
+ "sphinx-argparse>=0.4.0",
+ # sphinx-autoapi fails with astroid 3.0, see: https://github.com/readthedocs/sphinx-autoapi/issues/407
+ # This was fixed in sphinx-autoapi 3.0, however it has requirement sphinx>=6.1, but we stuck on 5.x
+ "sphinx-autoapi>=2.1.1",
+ "sphinx-copybutton>=0.5.2",
+ "sphinx-design>=0.5.0",
+ "sphinx-jinja>=2.0.2",
+ "sphinx-rtd-theme>=2.0.0",
+ # Currently we are using sphinx 5 but we need to migrate to Sphinx 7
+ "sphinx>=5.3.0,<6.0.0",
+ "sphinxcontrib-applehelp>=1.0.4",
+ "sphinxcontrib-devhelp>=1.0.2",
+ "sphinxcontrib-htmlhelp>=2.0.1",
+ "sphinxcontrib-httpdomain>=1.8.1",
+ "sphinxcontrib-jquery>=4.1",
+ "sphinxcontrib-jsmath>=1.0.1",
+ "sphinxcontrib-qthelp>=1.0.3",
+ "sphinxcontrib-redoc>=1.6.0",
+ "sphinxcontrib-serializinghtml==1.1.5",
+ "sphinxcontrib-spelling>=8.0.0",
+]
+doc-gen = [
+ "apache-airflow[doc]",
+ "eralchemy2>=1.3.8",
+]
+# END OF doc extras
+# START OF bundle extras
+all-dbs = [
+ "apache-airflow[apache-cassandra]",
+ "apache-airflow[apache-drill]",
+ "apache-airflow[apache-druid]",
+ "apache-airflow[apache-hdfs]",
+ "apache-airflow[apache-hive]",
+ "apache-airflow[apache-impala]",
+ "apache-airflow[apache-pinot]",
+ "apache-airflow[arangodb]",
+ "apache-airflow[cloudant]",
+ "apache-airflow[databricks]",
+ "apache-airflow[exasol]",
+ "apache-airflow[influxdb]",
+ "apache-airflow[microsoft-mssql]",
+ "apache-airflow[mongo]",
+ "apache-airflow[mysql]",
+ "apache-airflow[neo4j]",
+ "apache-airflow[postgres]",
+ "apache-airflow[presto]",
+ "apache-airflow[trino]",
+ "apache-airflow[vertica]",
+]
+devel = [
+ "apache-airflow[common.io]",
+ "apache-airflow[common.sql]",
+ "apache-airflow[devel-debuggers]",
+ "apache-airflow[devel-devscripts]",
+ "apache-airflow[devel-duckdb]",
+ "apache-airflow[devel-mypy]",
+ "apache-airflow[devel-sentry]",
+ "apache-airflow[devel-static-checks]",
+ "apache-airflow[devel-tests]",
+ "apache-airflow[fab]",
+ "apache-airflow[ftp]",
+ "apache-airflow[http]",
+ "apache-airflow[imap]",
+ "apache-airflow[sqlite]",
+]
+devel-all-dbs = [
+ "apache-airflow[apache-cassandra]",
+ "apache-airflow[apache-drill]",
+ "apache-airflow[apache-druid]",
+ "apache-airflow[apache-hdfs]",
+ "apache-airflow[apache-hive]",
+ "apache-airflow[apache-impala]",
+ "apache-airflow[apache-pinot]",
+ "apache-airflow[arangodb]",
+ "apache-airflow[cloudant]",
+ "apache-airflow[databricks]",
+ "apache-airflow[exasol]",
+ "apache-airflow[influxdb]",
+ "apache-airflow[microsoft-mssql]",
+ "apache-airflow[mongo]",
+ "apache-airflow[mysql]",
+ "apache-airflow[neo4j]",
+ "apache-airflow[postgres]",
+ "apache-airflow[presto]",
+ "apache-airflow[trino]",
+ "apache-airflow[vertica]",
+]
+devel-ci = [
+ "apache-airflow[devel-all]",
+]
+devel-hadoop = [
+ "apache-airflow[apache-hdfs]",
+ "apache-airflow[apache-hive]",
+ "apache-airflow[apache-impala]",
+ "apache-airflow[devel]",
+ "apache-airflow[hdfs]",
+ "apache-airflow[kerberos]",
+ "apache-airflow[presto]",
+]
+# END OF bundle extras
+#############################################################################################################
+# The whole section can be removed in Airflow 3.0 as those old aliases are deprecated in 2.* series
+#############################################################################################################
+# START OF deprecated extras
+atlas = [
+ "apache-airflow[apache-atlas]",
+]
+aws = [
+ "apache-airflow[amazon]",
+]
+azure = [
+ "apache-airflow[microsoft-azure]",
+]
+cassandra = [
+ "apache-airflow[apache-cassandra]",
+]
+# Empty alias extra just for backward compatibility with Airflow 1.10
+crypto = [
+]
+druid = [
+ "pache-airflow[apache-druid]",
+]
+gcp = [
+ "apache-airflow[google]",
+]
+gcp_api = [
+ "apache-airflow[google]",
+]
+hdfs = [
+ "apache-airflow[apache-hdfs]",
+]
+hive = [
+ "apache-airflow[apache-hive]",
+]
+kubernetes = [
+ "apache-airflow[cncf-kubernetes]",
+]
+mssql = [
+ "apache-airflow[microsoft-mssql]",
+]
+pinot = [
+ "apache-airflow[apache-pinot]",
+]
+s3 = [
+ "apache-airflow[amazon]",
+]
+spark = [
+ "apache-airflow[apache-spark]",
+]
+webhdfs = [
+ "apache-airflow[apache-webhdfs]",
+]
+winrm = [
+ "apache-airflow[microsoft-winrm]",
+]
+# END OF deprecated extras
+#############################################################################################################
+# The whole section below is automatically generated by `update-providers-dependencies` pre-commit based
+# on `provider.yaml` files present in the `providers` subdirectories. The `provider.yaml` files are
+# A single source of truth for provider dependencies,
+#
+# PLEASE DO NOT MODIFY THIS SECTION MANUALLY. IT WILL BE OVERWRITTEN BY PRE-COMMIT !!
+# If you want to modify these - modify the corresponding provider.yaml instead.
+#############################################################################################################
+# START OF GENERATED DEPENDENCIES
+airbyte = [
+ "apache-airflow[http]",
+]
+alibaba = [
+ "alibabacloud_adb20211201>=1.0.0",
+ "alibabacloud_tea_openapi>=0.3.7",
+ "oss2>=2.14.0",
+]
+amazon = [
+ "apache-airflow[common_sql]",
+ "apache-airflow[http]",
+ "asgiref",
+ "boto3>=1.33.0",
+ "botocore>=1.33.0",
+ "inflection>=0.5.1",
+ "jsonpath_ng>=1.5.3",
+ "redshift_connector>=2.0.918",
+ "sqlalchemy_redshift>=0.8.6",
+ "watchtower>=2.0.1,<4",
+ # Devel dependencies for the amazon provider
+ "aiobotocore>=2.7.0",
+ "aws_xray_sdk>=2.12.0",
+ "moto[cloudformation,glue]>=4.2.12",
+ "mypy-boto3-appflow>=1.33.0",
+ "mypy-boto3-rds>=1.33.0",
+ "mypy-boto3-redshift-data>=1.33.0",
+ "mypy-boto3-s3>=1.33.0",
+ "s3fs>=2023.10.0",
+]
+apache-beam = [
+ "apache-beam>=2.53.0",
+ "pyarrow>=14.0.1",
+]
+apache-cassandra = [
+ "cassandra-driver>=3.13.0",
+]
+apache-drill = [
+ "apache-airflow[common_sql]",
+ "sqlalchemy-drill>=1.1.0",
+]
+apache-druid = [
+ "apache-airflow[common_sql]",
+ "pydruid>=0.4.1",
+]
+apache-flink = [
+ "apache-airflow[cncf_kubernetes]",
+ "cryptography>=2.0.0",
+]
+apache-hdfs = [
+ "hdfs[avro,dataframe,kerberos]>=2.0.4",
+]
+apache-hive = [
+ "apache-airflow[common_sql]",
+ "hmsclient>=0.1.0",
+ "pandas>=1.2.5",
+ "pyhive[hive-pure-sasl]>=0.7.0",
+ "thrift>=0.9.2",
+]
+apache-impala = [
+ "impyla>=0.18.0,<1.0",
+]
+apache-kafka = [
+ "asgiref",
+ "confluent-kafka>=1.8.2",
+]
+apache-kylin = [
+ "kylinpy>=2.6",
+]
+apache-livy = [
+ "aiohttp",
+ "apache-airflow[http]",
+ "asgiref",
+]
+apache-pig = [
+]
+apache-pinot = [
+ "apache-airflow[common_sql]",
+ "pinotdb>0.4.7",
+]
+apache-spark = [
+ "grpcio-status>=1.59.0",
+ "pyspark",
+]
+apprise = [
+ "apprise",
+]
+arangodb = [
+ "python-arango>=7.3.2",
+]
+asana = [
+ "asana>=0.10,<4.0.0",
+]
+atlassian-jira = [
+ "atlassian-python-api>=1.14.2",
+]
+celery = [
+ "celery>=5.3.0,<6,!=5.3.3,!=5.3.2",
+ "flower>=1.0.0",
+ "google-re2>=1.0",
+]
+cloudant = [
+ "cloudant>=2.0",
+]
+cncf-kubernetes = [
+ "aiofiles>=23.2.0",
+ "asgiref>=3.5.2",
+ "cryptography>=2.0.0",
+ "google-re2>=1.0",
+ "kubernetes>=21.7.0,<24",
+ "kubernetes_asyncio>=18.20.1,<25",
+]
+cohere = [
+ "cohere>=4.37",
+]
+common-io = [
+]
+common-sql = [
+ "sqlparse>=0.4.2",
+]
+databricks = [
+ "aiohttp>=3.6.3, <4",
+ "apache-airflow[common_sql]",
+ "databricks-sql-connector>=2.0.0, <3.0.0, !=2.9.0",
+ "requests>=2.27,<3",
+ # Devel dependencies for the databricks provider
+ "deltalake>=0.12.0",
+]
+datadog = [
+ "datadog>=0.14.0",
+]
+dbt-cloud = [
+ "aiohttp",
+ "apache-airflow[http]",
+ "asgiref",
+]
+dingding = [
+ "apache-airflow[http]",
+]
+discord = [
+ "apache-airflow[http]",
+]
+docker = [
+ "docker>=5.0.3",
+ "python-dotenv>=0.21.0",
+]
+elasticsearch = [
+ "apache-airflow[common_sql]",
+ "elasticsearch>=8.10,<9",
+]
+exasol = [
+ "apache-airflow[common_sql]",
+ "pandas>=1.2.5",
+ "pyexasol>=0.5.1",
+]
+facebook = [
+ "facebook-business>=6.0.2",
+]
+ftp = [
+]
+github = [
+ "PyGithub!=1.58",
+]
+google = [
+ "PyOpenSSL",
+ "apache-airflow[common_sql]",
+ "asgiref>=3.5.2",
+ "gcloud-aio-auth>=4.0.0,<5.0.0",
+ "gcloud-aio-bigquery>=6.1.2",
+ "gcloud-aio-storage>=9.0.0",
+ "gcsfs>=2023.10.0",
+ "google-ads>=22.1.0",
+ "google-analytics-admin",
+ "google-api-core>=2.11.0",
+ "google-api-python-client>=1.6.0",
+ "google-auth-httplib2>=0.0.1",
+ "google-auth>=1.0.0",
+ "google-cloud-aiplatform>=1.22.1",
+ "google-cloud-automl>=2.12.0",
+ "google-cloud-batch>=0.13.0",
+ "google-cloud-bigquery-datatransfer>=3.13.0",
+ "google-cloud-bigtable>=2.17.0",
+ "google-cloud-build>=3.22.0",
+ "google-cloud-compute>=1.10.0",
+ "google-cloud-container>=2.17.4",
+ "google-cloud-datacatalog>=3.11.1",
+ "google-cloud-dataflow-client>=0.8.6",
+ "google-cloud-dataform>=0.5.0",
+ "google-cloud-dataplex>=1.10.0",
+ "google-cloud-dataproc-metastore>=1.12.0",
+ "google-cloud-dataproc>=5.8.0",
+ "google-cloud-dlp>=3.12.0",
+ "google-cloud-kms>=2.15.0",
+ "google-cloud-language>=2.9.0",
+ "google-cloud-logging>=3.5.0",
+ "google-cloud-memcache>=1.7.0",
+ "google-cloud-monitoring>=2.18.0",
+ "google-cloud-orchestration-airflow>=1.10.0",
+ "google-cloud-os-login>=2.9.1",
+ "google-cloud-pubsub>=2.19.0",
+ "google-cloud-redis>=2.12.0",
+ "google-cloud-run>=0.9.0",
+ "google-cloud-secret-manager>=2.16.0",
+ "google-cloud-spanner>=3.11.1",
+ "google-cloud-speech>=2.18.0",
+ "google-cloud-storage-transfer>=1.4.1",
+ "google-cloud-storage>=2.7.0",
+ "google-cloud-tasks>=2.13.0",
+ "google-cloud-texttospeech>=2.14.1",
+ "google-cloud-translate>=3.11.0",
+ "google-cloud-videointelligence>=2.11.0",
+ "google-cloud-vision>=3.4.0",
+ "google-cloud-workflows>=1.10.0",
+ "grpcio-gcp>=0.2.2",
+ "httpx",
+ "json-merge-patch>=0.2",
+ "looker-sdk>=22.2.0",
+ "pandas-gbq",
+ "pandas>=1.2.5",
+ "proto-plus>=1.19.6",
+ "sqlalchemy-bigquery>=1.2.1",
+ "sqlalchemy-spanner>=1.6.2",
+]
+grpc = [
+ "google-auth-httplib2>=0.0.1",
+ "google-auth>=1.0.0, <3.0.0",
+ "grpcio>=1.15.0",
+]
+hashicorp = [
+ "hvac>=1.1.0",
+]
+http = [
+ "aiohttp",
+ "asgiref",
+ "requests>=2.26.0",
+ "requests_toolbelt",
+]
+imap = [
+]
+influxdb = [
+ "influxdb-client>=1.19.0",
+ "requests>=2.26.0",
+]
+jdbc = [
+ "apache-airflow[common_sql]",
+ "jaydebeapi>=1.1.1",
+]
+jenkins = [
+ "python-jenkins>=1.0.0",
+]
+microsoft-azure = [
+ "adal>=1.2.7",
+ "adlfs>=2023.10.0",
+ "azure-batch>=8.0.0",
+ "azure-cosmos>=4.0.0",
+ "azure-datalake-store>=0.0.45",
+ "azure-identity>=1.3.1",
+ "azure-keyvault-secrets>=4.1.0",
+ "azure-kusto-data>=4.1.0",
+ "azure-mgmt-containerinstance>=9.0.0",
+ "azure-mgmt-containerregistry>=8.0.0",
+ "azure-mgmt-cosmosdb",
+ "azure-mgmt-datafactory>=2.0.0",
+ "azure-mgmt-datalake-store>=0.5.0",
+ "azure-mgmt-resource>=2.2.0",
+ "azure-mgmt-storage>=16.0.0",
+ "azure-servicebus>=7.6.1",
+ "azure-storage-blob>=12.14.0",
+ "azure-storage-file-datalake>=12.9.1",
+ "azure-storage-file-share",
+ "azure-synapse-artifacts>=0.17.0",
+ "azure-synapse-spark",
+ # Devel dependencies for the microsoft.azure provider
+ "pywinrm",
+]
+microsoft-mssql = [
+ "apache-airflow[common_sql]",
+ "pymssql>=2.1.8",
+]
+microsoft-psrp = [
+ "pypsrp>=0.8.0",
+]
+microsoft-winrm = [
+ "pywinrm>=0.4",
+]
+mongo = [
+ "dnspython>=1.13.0",
+ "pymongo>=3.6.0",
+ # Devel dependencies for the mongo provider
+ "mongomock",
+]
+mysql = [
+ "apache-airflow[common_sql]",
+ "mysql-connector-python>=8.0.29",
+ "mysqlclient>=1.3.6",
+]
+neo4j = [
+ "neo4j>=4.2.1",
+]
+odbc = [
+ "apache-airflow[common_sql]",
+ "pyodbc",
+]
+openai = [
+ "openai[datalib]>=1.0",
+]
+openfaas = [
+]
+openlineage = [
+ "apache-airflow[common_sql]",
+ "attrs>=22.2",
+ "openlineage-integration-common>=0.28.0",
+ "openlineage-python>=0.28.0",
+]
+opensearch = [
+ "opensearch-py>=2.2.0",
+]
+opsgenie = [
+ "opsgenie-sdk>=2.1.5",
+]
+oracle = [
+ "apache-airflow[common_sql]",
+ "oracledb>=1.0.0",
+]
+pagerduty = [
+ "pdpyras>=4.1.2",
+]
+papermill = [
+ "ipykernel",
+ "papermill[all]>=2.4.0",
+ "scrapbook[all]",
+]
+pgvector = [
+ "apache-airflow[postgres]",
+ "pgvector>=0.2.3",
+]
+pinecone = [
+ "pinecone-client>=2.2.4",
+]
+postgres = [
+ "apache-airflow[common_sql]",
+ "psycopg2-binary>=2.8.0",
+]
+presto = [
+ "apache-airflow[common_sql]",
+ "pandas>=1.2.5",
+ "presto-python-client>=0.8.4",
+]
+redis = [
+ "redis>=4.5.2,<5.0.0,!=4.5.5",
+]
+salesforce = [
+ "pandas>=1.2.5",
+ "simple-salesforce>=1.0.0",
+]
+samba = [
+ "smbprotocol>=1.5.0",
+]
+segment = [
+ "analytics-python>=1.2.9",
+]
+sendgrid = [
+ "sendgrid>=6.0.0",
+]
+sftp = [
+ "apache-airflow[ssh]",
+ "paramiko>=2.8.0",
+]
+singularity = [
+ "spython>=0.0.56",
+]
+slack = [
+ "apache-airflow[common_sql]",
+ "slack_sdk>=3.0.0",
+]
+smtp = [
+]
+snowflake = [
+ "apache-airflow[common_sql]",
+ "snowflake-connector-python>=2.7.8",
+ "snowflake-sqlalchemy>=1.1.0",
+]
+sqlite = [
+ "apache-airflow[common_sql]",
+]
+ssh = [
+ "paramiko>=2.6.0",
+ "sshtunnel>=0.3.2",
+]
+tableau = [
+ "tableauserverclient",
+]
+tabular = [
+ # Devel dependencies for the tabular provider
+ "pyiceberg>=0.5.0",
+]
+telegram = [
+ "python-telegram-bot>=20.2",
+]
+trino = [
+ "apache-airflow[common_sql]",
+ "pandas>=1.2.5",
+ "trino>=0.318.0",
+]
+vertica = [
+ "apache-airflow[common_sql]",
+ "vertica-python>=0.5.1",
+]
+weaviate = [
+ "pandas>=1.2.5",
+ "weaviate-client>=3.24.2",
+]
+yandex = [
+ "yandexcloud>=0.228.0",
+]
+zendesk = [
+ "zenpy>=2.0.24",
+]
+all = [
+ # core extras
+ "apache-airflow[aiobotocore]",
+ "apache-airflow[async]",
+ "apache-airflow[cgroups]",
+ "apache-airflow[deprecated-api]",
+ "apache-airflow[github-enterprise]",
+ "apache-airflow[google-auth]",
+ "apache-airflow[graphviz]",
+ "apache-airflow[kerberos]",
+ "apache-airflow[ldap]",
+ "apache-airflow[leveldb]",
+ "apache-airflow[otel]",
+ "apache-airflow[pandas]",
+ "apache-airflow[password]",
+ "apache-airflow[rabbitmq]",
+ "apache-airflow[s3fs]",
+ "apache-airflow[saml]",
+ "apache-airflow[sentry]",
+ "apache-airflow[statsd]",
+ "apache-airflow[virtualenv]",
+ # Apache no provider extras
+ "apache-airflow[apache-atlas]",
+ "apache-airflow[apache-webhdfs]",
+ "apache-airflow[all-core]",
+ # Provider extras
+ "apache-airflow[airbyte]",
+ "apache-airflow[alibaba]",
+ "apache-airflow[amazon]",
+ "apache-airflow[apache-beam]",
+ "apache-airflow[apache-cassandra]",
+ "apache-airflow[apache-drill]",
+ "apache-airflow[apache-druid]",
+ "apache-airflow[apache-flink]",
+ "apache-airflow[apache-hdfs]",
+ "apache-airflow[apache-hive]",
+ "apache-airflow[apache-impala]",
+ "apache-airflow[apache-kafka]",
+ "apache-airflow[apache-kylin]",
+ "apache-airflow[apache-livy]",
+ "apache-airflow[apache-pig]",
+ "apache-airflow[apache-pinot]",
+ "apache-airflow[apache-spark]",
+ "apache-airflow[apache-sqoop]",
+ "apache-airflow[apprise]",
+ "apache-airflow[arangodb]",
+ "apache-airflow[asana]",
+ "apache-airflow[atlassian-jira]",
+ "apache-airflow[celery]",
+ "apache-airflow[cloudant]",
+ "apache-airflow[cncf-kubernetes]",
+ "apache-airflow[cohere]",
+ "apache-airflow[common-io]",
+ "apache-airflow[common-sql]",
+ "apache-airflow[daskexecutor]",
+ "apache-airflow[databricks]",
+ "apache-airflow[datadog]",
+ "apache-airflow[dbt-cloud]",
+ "apache-airflow[dingding]",
+ "apache-airflow[discord]",
+ "apache-airflow[docker]",
+ "apache-airflow[elasticsearch]",
+ "apache-airflow[exasol]",
+ "apache-airflow[facebook]",
+ "apache-airflow[ftp]",
+ "apache-airflow[github]",
+ "apache-airflow[google]",
+ "apache-airflow[grpc]",
+ "apache-airflow[hashicorp]",
+ "apache-airflow[http]",
+ "apache-airflow[imap]",
+ "apache-airflow[influxdb]",
+ "apache-airflow[jdbc]",
+ "apache-airflow[jenkins]",
+ "apache-airflow[microsoft-azure]",
+ "apache-airflow[microsoft-mssql]",
+ "apache-airflow[microsoft-psrp]",
+ "apache-airflow[microsoft-winrm]",
+ "apache-airflow[mongo]",
+ "apache-airflow[mysql]",
+ "apache-airflow[neo4j]",
+ "apache-airflow[odbc]",
+ "apache-airflow[openai]",
+ "apache-airflow[openfaas]",
+ "apache-airflow[openlineage]",
+ "apache-airflow[opensearch]",
+ "apache-airflow[opsgenie]",
+ "apache-airflow[oracle]",
+ "apache-airflow[pagerduty]",
+ "apache-airflow[papermill]",
+ "apache-airflow[pgvector]",
+ "apache-airflow[pinecone]",
+ "apache-airflow[plexus]",
+ "apache-airflow[postgres]",
+ "apache-airflow[presto]",
+ "apache-airflow[redis]",
+ "apache-airflow[salesforce]",
+ "apache-airflow[samba]",
+ "apache-airflow[segment]",
+ "apache-airflow[sendgrid]",
+ "apache-airflow[sftp]",
+ "apache-airflow[singularity]",
+ "apache-airflow[slack]",
+ "apache-airflow[smtp]",
+ "apache-airflow[snowflake]",
+ "apache-airflow[sqlite]",
+ "apache-airflow[ssh]",
+ "apache-airflow[tableau]",
+ "apache-airflow[tabular]",
+ "apache-airflow[telegram]",
+ "apache-airflow[trino]",
+ "apache-airflow[vertica]",
+ "apache-airflow[weaviate]",
+ "apache-airflow[yandex]",
+ "apache-airflow[zendesk]",
+]
+devel-all = [
+ "apache-airflow[all]",
+ "apache-airflow[devel]",
+ "apache-airflow[doc]",
+ "apache-airflow[doc-gen]",
+ "apache-airflow[saml]",
+ # Apache no provider extras
+ "apache-airflow[apache-atlas]",
+ "apache-airflow[apache-webhdfs]",
+ "apache-airflow[all-core]",
+ # Include all provider deps
+ "apache-airflow[airbyte]",
+ "apache-airflow[alibaba]",
+ "apache-airflow[amazon]",
+ "apache-airflow[apache-beam]",
+ "apache-airflow[apache-cassandra]",
+ "apache-airflow[apache-drill]",
+ "apache-airflow[apache-druid]",
+ "apache-airflow[apache-flink]",
+ "apache-airflow[apache-hdfs]",
+ "apache-airflow[apache-hive]",
+ "apache-airflow[apache-impala]",
+ "apache-airflow[apache-kafka]",
+ "apache-airflow[apache-kylin]",
+ "apache-airflow[apache-livy]",
+ "apache-airflow[apache-pig]",
+ "apache-airflow[apache-pinot]",
+ "apache-airflow[apache-spark]",
+ "apache-airflow[apache-sqoop]",
+ "apache-airflow[apprise]",
+ "apache-airflow[arangodb]",
+ "apache-airflow[asana]",
+ "apache-airflow[atlassian-jira]",
+ "apache-airflow[celery]",
+ "apache-airflow[cloudant]",
+ "apache-airflow[cncf-kubernetes]",
+ "apache-airflow[cohere]",
+ "apache-airflow[common-io]",
+ "apache-airflow[common-sql]",
+ "apache-airflow[daskexecutor]",
+ "apache-airflow[databricks]",
+ "apache-airflow[datadog]",
+ "apache-airflow[dbt-cloud]",
+ "apache-airflow[dingding]",
+ "apache-airflow[discord]",
+ "apache-airflow[docker]",
+ "apache-airflow[elasticsearch]",
+ "apache-airflow[exasol]",
+ "apache-airflow[facebook]",
+ "apache-airflow[ftp]",
+ "apache-airflow[github]",
+ "apache-airflow[google]",
+ "apache-airflow[grpc]",
+ "apache-airflow[hashicorp]",
+ "apache-airflow[http]",
+ "apache-airflow[imap]",
+ "apache-airflow[influxdb]",
+ "apache-airflow[jdbc]",
+ "apache-airflow[jenkins]",
+ "apache-airflow[microsoft-azure]",
+ "apache-airflow[microsoft-mssql]",
+ "apache-airflow[microsoft-psrp]",
+ "apache-airflow[microsoft-winrm]",
+ "apache-airflow[mongo]",
+ "apache-airflow[mysql]",
+ "apache-airflow[neo4j]",
+ "apache-airflow[odbc]",
+ "apache-airflow[openai]",
+ "apache-airflow[openfaas]",
+ "apache-airflow[openlineage]",
+ "apache-airflow[opensearch]",
+ "apache-airflow[opsgenie]",
+ "apache-airflow[oracle]",
+ "apache-airflow[pagerduty]",
+ "apache-airflow[papermill]",
+ "apache-airflow[pgvector]",
+ "apache-airflow[pinecone]",
+ "apache-airflow[plexus]",
+ "apache-airflow[postgres]",
+ "apache-airflow[presto]",
+ "apache-airflow[redis]",
+ "apache-airflow[salesforce]",
+ "apache-airflow[samba]",
+ "apache-airflow[segment]",
+ "apache-airflow[sendgrid]",
+ "apache-airflow[sftp]",
+ "apache-airflow[singularity]",
+ "apache-airflow[slack]",
+ "apache-airflow[smtp]",
+ "apache-airflow[snowflake]",
+ "apache-airflow[sqlite]",
+ "apache-airflow[ssh]",
+ "apache-airflow[tableau]",
+ "apache-airflow[tabular]",
+ "apache-airflow[telegram]",
+ "apache-airflow[trino]",
+ "apache-airflow[vertica]",
+ "apache-airflow[weaviate]",
+ "apache-airflow[yandex]",
+ "apache-airflow[zendesk]",
+]
+# END OF GENERATED DEPENDENCIES
+#############################################################################################################
+# The rest of the pyproject.toml file should be manually maintained
+#############################################################################################################
+[project.scripts]
+airflow = "airflow.__main__:main"
+[project.urls]
+"Bug Tracker" = "https://github.com/apache/airflow/issues"
+Documentation = "https://airflow.apache.org/docs/"
+Downloads = "https://archive.apache.org/dist/airflow/"
+Homepage = "https://airflow.apache.org/"
+"Release Notes" = "https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html"
+"Slack Chat" = "https://s.apache.org/airflow-slack"
+"Source Code" = "https://github.com/apache/airflow"
+Twitter = "https://twitter.com/ApacheAirflow"
+YouTube = "https://www.youtube.com/channel/UCSXwxpWZQ7XZ1WL3wqevChA/"
+
+[tool.hatch.envs.default]
+python = "3.8"
+platforms = ["linux", "macos"]
+description = "Default environment with Python 3.8 for maximum compatibility"
+features = ["devel"]
+
+[tool.hatch.envs.airflow-38]
+python = "3.8"
+platforms = ["linux", "macos"]
+description = "Environment with Python 3.8. No devel installed."
+features = []
+
+[tool.hatch.envs.airflow-39]
+python = "3.9"
+platforms = ["linux", "macos"]
+description = "Environment with Python 3.9. No devel installed."
+features = []
+
+[tool.hatch.envs.airflow-310]
+python = "3.10"
+platforms = ["linux", "macos"]
+description = "Environment with Python 3.10. No devel installed."
+features = []
+
+[tool.hatch.envs.airflow-311]
+python = "3.11"
+platforms = ["linux", "macos"]
+description = "Environment with Python 3.11. No devel installed"
+features = []
+
+[tool.hatch.version]
+path = "airflow/__init__.py"
+
+[tool.hatch.build.targets.wheel.hooks.custom]
+path = "./dev/hatch_build.py"
+
+[tool.hatch.build.hooks.custom]
+path = "./dev/hatch_build.py"
+
+[tool.hatch.build.targets.custom]
+path = "./dev/hatch_build.py"
+
+[tool.hatch.build.targets.sdist]
+include = [
+ "/airflow",
+ "/airflow/git_version"
+]
+exclude = [
+ "/airflow/providers/",
+ "/airflow/www/node_modules/"
+]
+artifacts = [
+ "/airflow/www/static/dist/",
+ "/airflow/git_version",
+ "/generated/"
+]
+
+
+[tool.hatch.build.targets.wheel]
+include = [
+ "/airflow",
+]
+exclude = [
+ "/airflow/providers/",
+]
+artifacts = [
+ "/airflow/www/static/dist/",
+ "/airflow/git_version"
+]
+
+
[tool.black]
line-length = 110
target-version = ['py38', 'py39', 'py310', 'py311']
-# Editable installs are currently broken using setuptools 64.0.0 and above. The problem is tracked in
-# https://github.com/pypa/setuptools/issues/3548. We're also discussing how we could potentially fix
-# this problem on our end in issue https://github.com/apache/airflow/issues/30764. Until then we need
-# to use one of the following workarounds locally for editable installs:
-# 1) Pin setuptools <= 63.4.3 below in the [build-system] section.
-# 2) Include your airflow source code directory in PYTHONPATH.
-[build-system]
-requires = ['setuptools==67.2.0']
-build-backend = "setuptools.build_meta"
-
[tool.ruff]
target-version = "py38"
typing-modules = ["airflow.typing_compat"]
@@ -47,7 +1272,6 @@ extend-select = [
"I", # Missing required import (auto-fixable)
"UP", # Pyupgrade
"RUF100", # Unused noqa (auto-fixable)
-
# We ignore more pydocstyle than we enable, so be more selective at what we enable
"D101",
"D106",
@@ -188,3 +1412,45 @@ exclude_also = [
"@(typing(_extensions)?\\.)?overload",
"if (typing(_extensions)?\\.)?TYPE_CHECKING:"
]
+
+[tool.mypy]
+ignore_missing_imports = true
+no_implicit_optional = true
+warn_redundant_casts = true
+warn_unused_ignores = false
+plugins = [
+ "dev/mypy/plugin/decorators.py",
+ "dev/mypy/plugin/outputs.py",
+]
+pretty = true
+show_error_codes = true
+disable_error_code = [
+ "annotation-unchecked",
+]
+
+[[tool.mypy.overrides]]
+module="airflow.config_templates.default_webserver_config"
+disable_error_code = [
+ "var-annotated",
+]
+
+[[tool.mypy.overrides]]
+module="airflow.migrations.*"
+ignore_errors = true
+
+[[tool.mypy.overrides]]
+module= [
+ "google.cloud.*",
+ "azure.*",
+]
+no_implicit_optional = false
+
+[[tool.mypy.overrides]]
+module=[
+ "referencing.*",
+ # Beam has some old type annotations, and they introduced an error recently with bad signature of
+ # a function. This is captured in https://github.com/apache/beam/issues/29927
+ # and we should remove this exclusion when it is fixed.
+ "apache_beam.*"
+]
+ignore_errors = true
diff --git a/scripts/ci/docker-compose/devcontainer.env b/scripts/ci/docker-compose/devcontainer.env
index 465e7ab13f..4ef3fd0459 100644
--- a/scripts/ci/docker-compose/devcontainer.env
+++ b/scripts/ci/docker-compose/devcontainer.env
@@ -46,7 +46,6 @@ HOST_OS="linux"
INIT_SCRIPT_FILE="init.sh"
INSTALL_AIRFLOW_VERSION=
AIRFLOW_CONSTRAINTS_MODE=
-INSTALL_PROVIDERS_FROM_SOURCES=
INSTALL_SELECTED_PROVIDERS=
USE_AIRFLOW_VERSION=
USE_PACKAGES_FROM_DIST=
diff --git a/scripts/ci/docker-compose/local.yml b/scripts/ci/docker-compose/local.yml
index 1ee86fc773..9258b3b504 100644
--- a/scripts/ci/docker-compose/local.yml
+++ b/scripts/ci/docker-compose/local.yml
@@ -55,9 +55,6 @@ services:
- type: bind
source: ../../../LICENSE
target: /opt/airflow/LICENSE
- - type: bind
- source: ../../../MANIFEST.in
- target: /opt/airflow/MANIFEST.in
- type: bind
source: ../../../NOTICE
target: /opt/airflow/NOTICE
@@ -100,12 +97,6 @@ services:
- type: bind
source: ../../../scripts/docker/entrypoint_ci.sh
target: /entrypoint
- - type: bind
- source: ../../../setup.cfg
- target: /opt/airflow/setup.cfg
- - type: bind
- source: ../../../setup.py
- target: /opt/airflow/setup.py
- type: bind
source: ../../../tests
target: /opt/airflow/tests
diff --git a/scripts/ci/kubernetes/k8s_requirements.txt b/scripts/ci/kubernetes/k8s_requirements.txt
index 0f6cc2fd38..bba6871b51 100644
--- a/scripts/ci/kubernetes/k8s_requirements.txt
+++ b/scripts/ci/kubernetes/k8s_requirements.txt
@@ -1,4 +1,4 @@
--e .[kubernetes]
+-e .[cncf.kubernetes]
pytest
pytest-cov
pytest-instafail
diff --git a/scripts/ci/pre_commit/pre_commit_check_extra_packages_ref.py b/scripts/ci/pre_commit/pre_commit_check_extra_packages_ref.py
new file mode 100755
index 0000000000..1dbd2d9fbe
--- /dev/null
+++ b/scripts/ci/pre_commit/pre_commit_check_extra_packages_ref.py
@@ -0,0 +1,97 @@
+#!/usr/bin/env python
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Checks if all the libraries in setup.py are listed in installation.rst file
+"""
+from __future__ import annotations
+
+import re
+import sys
+from pathlib import Path
+
+from tabulate import tabulate
+
+# tomllib is available in Python 3.11+ and before that tomli offers same interface for parsing TOML files
+try:
+ import tomllib
+except ImportError:
+ import tomli as tomllib
+
+
+AIRFLOW_ROOT_PATH = Path(__file__).parents[3].resolve()
+EXTRA_PACKAGES_REF_FILE = AIRFLOW_ROOT_PATH / "docs" / "apache-airflow" / "extra-packages-ref.rst"
+PYPROJECT_TOML_FILE_PATH = AIRFLOW_ROOT_PATH / "pyproject.toml"
+
+sys.path.insert(0, str(Path(__file__).parent.resolve())) # make sure common_precommit_utils is imported
+
+from common_precommit_utils import console
+
+pyproject_toml_content = tomllib.loads(PYPROJECT_TOML_FILE_PATH.read_text())
+
+optional_dependencies: dict[str, list[str]] = pyproject_toml_content["project"]["optional-dependencies"]
+doc_ref_content = EXTRA_PACKAGES_REF_FILE.read_text()
+
+errors: list[str] = []
+regular_suggestions: list[str] = []
+devel_suggestions: list[str] = []
+suggestions: list[tuple] = []
+suggestions_devel: list[tuple] = []
+suggestions_providers: list[tuple] = []
+
+for dependency in optional_dependencies:
+ console.print(f"[bright_blue]Checking if {dependency} is mentioned in refs[/]")
+ find_matching = re.search(rf"^\| {dependency} *\|", doc_ref_content, flags=re.MULTILINE)
+ if not find_matching:
+ errors.append(f"[red]ERROR: {dependency} is not listed in {EXTRA_PACKAGES_REF_FILE}[/]")
+ is_devel_dep = dependency.startswith("devel") or dependency in ["doc", "doc-gen"]
+ short_dep = dependency.replace("devel-", "")
+ if is_devel_dep:
+ suggestions_devel.append(
+ (
+ dependency,
+ f"pip install -e '.[{dependency}]'",
+ f"Adds all test libraries needed to test {short_dep}",
+ )
+ )
+ else:
+ suggestions.append(
+ (
+ dependency,
+ f"pip install apache-airflow[{dependency}]",
+ f"{dependency.capitalize()} hooks and operators",
+ )
+ )
+
+HEADERS = ["extra", "install command", "enables"]
+if errors:
+ console.print("\n".join(errors))
+ console.print()
+ console.print("[bright_blue]Suggested tables to add to references::[/]")
+ if suggestions:
+ console.print("[bright_blue]Regular dependencies[/]")
+ console.print(tabulate(suggestions, headers=HEADERS, tablefmt="grid"), markup=False)
+ if suggestions_devel:
+ console.print("[bright_blue]Devel dependencies[/]")
+ console.print(tabulate(suggestions_devel, headers=HEADERS, tablefmt="grid"), markup=False)
+ if suggestions_providers:
+ console.print("[bright_blue]Devel dependencies[/]")
+ console.print(tabulate(suggestions_providers, headers=HEADERS, tablefmt="grid"), markup=False)
+ sys.exit(1)
+else:
+ console.print(f"[green]Checked: {len(optional_dependencies)} dependencies are mentioned[/]")
diff --git a/scripts/ci/pre_commit/pre_commit_check_order_dockerfile_extras.py b/scripts/ci/pre_commit/pre_commit_check_order_dockerfile_extras.py
index 31c8c88eae..a57eee9a89 100755
--- a/scripts/ci/pre_commit/pre_commit_check_order_dockerfile_extras.py
+++ b/scripts/ci/pre_commit/pre_commit_check_order_dockerfile_extras.py
@@ -17,7 +17,7 @@
# specific language governing permissions and limitations
# under the License.
"""
-Test for an order of dependencies in setup.py
+Check if extras in Dockerfile are reflected in docker build-arg-ref.rst and global constants.
"""
from __future__ import annotations
diff --git a/scripts/ci/pre_commit/pre_commit_check_order_pyproject_toml.py b/scripts/ci/pre_commit/pre_commit_check_order_pyproject_toml.py
new file mode 100755
index 0000000000..0fbdf357f3
--- /dev/null
+++ b/scripts/ci/pre_commit/pre_commit_check_order_pyproject_toml.py
@@ -0,0 +1,103 @@
+#!/usr/bin/env python
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Test for an order of dependencies in setup.py
+"""
+from __future__ import annotations
+
+import re
+import sys
+from pathlib import Path
+
+from rich import print
+
+errors: list[str] = []
+
+AIRFLOW_ROOT_PATH = Path(__file__).parents[3].resolve()
+PYPROJECT_TOML_PATH = AIRFLOW_ROOT_PATH / "pyproject.toml"
+
+sys.path.insert(0, str(Path(__file__).parent.resolve())) # make sure common_precommit_utils is imported
+from common_precommit_utils import check_list_sorted
+
+
+def check_extras(type: str, extra: str, extras: list[str]) -> None:
+ r"""
+ Test for an order of dependencies in extra defined
+ `^dependent_group_name = [.*?]\n` in setup.py
+ """
+ print(f"[info]Checking {type}:{extra}[/]")
+ extras = [extra.replace("[", "\\[") for extra in extras]
+ check_list_sorted(extras, f"Order of extra: {type}:{extra}", errors)
+
+
+def extract_deps(content: str, extra: str) -> list[str]:
+ deps: list[str] = []
+ extracting = False
+ for line in content.splitlines():
+ line = line.strip()
+ if line.startswith("#"):
+ continue
+ if not extracting and line == f"{extra} = [":
+ extracting = True
+ elif extracting and line == "]":
+ break
+ elif extracting:
+ deps.append(line.strip().strip(",").strip('"'))
+ return deps
+
+
+def check_type(pyproject_toml_contents: str, type: str) -> None:
+ """
+ Test for an order of dependencies groups between mark
+ '# Start dependencies group' and '# End dependencies group' in setup.py
+ """
+ print(f"[info]Checking {type}[/]")
+ pattern_type = re.compile(f"# START OF {type}\n(.*)# END OF {type}", re.DOTALL)
+ parsed_type_content = pattern_type.findall(pyproject_toml_contents)[0]
+ # strip comments
+ parsed_type_content = (
+ "\n".join([line for line in parsed_type_content.splitlines() if not line.startswith("#")]) + "\n"
+ )
+ pattern_extra_name = re.compile(r" = \[.*?]\n", re.DOTALL)
+ type_content = pattern_extra_name.sub(",", parsed_type_content)
+
+ list_extra_names = type_content.strip(",").split(",")
+ check_list_sorted(list_extra_names, "Order of dependencies", errors)
+ for extra in list_extra_names:
+ deps_list = extract_deps(parsed_type_content, extra)
+ check_extras(type, extra, deps_list)
+
+
+if __name__ == "__main__":
+ file_contents = PYPROJECT_TOML_PATH.read_text()
+ check_type(file_contents, "core extras")
+ check_type(file_contents, "Apache no provider extras")
+ check_type(file_contents, "devel extras")
+ check_type(file_contents, "doc extras")
+ check_type(file_contents, "bundle extras")
+ check_type(file_contents, "deprecated extras")
+
+ print()
+ for error in errors:
+ print(error)
+
+ print()
+
+ if errors:
+ sys.exit(1)
diff --git a/scripts/ci/pre_commit/pre_commit_check_order_setup.py b/scripts/ci/pre_commit/pre_commit_check_order_setup.py
deleted file mode 100755
index 95bc8e59ee..0000000000
--- a/scripts/ci/pre_commit/pre_commit_check_order_setup.py
+++ /dev/null
@@ -1,135 +0,0 @@
-#!/usr/bin/env python
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-"""
-Test for an order of dependencies in setup.py
-"""
-from __future__ import annotations
-
-import os
-import re
-import sys
-from pathlib import Path
-
-from rich import print
-
-errors: list[str] = []
-
-SOURCE_DIR_PATH = Path(__file__).parents[3].resolve()
-sys.path.insert(0, os.fspath(SOURCE_DIR_PATH))
-sys.path.insert(0, str(Path(__file__).parent.resolve())) # make sure common_precommit_utils is imported
-from common_precommit_utils import check_list_sorted
-
-
-def check_main_dependent_group(setup_contents: str) -> None:
- """
- Test for an order of dependencies groups between mark
- '# Start dependencies group' and '# End dependencies group' in setup.py
- """
- print("[info]Checking main dependency group[/]")
- pattern_main_dependent_group = re.compile(
- "# Start dependencies group\n(.*)# End dependencies group", re.DOTALL
- )
- main_dependent_group = pattern_main_dependent_group.findall(setup_contents)[0]
-
- pattern_sub_dependent = re.compile(r" = \[.*?]\n", re.DOTALL)
- main_dependent = pattern_sub_dependent.sub(",", main_dependent_group)
-
- src = main_dependent.strip(",").split(",")
- check_list_sorted(src, "Order of dependencies", errors)
-
- for group in src:
- check_sub_dependent_group(group)
-
-
-def check_sub_dependent_group(group_name: str) -> None:
- r"""
- Test for an order of each dependencies groups declare like
- `^dependent_group_name = [.*?]\n` in setup.py
- """
- print(f"[info]Checking dependency group {group_name}[/]")
- check_list_sorted(getattr(setup, group_name), f"Order of dependency group: {group_name}", errors)
-
-
-def check_alias_dependent_group(setup_context: str) -> None:
- """
- Test for an order of each dependencies groups declare like
- `alias_dependent_group = dependent_group_1 + ... + dependent_group_n` in setup.py
- """
- pattern = re.compile("^\\w+ = (\\w+ \\+.*)", re.MULTILINE)
- dependents = pattern.findall(setup_context)
-
- for dependent in dependents:
- print(f"[info]Checking alias-dependent group {dependent}[/]")
- src = dependent.split(" + ")
- check_list_sorted(src, f"Order of alias dependencies group: {dependent}", errors)
-
-
-def check_variable_order(var_name: str) -> None:
- print(f"[info]Checking {var_name}[/]")
-
- var = getattr(setup, var_name)
-
- if isinstance(var, dict):
- check_list_sorted(list(var.keys()), f"Order of dependencies in: {var_name}", errors)
- else:
- check_list_sorted(var, f"Order of dependencies in: {var_name}", errors)
-
-
-def check_install_and_setup_requires() -> None:
- """
- Test for an order of dependencies in function do_setup section
- install_requires and setup_requires in setup.cfg
- """
-
- from setuptools.config import read_configuration
-
- path = os.fspath(SOURCE_DIR_PATH / "setup.cfg")
- config = read_configuration(path)
-
- pattern_dependent_version = re.compile("[~|><=;].*")
-
- for key in ("install_requires", "setup_requires"):
- print(f"[info]Checking setup.cfg group {key}[/]")
- deps = config["options"][key]
- dists = [pattern_dependent_version.sub("", p) for p in deps]
- check_list_sorted(dists, f"Order of dependencies in do_setup section: {key}", errors)
-
-
-if __name__ == "__main__":
- import setup
-
- with open(setup.__file__) as setup_file:
- file_contents = setup_file.read()
- check_main_dependent_group(file_contents)
- check_alias_dependent_group(file_contents)
- check_variable_order("CORE_EXTRAS_DEPENDENCIES")
- check_variable_order("ADDITIONAL_EXTRAS_DEPENDENCIES")
- check_variable_order("EXTRAS_DEPRECATED_ALIASES")
- check_variable_order("PREINSTALLED_PROVIDERS")
- check_install_and_setup_requires()
-
- print()
- print()
- for error in errors:
- print(error)
-
- print()
-
- if errors:
- sys.exit(1)
diff --git a/scripts/ci/pre_commit/pre_commit_check_setup_extra_packages_ref.py b/scripts/ci/pre_commit/pre_commit_check_setup_extra_packages_ref.py
deleted file mode 100755
index 0997fc008d..0000000000
--- a/scripts/ci/pre_commit/pre_commit_check_setup_extra_packages_ref.py
+++ /dev/null
@@ -1,260 +0,0 @@
-#!/usr/bin/env python
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-"""
-Checks if all the libraries in setup.py are listed in installation.rst file
-"""
-from __future__ import annotations
-
-import os
-import re
-import sys
-from pathlib import Path
-
-from rich import print
-from rich.console import Console
-from rich.table import Table
-
-AIRFLOW_SOURCES_DIR = Path(__file__).parents[3].resolve()
-SETUP_PY_FILE = "setup.py"
-DOCS_FILE = os.path.join("docs", "apache-airflow", "extra-packages-ref.rst")
-PY_IDENTIFIER = r"[a-zA-Z_][a-zA-Z0-9_\.]*"
-
-sys.path.insert(0, os.fspath(AIRFLOW_SOURCES_DIR))
-
-os.environ["_SKIP_PYTHON_VERSION_CHECK"] = "true"
-
-from setup import (
- EXTRAS_DEPENDENCIES,
- EXTRAS_DEPRECATED_ALIASES,
- EXTRAS_DEPRECATED_ALIASES_IGNORED_FROM_REF_DOCS,
- PREINSTALLED_PROVIDERS,
- add_all_provider_packages,
-)
-
-
-def get_file_content(*path_elements: str) -> str:
- file_path = AIRFLOW_SOURCES_DIR.joinpath(*path_elements)
- return file_path.read_text()
-
-
-def get_extras_from_setup() -> set[str]:
- """Returns a set of regular (non-deprecated) extras from setup."""
- return (
- set(EXTRAS_DEPENDENCIES.keys())
- - set(EXTRAS_DEPRECATED_ALIASES.keys())
- - set(EXTRAS_DEPRECATED_ALIASES_IGNORED_FROM_REF_DOCS)
- )
-
-
-def get_extras_from_docs() -> set[str]:
- """
- Returns a list of extras from airflow.docs.
- """
- docs_content = get_file_content(DOCS_FILE)
- extras_section_regex = re.compile(
- rf"\|[^|]+\|.*pip install .apache-airflow\[({PY_IDENTIFIER})][^|]+\|[^|]+\|",
- re.MULTILINE,
- )
- doc_extra_set: set[str] = set()
- for doc_extra in extras_section_regex.findall(docs_content):
- doc_extra_set.add(doc_extra)
- return doc_extra_set
-
-
-def get_preinstalled_providers_from_docs() -> list[str]:
- """
- Returns list of pre-installed providers from the doc.
- """
- docs_content = get_file_content(DOCS_FILE)
- preinstalled_section_regex = re.compile(
- rf"\|\s*({PY_IDENTIFIER})\s*\|[^|]+pip install[^|]+\|[^|]+\|\s+\*\s+\|$",
- re.MULTILINE,
- )
- return preinstalled_section_regex.findall(docs_content)
-
-
-def get_deprecated_extras_from_docs() -> dict[str, str]:
- """
- Returns dict of deprecated extras from airflow.docs (alias -> target extra)
- """
- deprecated_extras = {}
- docs_content = get_file_content(DOCS_FILE)
-
- deprecated_extras_section_regex = re.compile(
- r"\| Deprecated extra \| Extra to be used instead \|\n(.*)\n", re.DOTALL
- )
- deprecated_extras_content = deprecated_extras_section_regex.findall(docs_content)[0]
-
- deprecated_extras_regexp = re.compile(r"\|\s(\S+)\s+\|\s(\S*)\s+\|$", re.MULTILINE)
- for extras in deprecated_extras_regexp.findall(deprecated_extras_content):
- deprecated_extras[extras[0]] = extras[1]
- return deprecated_extras
-
-
-def check_extras(console: Console) -> bool:
- """
- Checks if non-deprecated extras match setup vs. doc.
- :param console: print table there in case of errors
- :return: True if all ok, False otherwise
- """
- extras_table = Table()
- extras_table.add_column("NAME", justify="right", style="cyan")
- extras_table.add_column("SETUP", justify="center", style="magenta")
- extras_table.add_column("DOCS", justify="center", style="yellow")
- non_deprecated_setup_extras = get_extras_from_setup()
- non_deprecated_docs_extras = get_extras_from_docs()
- for extra in non_deprecated_setup_extras:
- if extra not in non_deprecated_docs_extras:
- extras_table.add_row(extra, "V", "")
- for extra in non_deprecated_docs_extras:
- if extra not in non_deprecated_setup_extras:
- extras_table.add_row(extra, "", "V")
- if extras_table.row_count != 0:
- print(
- f"""\
-[red bold]ERROR!![/red bold]
-
-The "[bold]CORE_EXTRAS_DEPENDENCIES[/bold]"
-sections in the setup file: [bold yellow]{SETUP_PY_FILE}[/bold yellow]
-should be synchronized with the "Extra Packages Reference"
-in the documentation file: [bold yellow]{DOCS_FILE}[/bold yellow].
-
-Below is the list of extras that:
-
- * are used but are not documented,
- * are documented but not used,
-
-[bold]Please synchronize setup/documentation files![/bold]
-
-"""
- )
- console.print(extras_table)
- return False
- return True
-
-
-def check_deprecated_extras(console: Console) -> bool:
- """
- Checks if deprecated extras match setup vs. doc.
- :param console: print table there in case of errors
- :return: True if all ok, False otherwise
- """
- deprecated_setup_extras = EXTRAS_DEPRECATED_ALIASES
- deprecated_docs_extras = get_deprecated_extras_from_docs()
-
- deprecated_extras_table = Table()
- deprecated_extras_table.add_column("DEPRECATED_IN_SETUP", justify="right", style="cyan")
- deprecated_extras_table.add_column("TARGET_IN_SETUP", justify="center", style="magenta")
- deprecated_extras_table.add_column("DEPRECATED_IN_DOCS", justify="right", style="cyan")
- deprecated_extras_table.add_column("TARGET_IN_DOCS", justify="center", style="magenta")
-
- for extra in deprecated_setup_extras.keys():
- if extra not in deprecated_docs_extras:
- deprecated_extras_table.add_row(extra, deprecated_setup_extras[extra], "", "")
- elif deprecated_docs_extras[extra] != deprecated_setup_extras[extra]:
- deprecated_extras_table.add_row(
- extra, deprecated_setup_extras[extra], extra, deprecated_docs_extras[extra]
- )
-
- for extra in deprecated_docs_extras.keys():
- if extra not in deprecated_setup_extras:
- deprecated_extras_table.add_row("", "", extra, deprecated_docs_extras[extra])
-
- if deprecated_extras_table.row_count != 0:
- print(
- f"""\
-[red bold]ERROR!![/red bold]
-
-The "[bold]EXTRAS_DEPRECATED_ALIASES[/bold]" section in the setup file:\
-[bold yellow]{SETUP_PY_FILE}[/bold yellow]
-should be synchronized with the "Extra Packages Reference"
-in the documentation file: [bold yellow]{DOCS_FILE}[/bold yellow].
-
-Below is the list of deprecated extras that:
-
- * are used but are not documented,
- * are documented but not used,
- * or have different target extra specified in the documentation or setup.
-
-[bold]Please synchronize setup/documentation files![/bold]
-
-"""
- )
- console.print(deprecated_extras_table)
- return False
- return True
-
-
-def check_preinstalled_extras(console: Console) -> bool:
- """
- Checks if preinstalled extras match setup vs. doc.
- :param console: print table there in case of errors
- :return: True if all ok, False otherwise
- """
- preinstalled_providers_from_docs = get_preinstalled_providers_from_docs()
- preinstalled_providers_from_setup = [provider.split(">=")[0] for provider in PREINSTALLED_PROVIDERS]
-
- preinstalled_providers_table = Table()
- preinstalled_providers_table.add_column("PREINSTALLED_IN_SETUP", justify="right", style="cyan")
- preinstalled_providers_table.add_column("PREINSTALLED_IN_DOCS", justify="center", style="magenta")
-
- for provider in preinstalled_providers_from_setup:
- if provider not in preinstalled_providers_from_docs:
- preinstalled_providers_table.add_row(provider, "")
-
- for provider in preinstalled_providers_from_docs:
- if provider not in preinstalled_providers_from_setup:
- preinstalled_providers_table.add_row("", provider)
-
- if preinstalled_providers_table.row_count != 0:
- print(
- f"""\
-[red bold]ERROR!![/red bold]
-
-The "[bold]PREINSTALLED_PROVIDERS[/bold]" section in the setup file:\
-[bold yellow]{SETUP_PY_FILE}[/bold yellow]
-should be synchronized with the "Extra Packages Reference"
-in the documentation file: [bold yellow]{DOCS_FILE}[/bold yellow].
-
-Below is the list of preinstalled providers that:
- * are used but are not documented,
- * or are documented but not used.
-
-[bold]Please synchronize setup/documentation files![/bold]
-
-"""
- )
- console.print(preinstalled_providers_table)
- return False
- return True
-
-
-if __name__ == "__main__":
- status: list[bool] = []
- # force adding all provider package dependencies, to check providers status
- add_all_provider_packages()
- main_console = Console()
- status.append(check_extras(main_console))
- status.append(check_deprecated_extras(main_console))
- status.append(check_preinstalled_extras(main_console))
-
- if all(status):
- print("All extras are synchronized: [green]OK[/]")
- sys.exit(0)
- sys.exit(1)
diff --git a/scripts/ci/pre_commit/pre_commit_compile_www_assets.py b/scripts/ci/pre_commit/pre_commit_compile_www_assets.py
index 3365d1fb87..d3529f1b13 100755
--- a/scripts/ci/pre_commit/pre_commit_compile_www_assets.py
+++ b/scripts/ci/pre_commit/pre_commit_compile_www_assets.py
@@ -20,6 +20,7 @@ from __future__ import annotations
import hashlib
import os
import re
+import shutil
import subprocess
import sys
from pathlib import Path
@@ -53,14 +54,21 @@ if __name__ not in ("__main__", "__mp_main__"):
if __name__ == "__main__":
www_directory = AIRFLOW_SOURCES_PATH / "airflow" / "www"
+ node_modules_directory = www_directory / "node_modules"
+ dist_directory = www_directory / "static" / "dist"
WWW_HASH_FILE.parent.mkdir(exist_ok=True)
- old_hash = WWW_HASH_FILE.read_text() if WWW_HASH_FILE.exists() else ""
- new_hash = get_directory_hash(www_directory, skip_path_regexp=r".*node_modules.*")
- if new_hash == old_hash:
- print("The WWW directory has not changed! Skip regeneration.")
- sys.exit(0)
+ if node_modules_directory.exists() and dist_directory.exists():
+ old_hash = WWW_HASH_FILE.read_text() if WWW_HASH_FILE.exists() else ""
+ new_hash = get_directory_hash(www_directory, skip_path_regexp=r".*node_modules.*")
+ if new_hash == old_hash:
+ print("The WWW directory has not changed! Skip regeneration.")
+ sys.exit(0)
+ else:
+ shutil.rmtree(node_modules_directory, ignore_errors=True)
+ shutil.rmtree(dist_directory, ignore_errors=True)
env = os.environ.copy()
env["FORCE_COLOR"] = "true"
subprocess.check_call(["yarn", "install", "--frozen-lockfile"], cwd=os.fspath(www_directory))
subprocess.check_call(["yarn", "run", "build"], cwd=os.fspath(www_directory), env=env)
+ new_hash = get_directory_hash(www_directory, skip_path_regexp=r".*node_modules.*")
WWW_HASH_FILE.write_text(new_hash)
diff --git a/scripts/ci/pre_commit/pre_commit_insert_extras.py b/scripts/ci/pre_commit/pre_commit_insert_extras.py
index 103cec230b..4933032e30 100755
--- a/scripts/ci/pre_commit/pre_commit_insert_extras.py
+++ b/scripts/ci/pre_commit/pre_commit_insert_extras.py
@@ -17,33 +17,85 @@
# under the License.
from __future__ import annotations
-import os
import sys
import textwrap
+from enum import Enum
from pathlib import Path
-AIRFLOW_SOURCES_DIR = Path(__file__).parents[3].resolve()
+# tomllib is available in Python 3.11+ and before that tomli offers same interface for parsing TOML files
+try:
+ import tomllib
+except ImportError:
+ import tomli as tomllib
+
+AIRFLOW_ROOT_PATH = Path(__file__).parents[3].resolve()
+PYPROJECT_TOML_FILE_PATH = AIRFLOW_ROOT_PATH / "pyproject.toml"
sys.path.insert(0, str(Path(__file__).parent.resolve())) # make sure common_precommit_utils is imported
-sys.path.insert(0, str(AIRFLOW_SOURCES_DIR)) # make sure setup is imported from Airflow
-# flake8: noqa: F401
+from common_precommit_utils import insert_documentation
-os.environ["_SKIP_PYTHON_VERSION_CHECK"] = "true"
-from common_precommit_utils import insert_documentation
+class ExtraType(Enum):
+ DEVEL = "DEVEL"
+ DOC = "DOC"
+ REGULAR = "REGULAR"
+
+
+def get_header_and_footer(extra_type: ExtraType, file_format: str) -> tuple[str, str]:
+ if file_format == "rst":
+ return f" .. START {extra_type.value} EXTRAS HERE", f" .. END {extra_type.value} EXTRAS HERE"
+ elif file_format == "txt":
+ return f"# START {extra_type.value} EXTRAS HERE", f"# END {extra_type.value} EXTRAS HERE"
+ else:
+ raise Exception(f"Bad format {format} passed. Only rst and txt are supported")
+
+
+def get_wrapped_list(extras_set: set[str]) -> list[str]:
+ return [line + "\n" for line in textwrap.wrap(", ".join(sorted(extras_set)), 100)]
+
+
+def get_extra_types_dict(extras: dict[str, list[str]]) -> dict[ExtraType, tuple[set[str], list[str]]]:
+ """
+ Split extras into four types.
+
+ :return: dictionary of extra types with tuple of two set,list - set of extras and text-wrapped list
+ """
+ extra_type_dict: dict[ExtraType, tuple[set[str], list[str]]] = {}
+
+ for extra_type in ExtraType:
+ extra_type_dict[extra_type] = (set(), [])
+
+ for key, value in extras.items():
+ if key.startswith("devel"):
+ extra_type_dict[ExtraType.DEVEL][0].add(key)
+ elif key in ["doc", "doc-gen"]:
+ extra_type_dict[ExtraType.DOC][0].add(key)
+ else:
+ extra_type_dict[ExtraType.REGULAR][0].add(key)
+
+ for extra_type in ExtraType:
+ extra_type_dict[extra_type][1].extend(get_wrapped_list(extra_type_dict[extra_type][0]))
+
+ return extra_type_dict
+
+
+def get_extras_from_pyproject_toml() -> dict[str, list[str]]:
+ pyproject_toml_content = tomllib.loads(PYPROJECT_TOML_FILE_PATH.read_text())
+ return pyproject_toml_content["project"]["optional-dependencies"]
-from setup import EXTRAS_DEPENDENCIES
-sys.path.append(str(AIRFLOW_SOURCES_DIR))
+FILES_TO_UPDATE = [(AIRFLOW_ROOT_PATH / "INSTALL", "txt"), (AIRFLOW_ROOT_PATH / "CONTRIBUTING.rst", "rst")]
-RST_HEADER = " .. START EXTRAS HERE"
-RST_FOOTER = " .. END EXTRAS HERE"
-INSTALL_HEADER = "# START EXTRAS HERE"
-INSTALL_FOOTER = "# END EXTRAS HERE"
+def process_documentation_files():
+ extra_type_dict = get_extra_types_dict(get_extras_from_pyproject_toml())
+ for file, file_format in FILES_TO_UPDATE:
+ if not file.exists():
+ raise Exception(f"File {file} does not exist")
+ for extra_type in ExtraType:
+ header, footer = get_header_and_footer(extra_type, file_format)
+ insert_documentation(file, extra_type_dict[extra_type][1], header, footer)
-CONSTANTS_HEADER = "# START EXTRAS HERE"
-CONSTANTS_FOOTER = "# END EXTRAS HERE"
DEFAULT_EXTRAS = (
"amazon,async,celery,cncf.kubernetes,daskexecutor,docker,elasticsearch,ftp,google,"
@@ -53,13 +105,4 @@ DEFAULT_EXTRAS = (
if __name__ == "__main__":
- install_file_path = AIRFLOW_SOURCES_DIR / "INSTALL"
- contributing_file_path = AIRFLOW_SOURCES_DIR / "CONTRIBUTING.rst"
- global_constants_file_path = (
- AIRFLOW_SOURCES_DIR / "dev" / "breeze" / "src" / "airflow_breeze" / "global_constants.py"
- )
- extras_list = textwrap.wrap(", ".join(EXTRAS_DEPENDENCIES.keys()), 100)
- extras_list = [line + "\n" for line in extras_list]
- extras_code = [f" {extra}\n" for extra in EXTRAS_DEPENDENCIES.keys()]
- insert_documentation(install_file_path, extras_list, INSTALL_HEADER, INSTALL_FOOTER)
- insert_documentation(contributing_file_path, extras_list, RST_HEADER, RST_FOOTER)
+ process_documentation_files()
diff --git a/scripts/ci/pre_commit/pre_commit_sort_installed_providers.py b/scripts/ci/pre_commit/pre_commit_sort_installed_providers.py
index df53dea94d..fb735569a3 100755
--- a/scripts/ci/pre_commit/pre_commit_sort_installed_providers.py
+++ b/scripts/ci/pre_commit/pre_commit_sort_installed_providers.py
@@ -27,6 +27,7 @@ if __name__ not in ("__main__", "__mp_main__"):
AIRFLOW_SOURCES = Path(__file__).parents[3].resolve()
+DEV_DIR_PATH = AIRFLOW_SOURCES / "dev"
def stable_sort(x):
@@ -37,8 +38,14 @@ def sort_uniq(sequence):
return sorted(set(sequence), key=stable_sort)
-if __name__ == "__main__":
- installed_providers_path = Path(AIRFLOW_SOURCES) / "airflow" / "providers" / "installed_providers.txt"
- content = installed_providers_path.read_text().splitlines(keepends=True)
+def sort_file(path: Path):
+ content = path.read_text().splitlines(keepends=True)
sorted_content = sort_uniq(content)
- installed_providers_path.write_text("".join(sorted_content))
+ path.write_text("".join(sorted_content))
+
+
+if __name__ == "__main__":
+ prod_image_installed_providers_path = DEV_DIR_PATH / "prod_image_installed_providers.txt"
+ airflow_pre_installed_providers_path = DEV_DIR_PATH / "airflow_pre_installed_providers.txt"
+ sort_file(prod_image_installed_providers_path)
+ sort_file(airflow_pre_installed_providers_path)
diff --git a/scripts/ci/pre_commit/pre_commit_update_providers_dependencies.py b/scripts/ci/pre_commit/pre_commit_update_providers_dependencies.py
index 5fa638505f..50cebee3e5 100755
--- a/scripts/ci/pre_commit/pre_commit_update_providers_dependencies.py
+++ b/scripts/ci/pre_commit/pre_commit_update_providers_dependencies.py
@@ -17,11 +17,13 @@
# under the License.
from __future__ import annotations
+import hashlib
import json
import os
import sys
from ast import Import, ImportFrom, NodeVisitor, parse
from collections import defaultdict
+from enum import Enum
from pathlib import Path
from typing import Any, List
@@ -40,6 +42,12 @@ AIRFLOW_SYSTEM_TESTS_PROVIDERS_DIR = AIRFLOW_SOURCES_ROOT / "system" / "tests" /
DEPENDENCIES_JSON_FILE_PATH = AIRFLOW_SOURCES_ROOT / "generated" / "provider_dependencies.json"
+PYPROJECT_TOML_FILE_PATH = AIRFLOW_SOURCES_ROOT / "pyproject.toml"
+
+MY_FILE = Path(__file__).resolve()
+MY_MD5SUM_FILE = MY_FILE.parent / MY_FILE.name.replace(".py", ".py.md5sum")
+
+
sys.path.insert(0, str(AIRFLOW_SOURCES_ROOT)) # make sure setup is imported from Airflow
warnings: list[str] = []
@@ -109,8 +117,6 @@ def get_provider_id_from_relative_import_or_file(relative_path_or_file: str) ->
provider_candidate = relative_path_or_file.replace(os.sep, ".").split(".")
while provider_candidate:
candidate_provider_id = ".".join(provider_candidate)
- if "google_vendor" in candidate_provider_id:
- candidate_provider_id = candidate_provider_id.replace("google_vendor", "google")
if candidate_provider_id in ALL_PROVIDERS:
return candidate_provider_id
provider_candidate = provider_candidate[:-1]
@@ -175,6 +181,149 @@ def check_if_different_provider_used(file_path: Path) -> None:
STATES: dict[str, str] = {}
+FOUND_EXTRAS: dict[str, list[str]] = defaultdict(list)
+
+
+class ParsedDependencyTypes(Enum):
+ CORE_EXTRAS = "core extras"
+ APACHE_NO_PROVIDER_EXTRAS = "Apache no provider extras"
+ DEVEL_EXTRAS = "devel extras"
+ DOC_EXTRAS = "doc extras"
+ BUNDLE_EXTRAS = "bundle extras"
+ DEPRECATED_EXTRAS = "deprecated extras"
+ MANUAL_EXTRAS = "manual extras"
+
+
+GENERATED_DEPENDENCIES_START = "# START OF GENERATED DEPENDENCIES"
+GENERATED_DEPENDENCIES_END = "# END OF GENERATED DEPENDENCIES"
+
+
+def normalize_extra(dependency: str) -> str:
+ return dependency.replace(".", "-").replace("_", "-")
+
+
+def normalize_package_name(dependency: str) -> str:
+ return f"apache-airflow-providers-{dependency.replace('.', '-').replace('_', '-')}"
+
+
+def convert_to_extra_dependency(dependency: str) -> str:
+ # if there is version in dependency - remove it as we do not need it in extra specification
+ # for editable installation
+ if ">=" in dependency:
+ dependency = dependency.split(">=")[0]
+ extra = dependency.replace("apache-airflow-providers-", "").replace("-", "_").replace(".", "_")
+ return f"apache-airflow[{extra}]"
+
+
+def generate_dependencies(
+ result_content: list[str],
+ dependencies: dict[str, dict[str, list[str] | str]],
+):
+ def generate_parsed_extras(type: ParsedDependencyTypes):
+ result_content.append(f" # {type.value}")
+ for extra in FOUND_EXTRAS[type.value]:
+ result_content.append(f' "apache-airflow[{extra}]",')
+
+ def get_python_exclusion(dependency_info: dict[str, list[str] | str]):
+ excluded_python_versions = dependency_info.get("excluded-python-versions")
+ exclusion = ""
+ if excluded_python_versions:
+ separator = ";"
+ for version in excluded_python_versions:
+ exclusion += f'{separator}python_version != \\"{version}\\"'
+ separator = " and "
+ return exclusion
+
+ for dependency, dependency_info in dependencies.items():
+ if dependency_info["state"] in ["suspended", "removed"]:
+ continue
+ result_content.append(f"{normalize_extra(dependency)} = [")
+ deps = dependency_info["deps"]
+ if not isinstance(deps, list):
+ raise TypeError(f"Wrong type of 'deps' {deps} for {dependency} in {DEPENDENCIES_JSON_FILE_PATH}")
+ for dep in deps:
+ if dep.startswith("apache-airflow-providers-"):
+ dep = convert_to_extra_dependency(dep)
+ elif dep.startswith("apache-airflow>="):
+ continue
+ result_content.append(f' "{dep}{get_python_exclusion(dependency_info)}",')
+ devel_deps = dependency_info.get("devel-deps")
+ if devel_deps:
+ result_content.append(f" # Devel dependencies for the {dependency} provider")
+ for dep in devel_deps:
+ result_content.append(f' "{dep}{get_python_exclusion(dependency_info)}",')
+ result_content.append("]")
+ result_content.append("all = [")
+ generate_parsed_extras(ParsedDependencyTypes.CORE_EXTRAS)
+ generate_parsed_extras(ParsedDependencyTypes.APACHE_NO_PROVIDER_EXTRAS)
+ result_content.append(" # Provider extras")
+ for dependency, dependency_info in dependencies.items():
+ result_content.append(f' "apache-airflow[{normalize_extra(dependency)}]",')
+ result_content.append("]")
+ result_content.append("devel-all = [")
+ result_content.append(' "apache-airflow[all]",')
+ result_content.append(' "apache-airflow[devel]",')
+ result_content.append(' "apache-airflow[doc]",')
+ result_content.append(' "apache-airflow[doc-gen]",')
+ result_content.append(' "apache-airflow[saml]",')
+ generate_parsed_extras(ParsedDependencyTypes.APACHE_NO_PROVIDER_EXTRAS)
+ result_content.append(" # Include all provider deps")
+ for dependency, dependency_info in dependencies.items():
+ result_content.append(f' "apache-airflow[{normalize_extra(dependency)}]",')
+ result_content.append("]")
+
+
+def get_dependency_type(dependency_type: str) -> ParsedDependencyTypes | None:
+ for dep_type in ParsedDependencyTypes:
+ if dep_type.value == dependency_type:
+ return dep_type
+ return None
+
+
+def update_pyproject_toml(dependencies: dict[str, dict[str, list[str] | str]]):
+ file_content = PYPROJECT_TOML_FILE_PATH.read_text()
+ result_content: list[str] = []
+ copying = True
+ current_type: str | None = None
+ line_count: int = 0
+ for line in file_content.splitlines():
+ if copying:
+ result_content.append(line)
+ if line.strip().startswith(GENERATED_DEPENDENCIES_START):
+ copying = False
+ generate_dependencies(result_content, dependencies)
+ elif line.strip().startswith(GENERATED_DEPENDENCIES_END):
+ copying = True
+ result_content.append(line)
+ elif line.strip().startswith("# START OF "):
+ current_type = line.strip().replace("# START OF ", "")
+ type_enum = get_dependency_type(current_type)
+ if type_enum is None:
+ console.print(
+ f"[red]Wrong start of section '{current_type}' in {PYPROJECT_TOML_FILE_PATH} "
+ f"at line {line_count}: Unknown section type"
+ )
+ sys.exit(1)
+ elif line.strip().startswith("# END OF "):
+ end_type = line.strip().replace("# END OF ", "")
+ if end_type != current_type:
+ console.print(
+ f"[red]Wrong end of section {end_type} in {PYPROJECT_TOML_FILE_PATH} at line {line_count}"
+ )
+ sys.exit(1)
+ if current_type:
+ if line.strip().endswith(" = ["):
+ FOUND_EXTRAS[current_type].append(line.split(" = [")[0].strip())
+ line_count += 1
+ PYPROJECT_TOML_FILE_PATH.write_text("\n".join(result_content) + "\n")
+
+
+def calculate_my_hash():
+ my_file = MY_FILE.resolve()
+ hash_md5 = hashlib.md5()
+ hash_md5.update(my_file.read_bytes())
+ return hash_md5.hexdigest()
+
if __name__ == "__main__":
find_all_providers_and_provider_files()
@@ -185,6 +334,7 @@ if __name__ == "__main__":
check_if_different_provider_used(file)
for provider, provider_yaml_content in ALL_PROVIDERS.items():
ALL_DEPENDENCIES[provider]["deps"].extend(provider_yaml_content["dependencies"])
+ ALL_DEPENDENCIES[provider]["devel-deps"].extend(provider_yaml_content.get("devel-dependencies") or [])
STATES[provider] = provider_yaml_content["state"]
if warnings:
console.print("[yellow]Warnings!\n")
@@ -199,6 +349,7 @@ if __name__ == "__main__":
unique_sorted_dependencies: dict[str, dict[str, list[str] | str]] = defaultdict(dict)
for key in sorted(ALL_DEPENDENCIES.keys()):
unique_sorted_dependencies[key]["deps"] = sorted(ALL_DEPENDENCIES[key]["deps"])
+ unique_sorted_dependencies[key]["devel-deps"] = ALL_DEPENDENCIES[key].get("devel-deps") or []
unique_sorted_dependencies[key]["cross-providers-deps"] = sorted(
set(ALL_DEPENDENCIES[key]["cross-providers-deps"])
)
@@ -210,13 +361,17 @@ if __name__ == "__main__":
console.print("[red]Errors found during verification. Exiting!")
console.print()
sys.exit(1)
- old_dependencies = DEPENDENCIES_JSON_FILE_PATH.read_text()
+ old_dependencies = (
+ DEPENDENCIES_JSON_FILE_PATH.read_text() if DEPENDENCIES_JSON_FILE_PATH.exists() else "{}"
+ )
new_dependencies = json.dumps(unique_sorted_dependencies, indent=2) + "\n"
- if new_dependencies != old_dependencies:
+ old_md5sum = MY_MD5SUM_FILE.read_text().strip() if MY_MD5SUM_FILE.exists() else ""
+ new_md5sum = calculate_my_hash()
+ if new_dependencies != old_dependencies or new_md5sum != old_md5sum:
DEPENDENCIES_JSON_FILE_PATH.write_text(json.dumps(unique_sorted_dependencies, indent=2) + "\n")
if os.environ.get("CI"):
console.print()
- console.print(f"[info]There is a need to regenerate {DEPENDENCIES_JSON_FILE_PATH}")
+ console.print(f"There is a need to regenerate {DEPENDENCIES_JSON_FILE_PATH}")
console.print(
f"[red]You need to run the following command locally and commit generated "
f"{DEPENDENCIES_JSON_FILE_PATH.relative_to(AIRFLOW_SOURCES_ROOT)} file:\n"
@@ -229,8 +384,12 @@ if __name__ == "__main__":
f"[yellow]Regenerated new dependencies. Please commit "
f"{DEPENDENCIES_JSON_FILE_PATH.relative_to(AIRFLOW_SOURCES_ROOT)}!\n"
)
- console.print(f"[info]Written {DEPENDENCIES_JSON_FILE_PATH}")
+ console.print(f"Written {DEPENDENCIES_JSON_FILE_PATH}")
+ console.print()
+ update_pyproject_toml(unique_sorted_dependencies)
+ console.print(f"Written {PYPROJECT_TOML_FILE_PATH}")
console.print()
+ MY_MD5SUM_FILE.write_text(new_md5sum + "\n")
sys.exit(1)
else:
console.print(
diff --git a/scripts/ci/pre_commit/pre_commit_update_providers_dependencies.py.md5sum b/scripts/ci/pre_commit/pre_commit_update_providers_dependencies.py.md5sum
new file mode 100644
index 0000000000..610f5562c8
--- /dev/null
+++ b/scripts/ci/pre_commit/pre_commit_update_providers_dependencies.py.md5sum
@@ -0,0 +1 @@
+ed25c4f6b220c14b40bbf370fee9388e
diff --git a/scripts/docker/entrypoint_ci.sh b/scripts/docker/entrypoint_ci.sh
index cf0ee85bab..52d79f4154 100755
--- a/scripts/docker/entrypoint_ci.sh
+++ b/scripts/docker/entrypoint_ci.sh
@@ -231,7 +231,7 @@ function check_download_sqlalchemy() {
if [[ ${DOWNGRADE_SQLALCHEMY=} != "true" ]]; then
return
fi
- min_sqlalchemy_version=$(grep "sqlalchemy>=" setup.cfg | sed "s/.*>=\([0-9\.]*\).*/\1/")
+ min_sqlalchemy_version=$(grep "\"sqlalchemy>=" pyproject.toml | sed "s/.*>=\([0-9\.]*\).*/\1/" | xargs)
echo
echo "${COLOR_BLUE}Downgrading sqlalchemy to minimum supported version: ${min_sqlalchemy_version}${COLOR_RESET}"
echo
diff --git a/scripts/docker/install_airflow.sh b/scripts/docker/install_airflow.sh
index 56fec40407..959b4befda 100644
--- a/scripts/docker/install_airflow.sh
+++ b/scripts/docker/install_airflow.sh
@@ -60,10 +60,11 @@ function install_airflow() {
"${AIRFLOW_INSTALLATION_METHOD}[${AIRFLOW_EXTRAS}]${AIRFLOW_VERSION_SPECIFICATION}" \
${EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS=}
if [[ -n "${AIRFLOW_INSTALL_EDITABLE_FLAG}" ]]; then
- # Remove airflow and reinstall it using editable flag
+ # Remove airflow and all providers and reinstall it using editable flag
# We can only do it when we install airflow from sources
set -x
- pip uninstall apache-airflow --yes
+ pip freeze | grep apache-airflow-providers | xargs pip uninstall --yes 2>/dev/null || true
+ pip uninstall apache-airflow --yes 2>/dev/null || true
pip install --root-user-action ignore ${AIRFLOW_INSTALL_EDITABLE_FLAG} \
${ADDITIONAL_PIP_INSTALL_FLAGS} \
"${AIRFLOW_INSTALLATION_METHOD}[${AIRFLOW_EXTRAS}]${AIRFLOW_VERSION_SPECIFICATION}"
@@ -85,7 +86,7 @@ function install_airflow() {
"${AIRFLOW_INSTALLATION_METHOD}[${AIRFLOW_EXTRAS}]${AIRFLOW_VERSION_SPECIFICATION}" \
--constraint "${AIRFLOW_CONSTRAINTS_LOCATION}"
common::install_pip_version
- # then upgrade if needed without using constraints to account for new limits in setup.py
+ # then upgrade if needed without using constraints to account for new limits in pyproject.toml
pip install --root-user-action ignore --upgrade --upgrade-strategy only-if-needed \
${ADDITIONAL_PIP_INSTALL_FLAGS} \
${AIRFLOW_INSTALL_EDITABLE_FLAG} \
diff --git a/scripts/docker/install_airflow_dependencies_from_branch_tip.sh b/scripts/docker/install_airflow_dependencies_from_branch_tip.sh
index 26279c4bc8..9c809039c7 100644
--- a/scripts/docker/install_airflow_dependencies_from_branch_tip.sh
+++ b/scripts/docker/install_airflow_dependencies_from_branch_tip.sh
@@ -18,11 +18,11 @@
# shellcheck shell=bash disable=SC2086
# Installs Airflow from $AIRFLOW_BRANCH tip. This is pure optimisation. It is done because we do not want
-# to reinstall all dependencies from scratch when setup.py changes. Problem with Docker caching is that
+# to reinstall all dependencies from scratch when pyproject.toml changes. Problem with Docker caching is that
# when a file is changed, when added to docker context, it invalidates the cache and it causes Docker
# build to reinstall all dependencies from scratch. This can take a loooooot of time. Therefore we install
# the dependencies first from main (and uninstall airflow right after) so that we can start installing
-# deps from those pre-installed dependencies. It saves few minutes of build time when setup.py changes.
+# deps from those pre-installed dependencies. It saves few minutes of build time when pyproject.toml changes.
#
# If INSTALL_MYSQL_CLIENT is set to false, mysql extra is removed
# If INSTALL_POSTGRES_CLIENT is set to false, postgres extra is removed
diff --git a/scripts/in_container/run_generate_constraints.py b/scripts/in_container/run_generate_constraints.py
index e03eff5126..99080fa3eb 100755
--- a/scripts/in_container/run_generate_constraints.py
+++ b/scripts/in_container/run_generate_constraints.py
@@ -17,6 +17,7 @@
# under the License.
from __future__ import annotations
+import json
import os
import sys
from dataclasses import dataclass
@@ -32,6 +33,9 @@ from in_container_utils import click, console, run_command
AIRFLOW_SOURCES = Path(__file__).resolve().parents[2]
DEFAULT_BRANCH = os.environ.get("DEFAULT_BRANCH", "main")
PYTHON_VERSION = os.environ.get("PYTHON_MAJOR_MINOR_VERSION", "3.8")
+GENERATED_PROVIDER_DEPENDENCIES_FILE = AIRFLOW_SOURCES / "generated" / "provider_dependencies.json"
+
+ALL_PROVIDER_DEPENDENCIES = json.loads(GENERATED_PROVIDER_DEPENDENCIES_FILE.read_text())
now = datetime.now().isoformat()
@@ -125,7 +129,7 @@ class ConfigParams:
def install_local_airflow_with_eager_upgrade(
- config_params: ConfigParams, eager_upgrade_additional_requirements: str, extras: list[str]
+ config_params: ConfigParams, eager_upgrade_additional_requirements: str
) -> None:
run_command(
[
@@ -133,7 +137,8 @@ def install_local_airflow_with_eager_upgrade(
"install",
"--root-user-action",
"ignore",
- f".[{','.join(extras)}]",
+ "-e",
+ ".[all-core]",
*eager_upgrade_additional_requirements.split(" "),
"--upgrade",
"--upgrade-strategy",
@@ -254,16 +259,12 @@ def uninstall_all_packages(config_params: ConfigParams):
)
-def get_core_airflow_dependencies() -> list[str]:
- import setup
-
- return list(setup.CORE_EXTRAS_DEPENDENCIES.keys())
-
-
-def get_all_provider_packages() -> list[str]:
- import setup
-
- return setup.get_all_provider_packages().split(" ")
+def get_all_active_provider_packages() -> list[str]:
+ return [
+ f"apache-airflow-providers-{provider.replace('.','-')}"
+ for provider in ALL_PROVIDER_DEPENDENCIES.keys()
+ if ALL_PROVIDER_DEPENDENCIES[provider]["state"] == "ready"
+ ]
def generate_constraints_source_providers(config_params: ConfigParams) -> None:
@@ -288,8 +289,7 @@ def generate_constraints_pypi_providers(config_params: ConfigParams) -> None:
:return:
"""
dist_dir = Path("/dist")
- core_dependencies = get_core_airflow_dependencies()
- all_provider_packages = get_all_provider_packages()
+ all_provider_packages = get_all_active_provider_packages()
chicken_egg_prefixes = []
packages_to_install = []
console.print("[bright_blue]Installing Airflow with PyPI providers with eager upgrade")
@@ -332,7 +332,7 @@ def generate_constraints_pypi_providers(config_params: ConfigParams) -> None:
"install",
"--root-user-action",
"ignore",
- f".[{','.join(core_dependencies)}]",
+ ".[all-core]",
*packages_to_install,
*config_params.eager_upgrade_additional_requirements.split(" "),
"--upgrade",
@@ -355,15 +355,15 @@ def generate_constraints_no_providers(config_params: ConfigParams) -> None:
Generates constraints without any provider dependencies. This is used mostly to generate SBOM
files - where we generate list of dependencies for Airflow without any provider installed.
"""
- core_dependencies = get_core_airflow_dependencies()
uninstall_all_packages(config_params)
console.print(
- f"[bright_blue]Installing airflow with [{core_dependencies}] extras only " f"with eager upgrade."
+ "[bright_blue]Installing airflow with [all-core] extras only with eager upgrade in "
+ "installable mode."
)
install_local_airflow_with_eager_upgrade(
- config_params, config_params.eager_upgrade_additional_requirements, core_dependencies
+ config_params, config_params.eager_upgrade_additional_requirements
)
- console.print(f"[success]Installed airflow with [{core_dependencies}] extras only with eager upgrade.")
+ console.print("[success]Installed airflow with [all-core] extras only with eager upgrade.")
with config_params.current_constraints_file.open("w") as constraints_file:
constraints_file.write(NO_PROVIDERS_CONSTRAINTS_PREFIX)
freeze_packages_to_file(config_params, constraints_file)
diff --git a/scripts/in_container/run_prepare_airflow_packages.py b/scripts/in_container/run_prepare_airflow_packages.py
index 4b319a722c..2f6f1912c2 100755
--- a/scripts/in_container/run_prepare_airflow_packages.py
+++ b/scripts/in_container/run_prepare_airflow_packages.py
@@ -22,95 +22,121 @@ import os
import re
import subprocess
import sys
+from contextlib import contextmanager
from pathlib import Path
from shutil import rmtree
-import rich
-
-
-def process_summary(success_message: str, error_message: str, completed_process: subprocess.CompletedProcess):
- if completed_process.returncode != 0:
- if os.environ.get("GITHUB_ACTIONS", "") != "":
- print("::endgroup::")
- print(f"::error::{error_message}")
- rich.print(f"[red]{error_message}")
- rich.print(completed_process.stdout)
- rich.print(completed_process.stderr)
- sys.exit(completed_process.returncode)
- else:
- rich.print(f"[green]{success_message}")
+from rich.console import Console
+console = Console(color_system="standard", width=200)
AIRFLOW_SOURCES_ROOT = Path(__file__).parents[2].resolve()
+AIRFLOW_INIT_FILE = AIRFLOW_SOURCES_ROOT / "airflow" / "__init__.py"
WWW_DIRECTORY = AIRFLOW_SOURCES_ROOT / "airflow" / "www"
+VERSION_SUFFIX = os.environ.get("VERSION_SUFFIX_FOR_PYPI", "")
+PACKAGE_FORMAT = os.environ.get("PACKAGE_FORMAT", "wheel")
+
+
+def clean_build_directory():
+ console.print("[bright_blue]Cleaning build directories\n")
+ for egg_info_file in AIRFLOW_SOURCES_ROOT.glob("*egg-info*"):
+ rmtree(egg_info_file, ignore_errors=True)
+ rmtree(AIRFLOW_SOURCES_ROOT / "build", ignore_errors=True)
+ console.print("[green]Cleaned build directories\n\n")
+
+
+def mark_git_directory_as_safe():
+ console.print(f"[bright_blue]Marking {AIRFLOW_SOURCES_ROOT} as safe directory for git commands.\n")
+ subprocess.run(
+ ["git", "config", "--global", "--unset-all", "safe.directory"],
+ cwd=AIRFLOW_SOURCES_ROOT,
+ stdout=subprocess.DEVNULL,
+ stderr=subprocess.DEVNULL,
+ check=False,
+ )
+ subprocess.run(
+ ["git", "config", "--global", "--add", "safe.directory", AIRFLOW_SOURCES_ROOT],
+ cwd=AIRFLOW_SOURCES_ROOT,
+ stdout=subprocess.DEVNULL,
+ stderr=subprocess.DEVNULL,
+ check=False,
+ )
+ console.print(f"[green]Marked {AIRFLOW_SOURCES_ROOT} as safe directory for git commands.\n")
+
+
+def get_current_airflow_version() -> str:
+ console.print("[bright_blue]Checking airflow version\n")
+ airflow_version = subprocess.check_output(
+ [sys.executable, "-m", "hatch", "version"], text=True, cwd=AIRFLOW_SOURCES_ROOT
+ ).strip()
+ console.print(f"[green]Airflow version: {airflow_version}\n")
+ return airflow_version
+
+
+def build_airflow_packages(package_format: str):
+ build_command = [sys.executable, "-m", "hatch", "build", "-t", "custom"]
+
+ if package_format in ["both", "wheel"]:
+ build_command.extend(["-t", "wheel"])
+ if package_format in ["both", "sdist"]:
+ build_command.extend(["-t", "sdist"])
+
+ console.print(f"[bright_blue]Building packages: {package_format}\n")
+ build_process = subprocess.run(build_command, capture_output=False, cwd=AIRFLOW_SOURCES_ROOT)
+
+ if build_process.returncode != 0:
+ console.print("[red]Error building Airflow packages")
+ sys.exit(build_process.returncode)
+ else:
+ console.print("[green]Airflow packages built successfully")
+
+
+def set_package_version(version: str) -> None:
+ console.print(f"\n[yellow]Setting {version} for Airflow package\n")
+ # replace __version__ with the version passed as argument in python
+ init_content = AIRFLOW_INIT_FILE.read_text()
+ init_content = re.sub(r'__version__ = "[^"]+"', f'__version__ = "{version}"', init_content)
+ AIRFLOW_INIT_FILE.write_text(init_content)
+
+
+@contextmanager
+def package_version(version_suffix: str):
+ release_version_matcher = re.compile(r"^\d+\.\d+\.\d+$")
+ airflow_version = get_current_airflow_version()
+
+ update_version = False
+ if version_suffix:
+ if airflow_version.endswith(f".{version_suffix}"):
+ console.print(
+ f"[bright_blue]The {airflow_version} already has suffix {version_suffix}. Not updating it.\n"
+ )
+ elif not release_version_matcher.match(airflow_version):
+ console.print(
+ f"[red]You should only pass version suffix if {airflow_version}"
+ f"does not have suffix in code. The version in code is: {airflow_version}.\n"
+ )
+ console.print(
+ "[yellow]Make sure that you remove the suffix before using `--version-suffix-for-pypi`!"
+ )
+ sys.exit(1)
+ else:
+ update_version = True
+ if update_version:
+ set_package_version(f"{airflow_version}.{version_suffix}")
+ try:
+ yield
+ finally:
+ # Set the version back to the original version
+ if update_version:
+ set_package_version(airflow_version)
+
+
+clean_build_directory()
+mark_git_directory_as_safe()
+
+with package_version(VERSION_SUFFIX):
+ build_airflow_packages(PACKAGE_FORMAT)
-rich.print("[bright_blue]Cleaning build directories\n")
-
-for egg_info_file in AIRFLOW_SOURCES_ROOT.glob("*egg-info*"):
- rmtree(egg_info_file, ignore_errors=True)
-
-rmtree(AIRFLOW_SOURCES_ROOT / "build", ignore_errors=True)
-
-rich.print("[green]Cleaned build directories\n\n")
-
-version_suffix = os.environ.get("VERSION_SUFFIX_FOR_PYPI", "")
-package_format = os.environ.get("PACKAGE_FORMAT", "wheel")
-
-rich.print(f"[bright_blue]Marking {AIRFLOW_SOURCES_ROOT} as safe directory for git commands.\n")
-
-subprocess.run(
- ["git", "config", "--global", "--unset-all", "safe.directory"],
- cwd=AIRFLOW_SOURCES_ROOT,
- stdout=subprocess.DEVNULL,
- stderr=subprocess.DEVNULL,
- check=False,
-)
-
-subprocess.run(
- ["git", "config", "--global", "--add", "safe.directory", AIRFLOW_SOURCES_ROOT],
- cwd=AIRFLOW_SOURCES_ROOT,
- stdout=subprocess.DEVNULL,
- stderr=subprocess.DEVNULL,
- check=True,
-)
-
-rich.print(f"[green]Marked {AIRFLOW_SOURCES_ROOT} as safe directory for git commands.\n")
-
-rich.print("[bright_blue]Checking airflow version\n")
-
-airflow_version = subprocess.check_output(
- [sys.executable, "setup.py", "--version"], text=True, cwd=AIRFLOW_SOURCES_ROOT
-).strip()
-
-rich.print(f"[green]Airflow version: {airflow_version}\n")
-
-RELEASED_VERSION_MATCHER = re.compile(r"^\d+\.\d+\.\d+$")
-
-command = [sys.executable, "setup.py"]
-
-if version_suffix:
- if RELEASED_VERSION_MATCHER.match(airflow_version):
- rich.print(f"[warning]Adding {version_suffix} suffix to the {airflow_version}")
- command.extend(["egg_info", "--tag-build", version_suffix])
- elif not airflow_version.endswith(version_suffix):
- rich.print(f"[red]Version {airflow_version} does not end with {version_suffix}. Using !")
- sys.exit(1)
-
-if package_format in ["both", "wheel"]:
- command.append("bdist_wheel")
-if package_format in ["both", "sdist"]:
- command.append("sdist")
-
-rich.print(f"[bright_blue]Building packages: {package_format}\n")
-
-process = subprocess.run(command, capture_output=True, text=True, cwd=AIRFLOW_SOURCES_ROOT)
-
-process_summary("Airflow packages built successfully", "Error building Airflow packages", process)
-
-if os.environ.get("GITHUB_ACTIONS", "") != "":
- print("::endgroup::")
-
-rich.print("[green]Packages built successfully:\n")
for file in (AIRFLOW_SOURCES_ROOT / "dist").glob("apache*"):
- rich.print(file.name)
-rich.print()
+ console.print(file.name)
+console.print()
diff --git a/scripts/in_container/verify_providers.py b/scripts/in_container/verify_providers.py
index 769f85d520..e0491963db 100755
--- a/scripts/in_container/verify_providers.py
+++ b/scripts/in_container/verify_providers.py
@@ -18,6 +18,7 @@
from __future__ import annotations
import importlib
+import json
import logging
import os
import pkgutil
@@ -40,6 +41,8 @@ console = Console(width=400, color_system="standard")
AIRFLOW_SOURCES_ROOT = Path(__file__).parents[2].resolve()
PROVIDERS_PATH = AIRFLOW_SOURCES_ROOT / "airflow" / "providers"
+GENERATED_PROVIDERS_DEPENDENCIES_FILE = AIRFLOW_SOURCES_ROOT / "generated" / "provider_dependencies.json"
+ALL_DEPENDENCIES = json.loads(GENERATED_PROVIDERS_DEPENDENCIES_FILE.read_text())
USE_AIRFLOW_VERSION = os.environ.get("USE_AIRFLOW_VERSION") or ""
IS_AIRFLOW_VERSION_PROVIDED = re.match("^(\d+)\.(\d+)\.(\d+)\S*$", USE_AIRFLOW_VERSION)
@@ -129,13 +132,7 @@ EXPECTED_SUFFIXES: dict[EntityType, str] = {
def get_all_providers() -> list[str]:
- """Returns all providers for regular packages.
-
- :return: list of providers that are considered for provider packages
- """
- from setup import ALL_PROVIDERS
-
- return list(ALL_PROVIDERS)
+ return list(ALL_DEPENDENCIES.keys())
def import_all_classes(
diff --git a/scripts/tools/initialize_virtualenv.py b/scripts/tools/initialize_virtualenv.py
index e5e5633fdf..1058a21af2 100755
--- a/scripts/tools/initialize_virtualenv.py
+++ b/scripts/tools/initialize_virtualenv.py
@@ -74,10 +74,13 @@ IMPORTANT NOTE ABOUT EXTRAS !!!
You can specify extras as single coma-separated parameter to install. For example
-* google,amazon,microsoft.azure
-* devel_all
+* devel - to have all development dependencies required to test core.
+* devel-* - to selectively install tools that we use to run scripts, tests, static checks etc.
+* google,amazon,microsoft_azure - to install dependencies needed at runtime by specified providers
+* devel-all-dbs - to have all development dependencies required for all DB providers
+* devel-all - to have all development dependencies required for all providers
-Note that "devel_all" installs all possible dependencies and we have > 600 of them,
+Note that "devel-all" installs all possible dependencies and we have > 600 of them,
which might not be possible to install cleanly on your host because of lack of
system packages. It's easier to install extras one-by-one as needed.
diff --git a/setup.cfg b/setup.cfg
deleted file mode 100644
index ab7b34511c..0000000000
--- a/setup.cfg
+++ /dev/null
@@ -1,244 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-[metadata]
-name = apache-airflow
-summary = Programmatically author, schedule and monitor data pipelines
-author = Apache Software Foundation
-author_email = dev@airflow.apache.org
-url = https://airflow.apache.org/
-version = attr: airflow.__version__
-long_description = file: generated/PYPI_README.md
-long_description_content_type = text/markdown
-license = Apache License 2.0
-license_files =
- LICENSE
- NOTICE
- licenses/*.txt
-classifiers =
- Development Status :: 5 - Production/Stable
- Environment :: Console
- Environment :: Web Environment
- Intended Audience :: Developers
- Intended Audience :: System Administrators
- License :: OSI Approved :: Apache Software License
- Programming Language :: Python :: 3.8
- Programming Language :: Python :: 3.9
- Programming Language :: Python :: 3.10
- Programming Language :: Python :: 3.11
- Topic :: System :: Monitoring
- Framework :: Apache Airflow
-project_urls =
- Bug Tracker=https://github.com/apache/airflow/issues
- Documentation=https://airflow.apache.org/docs/
- Downloads=https://archive.apache.org/dist/airflow/
- Release Notes=https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html
- Slack Chat=https://s.apache.org/airflow-slack
- Source Code=https://github.com/apache/airflow
- Twitter=https://twitter.com/ApacheAirflow
- YouTube=https://www.youtube.com/channel/UCSXwxpWZQ7XZ1WL3wqevChA/
-
-[options]
-zip_safe = False
-include_package_data = True
-# Mainly because of distutils deprecation and some packages not being compatible with it, we should
-# Limit airflow to < 3.12 until those dependencies are ready and until we can support Python 3.12
-python_requires = ~=3.8,<3.12
-packages = find:
-setup_requires =
- gitpython
- wheel
-#####################################################################################################
-# IMPORTANT NOTE!!!!!!!!!!!!!!!
-# IF you are removing dependencies from this list, please make sure that you also increase
-# DEPENDENCIES_EPOCH_NUMBER in the Dockerfile.ci
-#####################################################################################################
-install_requires =
- # Alembic is important to handle our migrations in predictable and performant way. It is developed
- # together with SQLAlchemy. Our experience with Alembic is that it very stable in minor version
- alembic>=1.6.3, <2.0
- argcomplete>=1.10
- asgiref
- attrs>=22.1.0
- blinker
- # Colorlog 6.x merges TTYColoredFormatter into ColoredFormatter, breaking backwards compatibility with 4.x
- # Update CustomTTYColoredFormatter to remove
- colorlog>=4.0.2, <5.0
- configupdater>=3.1.1
- # `airflow/www/extensions/init_views` imports `connexion.decorators.validation.RequestBodyValidator`
- # connexion v3 has refactored the entire module to middleware, see: /spec-first/connexion/issues/1525
- # Specifically, RequestBodyValidator was removed in: /spec-first/connexion/pull/1595
- # The usage was added in #30596, seemingly only to override and improve the default error message.
- # Either revert that change or find another way, preferably without using connexion internals.
- # This limit can be removed after https://github.com/apache/airflow/issues/35234 is fixed
- connexion[flask]>=2.10.0,<3.0
- cron-descriptor>=1.2.24
- croniter>=0.3.17
- cryptography>=0.9.3
- deprecated>=1.2.13
- dill>=0.2.2
- # Flask 2.3 is scheduled to introduce a number of deprecation removals - some of them might be breaking
- # for our dependencies - notably `_app_ctx_stack` and `_request_ctx_stack` removals.
- # We should remove the limitation after 2.3 is released and our dependencies are updated to handle it
- flask>=2.2,<2.3
- # We are tightly coupled with FAB version because we vendored in part of FAB code related to security manager
- # This is done as part of preparation to removing FAB as dependency, but we are not ready for it yet
- # Every time we update FAB version here, please make sure that you review the classes and models in
- # `airflow/www/fab_security` with their upstream counterparts. In particular, make sure any breaking changes,
- # for example any new methods, are accounted for.
- # NOTE! When you change the value here, you also have to update flask-appbuilder[oauth] in setup.py
- flask-appbuilder==4.3.10
- flask-caching>=1.5.0
- flask-login>=0.6.2
- flask-session>=0.4.0
- flask-wtf>=0.15
- fsspec>=2023.10.0
- google-re2>=1.0
- gunicorn>=20.1.0
- httpx
- importlib_metadata>=1.7;python_version<"3.9"
- importlib_resources>=5.2;python_version<"3.9"
- itsdangerous>=2.0
- jinja2>=3.0.0
- jsonschema>=4.18.0
- lazy-object-proxy
- linkify-it-py>=2.0.0
- lockfile>=0.12.2
- markdown>=3.0
- markdown-it-py>=2.1.0
- markupsafe>=1.1.1
- marshmallow-oneofschema>=2.0.1
- mdit-py-plugins>=0.3.0
- opentelemetry-api>=1.15.0
- opentelemetry-exporter-otlp
- packaging>=14.0
- pathspec>=0.9.0
- # When (if) pendulum 3 released it would introduce changes in module/objects imports,
- # since we are tightly coupled with pendulum library internally it will breaks Airflow functionality.
- pendulum>=2.0,<3.0
- pluggy>=1.0
- psutil>=4.2.0
- pydantic>=2.3.0
- pygments>=2.0.1
- pyjwt>=2.0.0
- python-daemon>=3.0.0
- python-dateutil>=2.3
- python-nvd3>=0.15.0
- python-slugify>=5.0
- rfc3339_validator>=0.1.4
- rich>=12.4.4
- rich-argparse>=1.0.0
- setproctitle>=1.1.8
- # We use some deprecated features of sqlalchemy 2.0 and we should replace them before we can upgrade
- # See https://sqlalche.me/e/b8d9 for details of deprecated features
- # you can set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings.
- # The issue tracking it is https://github.com/apache/airflow/issues/28723
- sqlalchemy>=1.4.28,<2.0
- sqlalchemy_jsonfield>=1.0
- tabulate>=0.7.5
- tenacity>=6.2.0,!=8.2.0
- termcolor>=1.1.0
- typing-extensions>=4.0.0
- # We should remove this dependency when Providers are limited to Airflow 2.7+
- # as we replaced the usage of unicodecsv with csv in Airflow 2.7
- # See https://github.com/apache/airflow/pull/31693
- # We should also remove "licenses/LICENSE-unicodecsv.txt" file when we remove this dependency
- unicodecsv>=0.14.1
- universal_pathlib>=0.1.4
- # Werkzug 3 breaks Flask-Login 0.6.2
- # we should remove this limitation when FAB supports Flask 2.3
- werkzeug>=2.0,<3
-
-[options.packages.find]
-include =
- airflow*
-
-[options.package_data]
-airflow=
- py.typed
- alembic.ini
- git_version
- customized_form_field_behaviours.schema.json
- provider_info.schema.json
-
-airflow.api_connexion.openapi=*.yaml
-airflow.auth.managers.fab.openapi=*.yaml
-airflow.serialization=*.json
-airflow.utils=
- context.pyi
-
-[options.data_files]
-generated=
- generated/provider_dependencies.json
-
-[options.entry_points]
-console_scripts=
- airflow=airflow.__main__:main
-
-[bdist_wheel]
-python-tag=py3
-
-
-[files]
-packages = airflow
-
-[easy_install]
-
-[mypy]
-ignore_missing_imports = True
-no_implicit_optional = True
-warn_redundant_casts = True
-warn_unused_ignores = False
-plugins =
- dev/mypy/plugin/decorators.py,
- dev/mypy/plugin/outputs.py
-pretty = True
-show_error_codes = True
-# Mypy since 0.991 warns about type annotations being present in an untyped
-# function since they are not checked and technically meaningless. Ideally we
-# should make them meaningful by setting check_untyped_defs = True, but there
-# are too many issues in those currently unchecked functions to make it viable
-# in the short term, so this error is simply ignored for now.
-disable_error_code = annotation-unchecked
-
-[mypy-airflow.config_templates.default_webserver_config]
-# This file gets written to user installs, so lets not litter it with type comments
-disable_error_code = var-annotated
-
-[mypy-airflow.migrations.*]
-ignore_errors = True
-
-# Let's assume all google.cloud packages have implicit optionals
-# Most of them don't but even if they do, it does not matter
-[mypy-google.cloud.*]
-no_implicit_optional = False
-
-# Let's assume all azure packages have implicit optionals
-[mypy-azure.*]
-no_implicit_optional = False
-
-
-[mypy-referencing.*]
-# Referencing has some old type annotations that are not compatible with new versions of mypy
-ignore_errors = True
-
-
-[mypy-apache_beam.*]
-# Beam has some old type annotations and they introduced an error recently with bad signature of
-# a function. This is captured in https://github.com/apache/beam/issues/29927
-# and we should remove this exclusion when it is fixed.
-ignore_errors = True
diff --git a/setup.py b/setup.py
deleted file mode 100644
index 1c831fb98d..0000000000
--- a/setup.py
+++ /dev/null
@@ -1,1084 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-"""Setup.py for the Airflow project."""
-# To make sure the CI build is using "upgrade to newer dependencies", which is useful when you want to check
-# if the dependencies are still compatible with the latest versions as they seem to break some unrelated
-# tests in main, you can modify this file. The modification can be simply modifying this particular comment.
-# e.g. you can modify the following number "00001" to something else to trigger it.
-from __future__ import annotations
-
-import glob
-import json
-import logging
-import os
-import subprocess
-import sys
-import textwrap
-import unittest
-from copy import deepcopy
-from pathlib import Path
-from typing import Iterable
-
-from setuptools import Command, Distribution, find_namespace_packages, setup
-from setuptools.command.develop import develop as develop_orig
-from setuptools.command.install import install as install_orig
-
-# Setuptools patches this import to point to a vendored copy instead of the
-# stdlib, which is deprecated in Python 3.10 and will be removed in 3.12.
-from distutils import log # isort: skip
-
-
-# Controls whether providers are installed from packages or directly from sources
-# It is turned on by default in case of development environments such as Breeze
-# And it is particularly useful when you add a new provider and there is no
-# PyPI version to install the provider package from
-INSTALL_PROVIDERS_FROM_SOURCES = "INSTALL_PROVIDERS_FROM_SOURCES"
-PY39 = sys.version_info >= (3, 9)
-
-logger = logging.getLogger(__name__)
-
-AIRFLOW_SOURCES_ROOT = Path(__file__).parent.resolve()
-PROVIDERS_ROOT = AIRFLOW_SOURCES_ROOT / "airflow" / "providers"
-
-CROSS_PROVIDERS_DEPS = "cross-providers-deps"
-DEPS = "deps"
-CURRENT_PYTHON_VERSION = f"{sys.version_info.major}.{sys.version_info.minor}"
-
-
-def apply_pypi_suffix_to_airflow_packages(dependencies: list[str]) -> None:
- """
- Apply version suffix to dependencies that do not have one.
-
- Looks through the list of dependencies, finds which one are airflow or airflow providers packages
- and applies the version suffix to those of them that do not have the suffix applied yet.
-
- :param dependencies: list of dependencies to add suffix to
- """
- for i in range(len(dependencies)):
- dependency = dependencies[i]
- if dependency.startswith("apache-airflow"):
- # in case we want to depend on other airflow package, the chance is the package
- # has not yet been released to PyPI and we only see it as a local package that is
- # being installed with .dev0 suffix in CI. Unfortunately, there is no way in standard
- # PEP-440 compliant way to specify version that would be both - releasable, and
- # testable to install on CI with .dev0 or .rc suffixes. We could add `--pre` flag to
- # enable it, but `--pre` flag is not selective and will work for all packages so
- # we would automatically install all "pre-release" packages for all packages that
- # we install from PyPI - and this is definitely not what we want. So in order to
- # install only airflow packages that are available in sources in .dev0 or .rc version
- # we need to dynamically modify the dependencies here.
- if ">=" in dependency:
- package, version = dependency.split(">=")
- version_spec = f">={version}"
- version_suffix = os.environ.get("VERSION_SUFFIX_FOR_PYPI")
- if version_suffix and version_suffix not in version_spec:
- version_spec += version_suffix
- dependencies[i] = f"{package}{version_spec}"
-
-
-# NOTE! IN Airflow 2.4.+ dependencies for providers are maintained in `provider.yaml` files for each
-# provider separately. They are loaded here and if you want to modify them, you need to modify
-# corresponding provider.yaml file.
-#
-def fill_provider_dependencies() -> dict[str, dict[str, list[str]]]:
- # in case we are loading setup from pre-commits, we want to skip the check for python version
- # because if someone uses a version of Python where providers are excluded, the setup will fail
- # to see the extras for those providers
- skip_python_version_check = os.environ.get("_SKIP_PYTHON_VERSION_CHECK")
- try:
- with AIRFLOW_SOURCES_ROOT.joinpath("generated", "provider_dependencies.json").open() as f:
- dependencies = json.load(f)
- provider_dict = {}
- for key, value in dependencies.items():
- if value["state"] in ["suspended", "removed"]:
- continue
- if value.get(DEPS):
- apply_pypi_suffix_to_airflow_packages(value[DEPS])
- if CURRENT_PYTHON_VERSION not in value["excluded-python-versions"] or skip_python_version_check:
- provider_dict[key] = value
- return provider_dict
- except Exception as e:
- print(f"Exception while loading provider dependencies {e}")
- # we can ignore loading dependencies when they are missing - they are only used to generate
- # correct extras when packages are build and when we install airflow from sources
- # (in both cases the provider_dependencies should be present).
- return {}
-
-
-PROVIDER_DEPENDENCIES = fill_provider_dependencies()
-
-
-def airflow_test_suite() -> unittest.TestSuite:
- """Test suite for Airflow tests."""
- test_loader = unittest.TestLoader()
- test_suite = test_loader.discover(str(AIRFLOW_SOURCES_ROOT / "tests"), pattern="test_*.py")
- return test_suite
-
-
-class CleanCommand(Command):
- """
- Command to tidy up the project root.
-
- Registered as cmdclass in setup() so it can be called with ``python setup.py extra_clean``.
- """
-
- description = "Tidy up the project root"
- user_options: list[str] = []
-
- def initialize_options(self) -> None:
- """Set default values for options."""
-
- def finalize_options(self) -> None:
- """Set final values for options."""
-
- @staticmethod
- def rm_all_files(files: list[str]) -> None:
- """Remove all files from the list."""
- for file in files:
- try:
- os.remove(file)
- except Exception as e:
- logger.warning("Error when removing %s: %s", file, e)
-
- def run(self) -> None:
- """Remove temporary files and directories."""
- os.chdir(str(AIRFLOW_SOURCES_ROOT))
- self.rm_all_files(glob.glob("./build/*"))
- self.rm_all_files(glob.glob("./**/__pycache__/*", recursive=True))
- self.rm_all_files(glob.glob("./**/*.pyc", recursive=True))
- self.rm_all_files(glob.glob("./dist/*"))
- self.rm_all_files(glob.glob("./*.egg-info"))
- self.rm_all_files(glob.glob("./docker-context-files/*.whl"))
- self.rm_all_files(glob.glob("./docker-context-files/*.tgz"))
-
-
-class CompileAssets(Command):
- """
- Compile and build the frontend assets using yarn and webpack.
-
- Registered as cmdclass in setup() so it can be called with ``python setup.py compile_assets``.
- """
-
- description = "Compile and build the frontend assets"
- user_options: list[str] = []
-
- def initialize_options(self) -> None:
- """Set default values for options."""
-
- def finalize_options(self) -> None:
- """Set final values for options."""
-
- def run(self) -> None:
- """Run a command to compile and build assets."""
- www_dir = AIRFLOW_SOURCES_ROOT / "airflow" / "www"
- subprocess.check_call(["yarn", "install", "--frozen-lockfile"], cwd=str(www_dir))
- subprocess.check_call(["yarn", "run", "build"], cwd=str(www_dir))
-
-
-class ListExtras(Command):
- """
- List all available extras.
-
- Registered as cmdclass in setup() so it can be called with ``python setup.py list_extras``.
- """
-
- description = "List available extras"
- user_options: list[str] = []
-
- def initialize_options(self) -> None:
- """Set default values for options."""
-
- def finalize_options(self) -> None:
- """Set final values for options."""
-
- def run(self) -> None:
- """List extras."""
- print("\n".join(textwrap.wrap(", ".join(EXTRAS_DEPENDENCIES.keys()), 100)))
-
-
-def git_version() -> str:
- """
- Return a version to identify the state of the underlying git repo.
-
- The version will indicate whether the head of the current git-backed working directory
- is tied to a release tag or not : it will indicate the former with a 'release:{version}'
- prefix and the latter with a '.dev0' suffix. Following the prefix will be a sha of the
- current branch head. Finally, a "dirty" suffix is appended to indicate that uncommitted
- changes are present.
-
- :return: Found Airflow version in Git repo
- """
- try:
- import git
-
- try:
- repo = git.Repo(str(AIRFLOW_SOURCES_ROOT / ".git"))
- except git.NoSuchPathError:
- logger.warning(".git directory not found: Cannot compute the git version")
- return ""
- except git.InvalidGitRepositoryError:
- logger.warning("Invalid .git directory not found: Cannot compute the git version")
- return ""
- except ImportError:
- logger.warning("gitpython not found: Cannot compute the git version.")
- return ""
- if repo:
- sha = repo.head.commit.hexsha
- if repo.is_dirty():
- return f".dev0+{sha}.dirty"
- # commit is clean
- return f".release:{sha}"
- return "no_git_version"
-
-
-def write_version(filename: str = str(AIRFLOW_SOURCES_ROOT / "airflow" / "git_version")) -> None:
- """
- Write the Semver version + git hash to file, e.g. ".dev0+2f635dc265e78db6708f59f68e8009abb92c1e65".
-
- :param str filename: Destination file to write.
- """
- text = git_version()
- with open(filename, "w") as file:
- file.write(text)
-
-
-#
-# NOTE! IN Airflow 2.4.+ dependencies for providers are maintained in `provider.yaml` files for each
-# provider separately. Before, the provider dependencies were kept here. THEY ARE NOT HERE ANYMORE.
-#
-# 'Start dependencies group' and 'End dependencies group' are marks for ./scripts/ci/check_order_setup.py
-# If you change these marks you should also change ./scripts/ci/check_order_setup.py
-# Start dependencies group
-async_packages = [
- "eventlet>=0.33.3",
- "gevent>=0.13",
- "greenlet>=0.4.9",
-]
-atlas = [
- "atlasclient>=0.1.2",
-]
-celery = [
- # The Celery is known to introduce problems when upgraded to a MAJOR version. Airflow Core
- # Uses Celery for CeleryExecutor, and we also know that Kubernetes Python client follows SemVer
- # (https://docs.celeryq.dev/en/stable/contributing.html?highlight=semver#versions).
- # This is a crucial component of Airflow, so we should limit it to the next MAJOR version and only
- # deliberately bump the version when we tested it, and we know it can be bumped.
- # Bumping this version should also be connected with
- # limiting minimum airflow version supported in celery provider due to the
- # potential breaking changes in Airflow Core as well (celery is added as extra, so Airflow
- # core is not hard-limited via install-requires, only by extra).
- "celery>=5.3.0,<6"
-]
-cgroups = [
- # Cgroupspy 0.2.2 added Python 3.10 compatibility
- "cgroupspy>=0.2.2",
-]
-deprecated_api = [
- "requests>=2.26.0",
-]
-doc = [
- # sphinx-autoapi fails with astroid 3.0, see: https://github.com/readthedocs/sphinx-autoapi/issues/407
- # This was fixed in sphinx-autoapi 3.0, however it has requirement sphinx>=6.1, but we stuck on 5.x
- "astroid>=2.12.3, <3.0",
- "checksumdir",
- # click 8.1.4 and 8.1.5 generate mypy errors due to typing issue in the upstream package:
- # https://github.com/pallets/click/issues/2558
- "click>=8.0,!=8.1.4,!=8.1.5",
- # Docutils 0.17.0 converts generated <div class="section"> into <section> and breaks our doc formatting
- # By adding a lot of whitespace separation. This limit can be lifted when we update our doc to handle
- # <section> tags for sections
- "docutils<0.17.0",
- "eralchemy2",
- "sphinx-airflow-theme",
- "sphinx-argparse>=0.1.13",
- "sphinx-autoapi>=2.0.0",
- "sphinx-copybutton",
- "sphinx-design>=0.5.0",
- "sphinx-jinja>=2.0",
- "sphinx-rtd-theme>=0.1.6",
- "sphinx>=5.2.0",
- "sphinxcontrib-httpdomain>=1.7.0",
- "sphinxcontrib-redoc>=1.6.0",
- "sphinxcontrib-spelling>=7.3",
-]
-doc_gen = [
- "eralchemy2",
- "graphviz>=0.12",
-]
-flask_appbuilder_oauth = [
- "authlib>=1.0.0",
- # The version here should be upgraded at the same time as flask-appbuilder in setup.cfg
- "flask-appbuilder[oauth]==4.3.10",
-]
-graphviz = ["graphviz>=0.12"]
-kerberos = [
- "pykerberos>=1.1.13",
- "requests_kerberos>=0.10.0",
- "thrift_sasl>=0.2.0",
-]
-kubernetes = [
- # The Kubernetes API is known to introduce problems when upgraded to a MAJOR version. Airflow Core
- # Uses Kubernetes for Kubernetes executor, and we also know that Kubernetes Python client follows SemVer
- # (https://github.com/kubernetes-client/python#compatibility). This is a crucial component of Airflow
- # So we should limit it to the next MAJOR version and only deliberately bump the version when we
- # tested it, and we know it can be bumped. Bumping this version should also be connected with
- # limiting minimum airflow version supported in cncf.kubernetes provider, due to the
- # potential breaking changes in Airflow Core as well (kubernetes is added as extra, so Airflow
- # core is not hard-limited via install-requires, only by extra).
- "cryptography>=2.0.0",
- "kubernetes>=21.7.0,<24",
-]
-ldap = [
- "ldap3>=2.5.1",
- "python-ldap",
-]
-leveldb = ["plyvel"]
-otel = ["opentelemetry-exporter-prometheus"]
-pandas = [
- "pandas>=1.2.5",
-]
-password = [
- "bcrypt>=2.0.0",
- "flask-bcrypt>=0.7.1",
-]
-rabbitmq = [
- "amqp",
-]
-sentry = [
- "blinker>=1.1",
- # Sentry SDK 1.33 is broken when greenlets are installed and fails to import
- # See https://github.com/getsentry/sentry-python/issues/2473
- "sentry-sdk>=1.32.0,!=1.33.0",
-]
-statsd = [
- "statsd>=3.3.0",
-]
-virtualenv = [
- "virtualenv",
-]
-webhdfs = [
- "hdfs[avro,dataframe,kerberos]>=2.0.4",
-]
-# End dependencies group
-
-# Mypy 0.900 and above ships only with stubs from stdlib so if we need other stubs, we need to install them
-# manually as `types-*`. See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
-# for details. We want to install them explicitly because we want to eventually move to
-# mypyd which does not support installing the types dynamically with --install-types
-mypy_dependencies = [
- # TODO: upgrade to newer versions of MyPy continuously as they are released
- # Make sure to upgrade the mypy version in update-common-sql-api-stubs in .pre-commit-config.yaml
- # when you upgrade it here !!!!
- "mypy==1.2.0",
- "types-aiofiles",
- "types-certifi",
- "types-croniter",
- "types-Deprecated",
- "types-docutils",
- "types-paramiko",
- "types-protobuf",
- "types-python-dateutil",
- "types-python-slugify",
- "types-pytz",
- "types-redis",
- "types-requests",
- "types-setuptools",
- "types-termcolor",
- "types-tabulate",
- "types-toml",
- "types-Markdown",
- "types-PyMySQL",
- "types-PyYAML",
-]
-
-# make sure to update providers/amazon/provider.yaml botocore min version when you update it here
-_MIN_BOTO3_VERSION = "1.33.0"
-
-_devel_only_amazon = [
- "aws_xray_sdk>=2.12.0",
- "moto[cloudformation,glue]>=4.2.12",
- f"mypy-boto3-appflow>={_MIN_BOTO3_VERSION}",
- f"mypy-boto3-rds>={_MIN_BOTO3_VERSION}",
- f"mypy-boto3-redshift-data>={_MIN_BOTO3_VERSION}",
- f"mypy-boto3-s3>={_MIN_BOTO3_VERSION}",
-]
-
-_devel_only_azure = [
- "pywinrm",
-]
-
-_devel_only_breeze = [
- "filelock",
-]
-
-_devel_only_debuggers = [
- "ipdb",
-]
-
-_devel_only_deltalake = [
- "deltalake>=0.12.0",
-]
-
-_devel_only_devscripts = [
- "click>=8.0",
- "gitpython",
- "pipdeptree",
- "pygithub",
- "rich-click>=1.7.0",
- "restructuredtext-lint",
- "semver",
- "towncrier",
- "twine",
- "wheel",
-]
-
-_devel_only_duckdb = [
- "duckdb>=0.9.0",
-]
-
-_devel_only_mongo = [
- "mongomock",
-]
-
-_devel_only_iceberg = [
- "pyiceberg>=0.5.0",
-]
-
-_devel_only_sentry = [
- "blinker",
-]
-
-_devel_only_static_checks = [
- "pre-commit",
- "black",
- "ruff==0.1.11",
- "yamllint",
-]
-
-_devel_only_tests = [
- "aioresponses",
- "backports.zoneinfo>=0.2.1;python_version<'3.9'",
- "beautifulsoup4>=4.7.1",
- "coverage>=7.2",
- "pytest>=7.1",
- # Pytest-asyncio 0.23.0 and 0.23.1 break test collection
- # See https://github.com/pytest-dev/pytest-asyncio/issues/703 for details.
- "pytest-asyncio!=0.23.0,!=0.23.1",
- "pytest-cov",
- "pytest-httpx",
- "pytest-icdiff",
- "pytest-instafail",
- "pytest-mock",
- "pytest-rerunfailures",
- "pytest-timeouts",
- "pytest-xdist",
- "requests_mock",
- "time-machine",
-]
-
-# Dependencies needed for development only
-devel_only = [
- *_devel_only_amazon,
- *_devel_only_azure,
- *_devel_only_breeze,
- *_devel_only_debuggers,
- *_devel_only_deltalake,
- *_devel_only_devscripts,
- *_devel_only_duckdb,
- *_devel_only_mongo,
- *_devel_only_iceberg,
- *_devel_only_sentry,
- *_devel_only_static_checks,
- *_devel_only_tests,
-]
-
-aiobotocore = [
- # This required for AWS deferrable operators.
- # There is conflict between boto3 and aiobotocore dependency botocore.
- # TODO: We can remove it once boto3 and aiobotocore both have compatible botocore version or
- # boto3 have native aync support and we move away from aio aiobotocore
- "aiobotocore>=2.1.1",
-]
-
-s3fs = [
- # This is required for support of S3 file system which uses aiobotocore
- # which can have a conflict with boto3 as mentioned above
- "s3fs>=2023.10.0",
-]
-
-saml = [
- # This is required for support of SAML which might be used by some providers (e.g. Amazon)
- "python3-saml>=1.16.0",
-]
-
-
-def get_provider_dependencies(provider_name: str) -> list[str]:
- if provider_name not in PROVIDER_DEPENDENCIES:
- return []
- return PROVIDER_DEPENDENCIES[provider_name][DEPS]
-
-
-def get_unique_dependency_list(req_list_iterable: Iterable[list[str]]):
- _all_reqs: set[str] = set()
- for req_list in req_list_iterable:
- for req in req_list:
- _all_reqs.add(req)
- return list(_all_reqs)
-
-
-devel = get_unique_dependency_list(
- [
- aiobotocore,
- cgroups,
- devel_only,
- doc,
- kubernetes,
- mypy_dependencies,
- get_provider_dependencies("mysql"),
- pandas,
- password,
- s3fs,
- saml,
- ]
-)
-
-devel_hadoop = get_unique_dependency_list(
- [
- devel,
- get_provider_dependencies("apache.hdfs"),
- get_provider_dependencies("apache.hive"),
- get_provider_dependencies("apache.hdfs"),
- get_provider_dependencies("apache.hive"),
- get_provider_dependencies("apache.impala"),
- kerberos,
- get_provider_dependencies("presto"),
- webhdfs,
- ]
-)
-
-# Those are all additional extras which do not have their own 'providers'
-# The 'apache.atlas' and 'apache.webhdfs' are extras that provide additional libraries
-# but they do not have separate providers (yet?), they are merely there to add extra libraries
-# That can be used in custom python/bash operators.
-ADDITIONAL_EXTRAS_DEPENDENCIES: dict[str, list[str]] = {
- "apache.atlas": atlas,
- "apache.webhdfs": webhdfs,
-}
-
-# Those are extras that are extensions of the 'core' Airflow. They provide additional features
-# To airflow core. They do not have separate providers because they do not have any operators/hooks etc.
-CORE_EXTRAS_DEPENDENCIES: dict[str, list[str]] = {
- "aiobotocore": aiobotocore,
- "async": async_packages,
- "celery": celery, # TODO: remove and move to a regular provider package in a separate PR
- "cgroups": cgroups,
- "cncf.kubernetes": kubernetes, # TODO: remove and move to a regular provider package in a separate PR
- "deprecated_api": deprecated_api,
- "github_enterprise": flask_appbuilder_oauth,
- "google_auth": flask_appbuilder_oauth,
- "graphviz": graphviz,
- "kerberos": kerberos,
- "ldap": ldap,
- "leveldb": leveldb,
- "otel": otel,
- "pandas": pandas,
- "password": password,
- "rabbitmq": rabbitmq,
- "s3fs": s3fs,
- "saml": saml,
- "sentry": sentry,
- "statsd": statsd,
- "virtualenv": virtualenv,
-}
-
-
-def filter_out_excluded_extras() -> Iterable[tuple[str, list[str]]]:
- for key, value in CORE_EXTRAS_DEPENDENCIES.items():
- if value:
- yield key, value
- else:
- print(f"Removing extra {key} as it has been excluded")
-
-
-CORE_EXTRAS_DEPENDENCIES = dict(filter_out_excluded_extras())
-
-EXTRAS_DEPENDENCIES: dict[str, list[str]] = deepcopy(CORE_EXTRAS_DEPENDENCIES)
-
-
-def add_extras_for_all_providers() -> None:
- for provider_name, provider_dict in PROVIDER_DEPENDENCIES.items():
- EXTRAS_DEPENDENCIES[provider_name] = provider_dict[DEPS]
-
-
-def add_additional_extras() -> None:
- for extra_name, extra_dependencies in ADDITIONAL_EXTRAS_DEPENDENCIES.items():
- EXTRAS_DEPENDENCIES[extra_name] = extra_dependencies
-
-
-add_extras_for_all_providers()
-add_additional_extras()
-
-#############################################################################################################
-# The whole section can be removed in Airflow 3.0 as those old aliases are deprecated in 2.* series
-#############################################################################################################
-
-# Dictionary of aliases from 1.10 - deprecated in Airflow 2.*
-EXTRAS_DEPRECATED_ALIASES: dict[str, str] = {
- "atlas": "apache.atlas",
- "aws": "amazon",
- "azure": "microsoft.azure",
- "cassandra": "apache.cassandra",
- "crypto": "", # this is legacy extra - all dependencies are already "install-requires"
- "druid": "apache.druid",
- "gcp": "google",
- "gcp_api": "google",
- "hdfs": "apache.hdfs",
- "hive": "apache.hive",
- "kubernetes": "cncf.kubernetes",
- "mssql": "microsoft.mssql",
- "pinot": "apache.pinot",
- "s3": "amazon",
- "spark": "apache.spark",
- "webhdfs": "apache.webhdfs",
- "winrm": "microsoft.winrm",
-}
-
-EXTRAS_DEPRECATED_ALIASES_NOT_PROVIDERS: list[str] = [
- "crypto",
- "webhdfs",
-]
-
-EXTRAS_DEPRECATED_ALIASES_IGNORED_FROM_REF_DOCS: list[str] = [
- "jira",
-]
-
-
-def add_extras_for_all_deprecated_aliases() -> None:
- """
- Add extras for all deprecated aliases.
-
- Requirements for those deprecated aliases are the same as the extras they are replaced with.
- The dependencies are not copies - those are the same lists as for the new extras. This is intended.
- Thanks to that if the original extras are later extended with providers, aliases are extended as well.
- """
- for alias, extra in EXTRAS_DEPRECATED_ALIASES.items():
- dependencies = EXTRAS_DEPENDENCIES.get(extra) if extra != "" else []
- if dependencies is not None:
- EXTRAS_DEPENDENCIES[alias] = dependencies
-
-
-def add_all_deprecated_provider_packages() -> None:
- """
- For deprecated aliases that are providers, swap the providers dependencies to be the provider itself.
-
- e.g. {"kubernetes": ["kubernetes>=3.0.0, <12.0.0", ...]} becomes
- {"kubernetes": ["apache-airflow-provider-cncf-kubernetes"]}
- """
- for alias, provider in EXTRAS_DEPRECATED_ALIASES.items():
- if alias not in EXTRAS_DEPRECATED_ALIASES_NOT_PROVIDERS:
- replace_extra_dependencies_with_provider_packages(alias, [provider])
-
-
-add_extras_for_all_deprecated_aliases()
-
-#############################################################################################################
-# End of deprecated section
-#############################################################################################################
-
-# This is list of all providers. It's a shortcut for anyone who would like to easily get list of
-# All providers. It is used by pre-commits.
-ALL_PROVIDERS = list(PROVIDER_DEPENDENCIES.keys())
-
-ALL_DB_PROVIDERS = [
- "apache.cassandra",
- "apache.drill",
- "apache.druid",
- "apache.hdfs",
- "apache.hive",
- "apache.impala",
- "apache.pinot",
- "arangodb",
- "cloudant",
- "databricks",
- "exasol",
- "influxdb",
- "microsoft.mssql",
- "mongo",
- "mysql",
- "neo4j",
- "postgres",
- "presto",
- "trino",
- "vertica",
-]
... 444 lines suppressed ...
(airflow) 08/20: Bump min version for grpcio-status in spark provider (#36662)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 3b080e0e843680f18ed630f2c4139ebc883d0dc0
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Mon Jan 8 18:51:24 2024 +0100
Bump min version for grpcio-status in spark provider (#36662)
Previously we limited grpcio minimum version to stop backtracking
of `pip` from happening and we could not do it in the limits of
spark provider, becaue some google dependencies used it and
conflicted with it. This problem is now gone as we have newer
versions of google dependencies and we can not only safely move
it to spark provider but also bump it slightly higher to limit
the amount of backtracking we need to do.
Extracted from #36537
(cherry picked from commit ded01a5aba337882fb19e03c24d7736c7154fdd8)
---
Dockerfile.ci | 5 +----
airflow/providers/apache/spark/provider.yaml | 2 +-
generated/provider_dependencies.json | 2 +-
3 files changed, 3 insertions(+), 6 deletions(-)
diff --git a/Dockerfile.ci b/Dockerfile.ci
index 411ff9b688..3364ee234f 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -1116,14 +1116,11 @@ RUN echo "Airflow version: ${AIRFLOW_VERSION}"
# force them on the main Airflow package. Currently we need no extra limits as PIP 23.1+ has much better
# dependency resolution and we do not need to limit the versions of the dependencies
#
-# Without grpcio-status limit, pip gets into very long backtracking
-# We should attempt to remove it in the future
-#
# Aiobotocore is limited for eager upgrade because it either causes a long backtracking or
# conflict when we do not limit it. It seems that `pip` has a hard time figuring the right
# combination of dependencies for aiobotocore, botocore, boto3 and s3fs together
#
-ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="grpcio-status>=1.55.0 aiobotocore>=2.5.4"
+ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="aiobotocore>=2.5.4"
ARG UPGRADE_TO_NEWER_DEPENDENCIES="false"
ARG VERSION_SUFFIX_FOR_PYPI=""
diff --git a/airflow/providers/apache/spark/provider.yaml b/airflow/providers/apache/spark/provider.yaml
index fd5547f6a3..992deffadd 100644
--- a/airflow/providers/apache/spark/provider.yaml
+++ b/airflow/providers/apache/spark/provider.yaml
@@ -54,7 +54,7 @@ versions:
dependencies:
- apache-airflow>=2.6.0
- pyspark
- - grpcio-status
+ - grpcio-status>=1.59.0
integrations:
- integration-name: Apache Spark
diff --git a/generated/provider_dependencies.json b/generated/provider_dependencies.json
index d92a2559ea..b7007c4fe2 100644
--- a/generated/provider_dependencies.json
+++ b/generated/provider_dependencies.json
@@ -207,7 +207,7 @@
"apache.spark": {
"deps": [
"apache-airflow>=2.6.0",
- "grpcio-status",
+ "grpcio-status>=1.59.0",
"pyspark"
],
"cross-providers-deps": [
(airflow) 11/20: Bump min provider version for papermill (#36669)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 55de7176fcf679d96acf080b3212ca3fbac43d5b
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Mon Jan 8 22:57:34 2024 +0100
Bump min provider version for papermill (#36669)
Yet another long backtracking avoidance,
We are bumping REALLY OLD papermill min version (2019) with just
old from 2022.
(cherry picked from commit b5c6aead4415b53a294e1671990cbba924ed9445)
---
airflow/providers/papermill/provider.yaml | 2 +-
generated/provider_dependencies.json | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/airflow/providers/papermill/provider.yaml b/airflow/providers/papermill/provider.yaml
index 1f20ecf359..ccc762b656 100644
--- a/airflow/providers/papermill/provider.yaml
+++ b/airflow/providers/papermill/provider.yaml
@@ -45,7 +45,7 @@ versions:
dependencies:
- apache-airflow>=2.6.0
- - papermill[all]>=1.2.1
+ - papermill[all]>=2.4.0
- scrapbook[all]
- ipykernel
diff --git a/generated/provider_dependencies.json b/generated/provider_dependencies.json
index 3751cb0811..d1d3c0fe20 100644
--- a/generated/provider_dependencies.json
+++ b/generated/provider_dependencies.json
@@ -800,7 +800,7 @@
"deps": [
"apache-airflow>=2.6.0",
"ipykernel",
- "papermill[all]>=1.2.1",
+ "papermill[all]>=2.4.0",
"scrapbook[all]"
],
"cross-providers-deps": [],
(airflow) 16/20: Update providers metadata 2024-01-10 (#36718)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 4206c97ba4fbf2364090451fdf15417dd85e6be4
Author: Elad Kalif <45...@users.noreply.github.com>
AuthorDate: Wed Jan 10 21:05:01 2024 +0200
Update providers metadata 2024-01-10 (#36718)
(cherry picked from commit c59f8dee7802c03634faf8412dfb2bf122d7abc6)
---
generated/provider_metadata.json | 48 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 48 insertions(+)
diff --git a/generated/provider_metadata.json b/generated/provider_metadata.json
index 805566ee23..b56d95f495 100644
--- a/generated/provider_metadata.json
+++ b/generated/provider_metadata.json
@@ -343,6 +343,10 @@
"8.15.0": {
"associated_airflow_version": "2.8.0",
"date_released": "2023-12-31T10:31:13Z"
+ },
+ "8.16.0": {
+ "associated_airflow_version": "2.8.0",
+ "date_released": "2024-01-10T18:50:00Z"
}
},
"apache.beam": {
@@ -953,6 +957,10 @@
"6.4.0": {
"associated_airflow_version": "2.8.0",
"date_released": "2023-12-27T23:07:26Z"
+ },
+ "6.4.1": {
+ "associated_airflow_version": "2.8.0",
+ "date_released": "2024-01-10T18:50:01Z"
}
},
"apache.impala": {
@@ -1161,6 +1169,10 @@
"3.7.0": {
"associated_airflow_version": "2.8.0",
"date_released": "2023-12-12T07:17:12Z"
+ },
+ "3.7.1": {
+ "associated_airflow_version": "2.8.0",
+ "date_released": "2024-01-10T18:50:01Z"
}
},
"apache.pig": {
@@ -1407,6 +1419,10 @@
"4.6.0": {
"associated_airflow_version": "2.8.0",
"date_released": "2023-12-27T23:07:26Z"
+ },
+ "4.7.0": {
+ "associated_airflow_version": "2.8.0",
+ "date_released": "2024-01-10T18:50:01Z"
}
},
"apprise": {
@@ -1571,6 +1587,10 @@
"2.4.0": {
"associated_airflow_version": "2.8.0",
"date_released": "2023-12-12T07:17:12Z"
+ },
+ "2.5.0": {
+ "associated_airflow_version": "2.8.0",
+ "date_released": "2024-01-10T18:50:01Z"
}
},
"celery": {
@@ -1945,6 +1965,10 @@
"1.1.0": {
"associated_airflow_version": "2.8.0",
"date_released": "2023-11-29T07:17:22Z"
+ },
+ "1.2.0": {
+ "associated_airflow_version": "2.8.0",
+ "date_released": "2024-01-10T18:50:01Z"
}
},
"common.sql": {
@@ -4431,6 +4455,10 @@
"5.5.0": {
"associated_airflow_version": "2.8.0",
"date_released": "2023-12-12T07:17:12Z"
+ },
+ "5.5.1": {
+ "associated_airflow_version": "2.8.0",
+ "date_released": "2024-01-10T18:50:01Z"
}
},
"neo4j": {
@@ -4693,6 +4721,10 @@
"1.3.1": {
"associated_airflow_version": "2.8.0",
"date_released": "2023-12-27T23:07:27Z"
+ },
+ "1.4.0": {
+ "associated_airflow_version": "2.8.0",
+ "date_released": "2024-01-10T18:50:01Z"
}
},
"opensearch": {
@@ -4777,6 +4809,10 @@
"5.4.0": {
"associated_airflow_version": "2.8.0",
"date_released": "2023-12-12T07:17:12Z"
+ },
+ "5.5.0": {
+ "associated_airflow_version": "2.8.0",
+ "date_released": "2024-01-10T18:50:01Z"
}
},
"oracle": {
@@ -4949,6 +4985,10 @@
"3.5.1": {
"associated_airflow_version": "2.8.0",
"date_released": "2023-12-27T23:07:27Z"
+ },
+ "3.6.0": {
+ "associated_airflow_version": "2.8.0",
+ "date_released": "2024-01-10T18:50:01Z"
}
},
"papermill": {
@@ -5353,6 +5393,10 @@
"3.5.0": {
"associated_airflow_version": "2.8.0",
"date_released": "2023-12-12T07:17:13Z"
+ },
+ "3.6.0": {
+ "associated_airflow_version": "2.8.0",
+ "date_released": "2024-01-10T18:50:01Z"
}
},
"salesforce": {
@@ -5517,6 +5561,10 @@
"4.4.0": {
"associated_airflow_version": "2.8.0",
"date_released": "2023-12-12T07:17:13Z"
+ },
+ "4.5.0": {
+ "associated_airflow_version": "2.8.0",
+ "date_released": "2024-01-10T18:50:01Z"
}
},
"segment": {
(airflow) 20/20: Update Airflow release process to include reproducible tarballs (#36744)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit bf7fb4b01907240cecfa5a259506027935f06e80
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Fri Jan 12 17:39:50 2024 +0100
Update Airflow release process to include reproducible tarballs (#36744)
Source tarball is the main artifact produced by the release
process - one that is the "official" release and named like that
by the Apache Software Foundation.
This PR makes the source tarball generation reproducible - following
reproducibility of the `.whl` and `sdist` packages.
This change adds:
* vendors-in reproducible.py script that repacks .tar.gz package
in reproducible way using source-date-epoch as timestamps
* breeze release-management prepare-airflow-tarball command
* adds verification of the tarballs to PMC verification process
* adds --use-local-hatch for package building command to allow for
faster / non-docker build of packages for PMC verification
* improves diagnostic output of the release and build commands
(cherry picked from commit 72a571dc6d21d90f92d5ce683a5d40c6a527fcb0)
---
.rat-excludes | 2 +
3rd-party-licenses/LICENSE-reproducible.txt | 8 ++
BREEZE.rst | 24 ++++
dev/README_RELEASE_AIRFLOW.md | 15 ++-
.../commands/release_candidate_command.py | 136 +++++++++++++++------
.../commands/release_management_commands.py | 31 ++++-
.../commands/release_management_commands_config.py | 15 ++-
.../src/airflow_breeze/utils/reproducible.py | 135 ++++++++++++++++++++
images/breeze/output_release-management.svg | 54 ++++----
images/breeze/output_release-management.txt | 2 +-
..._release-management_prepare-airflow-package.svg | 28 +++--
..._release-management_prepare-airflow-package.txt | 2 +-
..._release-management_prepare-airflow-tarball.svg | 106 ++++++++++++++++
..._release-management_prepare-airflow-tarball.txt | 1 +
.../output_setup_check-all-params-in-groups.svg | 20 +--
.../output_setup_check-all-params-in-groups.txt | 2 +-
.../output_setup_regenerate-command-images.svg | 46 +++----
.../output_setup_regenerate-command-images.txt | 2 +-
.../in_container/run_prepare_airflow_packages.py | 5 +-
19 files changed, 514 insertions(+), 120 deletions(-)
diff --git a/.rat-excludes b/.rat-excludes
index 9822c9ac41..6464d827d5 100644
--- a/.rat-excludes
+++ b/.rat-excludes
@@ -132,6 +132,8 @@ scripts/*
images/*
dev/*
chart/*.iml
+out/*
+airflow-build-dockerfile*
# Sha files
.*sha256
diff --git a/3rd-party-licenses/LICENSE-reproducible.txt b/3rd-party-licenses/LICENSE-reproducible.txt
new file mode 100644
index 0000000000..65310b4f85
--- /dev/null
+++ b/3rd-party-licenses/LICENSE-reproducible.txt
@@ -0,0 +1,8 @@
+# Copyright 2013 The Servo Project Developers.
+# Copyright 2017 zerolib Developers.
+#
+# Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
+# http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+# <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
+# option. This file may not be copied, modified, or distributed
+# except according to those terms.
diff --git a/BREEZE.rst b/BREEZE.rst
index 61b42c510f..e5a12a8ac9 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -1981,6 +1981,30 @@ default is to build ``both`` type of packages ``sdist`` and ``wheel``.
:alt: Breeze release-management prepare-airflow-package
+Preparing airflow tarball
+"""""""""""""""""""""""""
+
+You can prepare airflow source tarball using Breeze:
+
+.. code-block:: bash
+
+ breeze release-management prepare-airflow-tarball
+
+This prepares airflow -source.tar.gz package in the dist folder.
+
+You must specify ``--version`` flag which is a pre-release version of Airflow you are preparing the
+tarball for.
+
+.. code-block:: bash
+
+ breeze release-management prepare-airflow-tarball --version 2.8.0rc1
+
+.. image:: ./images/breeze/output_release-management_prepare-airflow-tarball.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/images/breeze/output_release-management_prepare-airflow-tarball.svg
+ :width: 100%
+ :alt: Breeze release-management prepare-airflow-tarball
+
+
Start minor branch of Airflow
"""""""""""""""""""""""""""""
diff --git a/dev/README_RELEASE_AIRFLOW.md b/dev/README_RELEASE_AIRFLOW.md
index 50ee3b280d..22013c90db 100644
--- a/dev/README_RELEASE_AIRFLOW.md
+++ b/dev/README_RELEASE_AIRFLOW.md
@@ -585,16 +585,23 @@ Airflow supports reproducible builds, which means that the packages prepared fro
produce binary identical packages in reproducible way. You should check if the packages can be
binary-reproduced when built from the sources.
-Checkout airflow sources and build packages in dist folder:
+Checkout airflow sources and build packages in dist folder (replace X.Y.Zrc1 with the version
+you are checking):
```shell script
-git checkout X.Y.Zrc1
+VERSION=X.Y.Zrc1
+git checkout ${VERSION}
export AIRFLOW_REPO_ROOT=$(pwd)
rm -rf dist/*
-breeze release-management prepare-airflow-package --package-format both
+breeze release-management prepare-airflow-tarball --version ${VERSION}
+breeze release-management prepare-airflow-package --package-format both --use-local-hatch
```
-That should produce `.whl` and `.tar.gz` packages in dist folder.
+Note that you need to have `hatch` installed in order to build the packages with the last command.
+If you do not have `hatch`, you can remove the `--use-local-hatch` flag and it will build and use
+docker image that has `hatch` and other necessary tools installed.
+
+That should produce `-source.tar.gz` tarball of sources and `.whl`, `.tar.gz` packages in dist folder.
Change to the directory where you have the packages from svn:
diff --git a/dev/breeze/src/airflow_breeze/commands/release_candidate_command.py b/dev/breeze/src/airflow_breeze/commands/release_candidate_command.py
index 09cc67c226..11cba8d9f5 100644
--- a/dev/breeze/src/airflow_breeze/commands/release_candidate_command.py
+++ b/dev/breeze/src/airflow_breeze/commands/release_candidate_command.py
@@ -17,6 +17,7 @@
from __future__ import annotations
import os
+import shutil
import click
@@ -25,6 +26,7 @@ from airflow_breeze.commands.release_management_group import release_management
from airflow_breeze.utils.confirm import confirm_action
from airflow_breeze.utils.console import console_print
from airflow_breeze.utils.path_utils import AIRFLOW_SOURCES_ROOT
+from airflow_breeze.utils.reproducible import archive_deterministically, get_source_date_epoch
from airflow_breeze.utils.run_utils import run_command
CI = os.environ.get("CI")
@@ -59,21 +61,30 @@ def merge_pr(version_branch):
def git_tag(version):
if confirm_action(f"Tag {version}?"):
run_command(["git", "tag", "-s", f"{version}", "-m", f"Apache Airflow {version}"], check=True)
- console_print("Tagged")
+ console_print("[success]Tagged")
def git_clean():
if confirm_action("Clean git repo?"):
run_command(["breeze", "ci", "fix-ownership"], dry_run_override=DRY_RUN, check=True)
run_command(["git", "clean", "-fxd"], dry_run_override=DRY_RUN, check=True)
- console_print("Git repo cleaned")
+ console_print("[success]Git repo cleaned")
-def tarball_release(version, version_without_rc):
- if confirm_action("Create tarball?"):
- run_command(["rm", "-rf", "dist"], check=True)
+DIST_DIR = AIRFLOW_SOURCES_ROOT / "dist"
+OUT_DIR = AIRFLOW_SOURCES_ROOT / "out"
+REPRODUCIBLE_DIR = OUT_DIR / "reproducible"
+
- run_command(["mkdir", "dist"], check=True)
+def tarball_release(version: str, version_without_rc: str, source_date_epoch: int):
+ if confirm_action("Create tarball?"):
+ console_print(f"[info]Creating tarball for Airflow {version}")
+ shutil.rmtree(OUT_DIR, ignore_errors=True)
+ DIST_DIR.mkdir(exist_ok=True)
+ OUT_DIR.mkdir(exist_ok=True)
+ REPRODUCIBLE_DIR.mkdir(exist_ok=True)
+ archive_name = f"apache-airflow-{version_without_rc}-source.tar.gz"
+ temporary_archive = OUT_DIR / archive_name
run_command(
[
"git",
@@ -82,19 +93,48 @@ def tarball_release(version, version_without_rc):
f"{version}",
f"--prefix=apache-airflow-{version_without_rc}/",
"-o",
- f"dist/apache-airflow-{version_without_rc}-source.tar.gz",
+ temporary_archive.as_posix(),
],
check=True,
)
- console_print("Tarball created")
+ run_command(
+ [
+ "tar",
+ "-xf",
+ temporary_archive.as_posix(),
+ "-C",
+ REPRODUCIBLE_DIR.as_posix(),
+ "--strip",
+ "1",
+ ]
+ )
+ final_archive = DIST_DIR / archive_name
+ archive_deterministically(
+ dir_to_archive=REPRODUCIBLE_DIR.as_posix(),
+ dest_archive=final_archive.as_posix(),
+ prepend_path=None,
+ timestamp=source_date_epoch,
+ )
+ console_print(f"[success]Tarball created in {final_archive}")
-def create_artifacts_with_sdist():
- run_command(["hatch", "build", "-t", "sdist", "-t", "wheel"], check=True)
- console_print("Artifacts created")
+def create_artifacts_with_hatch(source_date_epoch: int):
+ console_print("[info]Creating artifacts with hatch")
+ shutil.rmtree(DIST_DIR, ignore_errors=True)
+ DIST_DIR.mkdir(exist_ok=True)
+ env_copy = os.environ.copy()
+ env_copy["SOURCE_DATE_EPOCH"] = str(source_date_epoch)
+ run_command(
+ ["hatch", "build", "-c", "-t", "custom", "-t", "sdist", "-t", "wheel"], check=True, env=env_copy
+ )
+ console_print("[success]Successfully prepared Airflow packages:")
+ for file in sorted(DIST_DIR.glob("apache_airflow*")):
+ console_print(print(file.name))
+ console_print()
-def create_artifacts_with_breeze():
+def create_artifacts_with_docker():
+ console_print("[info]Creating artifacts with docker")
run_command(
[
"breeze",
@@ -105,14 +145,14 @@ def create_artifacts_with_breeze():
],
check=True,
)
- console_print("Artifacts created")
+ console_print("[success]Artifacts created")
def sign_the_release(repo_root):
if confirm_action("Do you want to sign the release?"):
os.chdir(f"{repo_root}/dist")
run_command("./../dev/sign.sh *", dry_run_override=DRY_RUN, check=True, shell=True)
- console_print("Release signed")
+ console_print("[success]Release signed")
def tag_and_push_constraints(version, version_branch):
@@ -135,7 +175,7 @@ def tag_and_push_constraints(version, version_branch):
run_command(
["git", "push", "origin", "tag", f"constraints-{version}"], dry_run_override=DRY_RUN, check=True
)
- console_print("Constraints tagged and pushed")
+ console_print("[success]Constraints tagged and pushed")
def clone_asf_repo(version, repo_root):
@@ -146,7 +186,7 @@ def clone_asf_repo(version, repo_root):
check=True,
)
run_command(["svn", "update", "--set-depth=infinity", "asf-dist/dev/airflow"], check=True)
- console_print("Cloned ASF repo successfully")
+ console_print("[success]Cloned ASF repo successfully")
def move_artifacts_to_svn(version, repo_root):
@@ -154,7 +194,7 @@ def move_artifacts_to_svn(version, repo_root):
os.chdir(f"{repo_root}/asf-dist/dev/airflow")
run_command(["svn", "mkdir", f"{version}"], dry_run_override=DRY_RUN, check=True)
run_command(f"mv {repo_root}/dist/* {version}/", dry_run_override=DRY_RUN, check=True, shell=True)
- console_print("Moved artifacts to SVN:")
+ console_print("[success]Moved artifacts to SVN:")
run_command(["ls"], dry_run_override=DRY_RUN)
@@ -171,7 +211,7 @@ def push_artifacts_to_asf_repo(version, repo_root):
dry_run_override=DRY_RUN,
check=True,
)
- console_print("Files pushed to svn")
+ console_print("[success]Files pushed to svn")
def delete_asf_repo(repo_root):
@@ -182,7 +222,7 @@ def delete_asf_repo(repo_root):
def prepare_pypi_packages(version, version_suffix, repo_root):
if confirm_action("Prepare pypi packages?"):
- console_print("Preparing PyPI packages")
+ console_print("[info]Preparing PyPI packages")
os.chdir(repo_root)
run_command(["git", "checkout", f"{version}"], dry_run_override=DRY_RUN, check=True)
run_command(
@@ -198,13 +238,13 @@ def prepare_pypi_packages(version, version_suffix, repo_root):
check=True,
)
run_command(["twine", "check", "dist/*"], check=True)
- console_print("PyPI packages prepared")
+ console_print("[success]PyPI packages prepared")
def push_packages_to_pypi(version):
if confirm_action("Do you want to push packages to production PyPI?"):
run_command(["twine", "upload", "-r", "pypi", "dist/*"], dry_run_override=DRY_RUN, check=True)
- console_print("Packages pushed to production PyPI")
+ console_print("[success]Packages pushed to production PyPI")
console_print(
"Again, confirm that the package is available here: https://pypi.python.org/pypi/apache-airflow"
)
@@ -240,7 +280,7 @@ def push_release_candidate_tag_to_github(version):
)
confirm_action(f"Confirm that {version} is pushed to PyPI(not PyPI test). Is it pushed?", abort=True)
run_command(["git", "push", "origin", "tag", f"{version}"], dry_run_override=DRY_RUN, check=True)
- console_print("Release candidate tag pushed to GitHub")
+ console_print("[success]Release candidate tag pushed to GitHub")
def create_issue_for_testing(version, previous_version, github_token):
@@ -293,10 +333,31 @@ def remove_old_releases(version, repo_root):
dry_run_override=DRY_RUN,
check=True,
)
-
+ console_print("[success]Old releases removed")
os.chdir(repo_root)
+@release_management.command(
+ name="prepare-airflow-tarball",
+ help="Prepare airflow's source tarball.",
+)
+@click.option(
+ "--version", required=True, help="The release candidate version e.g. 2.4.3rc1", envvar="VERSION"
+)
+def prepare_airflow_tarball(version: str):
+ from packaging.version import Version
+
+ airflow_version = Version(version)
+ if not airflow_version.is_prerelease:
+ exit("--version value must be a pre-release")
+ source_date_epoch = get_source_date_epoch()
+ version_without_rc = airflow_version.base_version
+ # Create the tarball
+ tarball_release(
+ version=version, version_without_rc=version_without_rc, source_date_epoch=source_date_epoch
+ )
+
+
@release_management.command(
name="start-rc-process",
short_help="Start RC process",
@@ -311,7 +372,8 @@ def remove_old_releases(version, repo_root):
def publish_release_candidate(version, previous_version, github_token):
from packaging.version import Version
- if not Version(version).is_prerelease:
+ airflow_version = Version(version)
+ if not airflow_version.is_prerelease:
exit("--version value must be a pre-release")
if Version(previous_version).is_prerelease:
exit("--previous-version value must be a release not a pre-release")
@@ -320,9 +382,10 @@ def publish_release_candidate(version, previous_version, github_token):
if not github_token:
console_print("GITHUB_TOKEN is not set! Issue generation will fail.")
confirm_action("Do you want to continue?", abort=True)
- version_suffix = version[5:]
- version_branch = version[:3].replace(".", "-")
- version_without_rc = version[:5]
+
+ version_suffix = airflow_version.pre[0] + str(airflow_version.pre[1])
+ version_branch = str(airflow_version.release[0]) + "-" + str(airflow_version.release[1])
+ version_without_rc = airflow_version.base_version
os.chdir(AIRFLOW_SOURCES_ROOT)
airflow_repo_root = os.getcwd()
@@ -343,20 +406,21 @@ def publish_release_candidate(version, previous_version, github_token):
confirm_action("Pushes will be made to origin. Do you want to continue?", abort=True)
# Merge the sync PR
merge_pr(version_branch)
-
- # Tag & clean the repo
+ #
+ # # Tag & clean the repo
git_tag(version)
git_clean()
- # Build the latest image
- if confirm_action("Build latest breeze image?"):
- run_command(["breeze", "ci-image", "build", "--python", "3.8"], dry_run_override=DRY_RUN, check=True)
+ source_date_epoch = get_source_date_epoch()
+ shutil.rmtree(DIST_DIR, ignore_errors=True)
# Create the tarball
- tarball_release(version, version_without_rc)
+ tarball_release(
+ version=version, version_without_rc=version_without_rc, source_date_epoch=source_date_epoch
+ )
# Create the artifacts
- if confirm_action("Use breeze to create artifacts?"):
- create_artifacts_with_breeze()
+ if confirm_action("Use docker to create artifacts?"):
+ create_artifacts_with_docker()
elif confirm_action("Use hatch to create artifacts?"):
- create_artifacts_with_sdist()
+ create_artifacts_with_hatch()
# Sign the release
sign_the_release(airflow_repo_root)
# Tag and push constraints
diff --git a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
index ca2041bd94..c19e9890bc 100644
--- a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
@@ -142,6 +142,7 @@ from airflow_breeze.utils.provider_dependencies import (
get_related_providers,
)
from airflow_breeze.utils.python_versions import get_python_version_list
+from airflow_breeze.utils.reproducible import get_source_date_epoch
from airflow_breeze.utils.run_utils import (
run_command,
)
@@ -282,6 +283,11 @@ AIRFLOW_BUILD_DOCKERFILE_IGNORE_PATH = AIRFLOW_SOURCES_ROOT / "airflow-build-doc
name="prepare-airflow-package",
help="Prepare sdist/whl package of Airflow.",
)
+@click.option(
+ "--use-local-hatch",
+ is_flag=True,
+ help="Use local hatch instead of docker to build the package. You need to have hatch installed.",
+)
@option_package_format
@option_version_suffix_for_pypi
@option_verbose
@@ -289,10 +295,30 @@ AIRFLOW_BUILD_DOCKERFILE_IGNORE_PATH = AIRFLOW_SOURCES_ROOT / "airflow-build-doc
def prepare_airflow_packages(
package_format: str,
version_suffix_for_pypi: str,
+ use_local_hatch: bool,
):
perform_environment_checks()
fix_ownership_using_docker()
cleanup_python_generated_files()
+ source_date_epoch = get_source_date_epoch()
+ if use_local_hatch:
+ hatch_build_command = ["hatch", "build", "-c", "-t", "custom"]
+ if package_format in ["sdist", "both"]:
+ hatch_build_command.extend(["-t", "sdist"])
+ if package_format in ["wheel", "both"]:
+ hatch_build_command.extend(["-t", "wheel"])
+ env_copy = os.environ.copy()
+ env_copy["SOURCE_DATE_EPOCH"] = str(source_date_epoch)
+ run_command(
+ hatch_build_command,
+ check=True,
+ env=env_copy,
+ )
+ get_console().print("[success]Successfully prepared Airflow packages:")
+ for file in sorted(DIST_DIR.glob("apache_airflow*")):
+ get_console().print(file.name)
+ get_console().print()
+ return
# This is security feature.
#
# Building the image needed to build airflow package including .git directory
@@ -350,7 +376,10 @@ def prepare_airflow_packages(
# Copy all files in the dist directory in container to the host dist directory (note '/.' in SRC)
run_command(["docker", "cp", f"{container_id}:/opt/airflow/dist/.", "./dist"], check=True)
run_command(["docker", "rm", "--force", container_id], check=True)
- get_console().print("[success]Successfully prepared Airflow package!\n\n")
+ get_console().print("[success]Successfully prepared Airflow packages:")
+ for file in sorted(DIST_DIR.glob("apache_airflow*")):
+ get_console().print(file.name)
+ get_console().print()
def provider_action_summary(description: str, message_type: MessageType, packages: list[str]):
diff --git a/dev/breeze/src/airflow_breeze/commands/release_management_commands_config.py b/dev/breeze/src/airflow_breeze/commands/release_management_commands_config.py
index e6ddec7e16..b0b1bb96e0 100644
--- a/dev/breeze/src/airflow_breeze/commands/release_management_commands_config.py
+++ b/dev/breeze/src/airflow_breeze/commands/release_management_commands_config.py
@@ -19,8 +19,9 @@ from __future__ import annotations
RELEASE_AIRFLOW_COMMANDS: dict[str, str | list[str]] = {
"name": "Airflow release commands",
"commands": [
- "prepare-airflow-package",
"create-minor-branch",
+ "prepare-airflow-package",
+ "prepare-airflow-tarball",
"start-rc-process",
"start-release",
"release-prod-images",
@@ -43,9 +44,10 @@ RELEASE_PROVIDERS_COMMANDS: dict[str, str | list[str]] = {
RELEASE_OTHER_COMMANDS: dict[str, str | list[str]] = {
"name": "Other release commands",
"commands": [
+ "add-back-references",
"publish-docs",
"generate-constraints",
- "add-back-references",
+ "update-constraints",
],
}
@@ -55,10 +57,19 @@ RELEASE_MANAGEMENT_PARAMETERS: dict[str, list[dict[str, str | list[str]]]] = {
"name": "Package flags",
"options": [
"--package-format",
+ "--use-local-hatch",
"--version-suffix-for-pypi",
],
}
],
+ "breeze release-management prepare-airflow-tarball": [
+ {
+ "name": "Package flags",
+ "options": [
+ "--version",
+ ],
+ }
+ ],
"breeze release-management verify-provider-packages": [
{
"name": "Provider verification flags",
diff --git a/dev/breeze/src/airflow_breeze/utils/reproducible.py b/dev/breeze/src/airflow_breeze/utils/reproducible.py
new file mode 100644
index 0000000000..a85d871a3c
--- /dev/null
+++ b/dev/breeze/src/airflow_breeze/utils/reproducible.py
@@ -0,0 +1,135 @@
+#!/usr/bin/env python3
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Copyright 2013 The Servo Project Developers.
+# Copyright 2017 zerolib Developers.
+#
+# Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
+# http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+# <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
+# option. This file may not be copied, modified, or distributed
+# except according to those terms.
+
+# This command is a largely vendored-in script from
+# https://github.com/MuxZeroNet/reproducible/blob/master/reproducible.py
+from __future__ import annotations
+
+import contextlib
+import gzip
+import itertools
+import locale
+import os
+import tarfile
+from argparse import ArgumentParser
+
+from airflow_breeze.utils.path_utils import AIRFLOW_SOURCES_ROOT
+
+
+def get_source_date_epoch():
+ import yaml
+
+ reproducible_build_yaml = AIRFLOW_SOURCES_ROOT / "airflow" / "reproducible_build.yaml"
+ reproducible_build_dict = yaml.safe_load(reproducible_build_yaml.read_text())
+ source_date_epoch: int = reproducible_build_dict["source-date-epoch"]
+ return source_date_epoch
+
+
+@contextlib.contextmanager
+def cd(new_path):
+ """Context manager for changing the current working directory"""
+ previous_path = os.getcwd()
+ try:
+ os.chdir(new_path)
+ yield
+ finally:
+ os.chdir(previous_path)
+
+
+@contextlib.contextmanager
+def setlocale(name):
+ """Context manager for changing the current locale"""
+ saved_locale = locale.setlocale(locale.LC_ALL)
+ try:
+ yield locale.setlocale(locale.LC_ALL, name)
+ finally:
+ locale.setlocale(locale.LC_ALL, saved_locale)
+
+
+def archive_deterministically(dir_to_archive, dest_archive, prepend_path=None, timestamp=0):
+ """Create a .tar.gz archive in a deterministic (reproducible) manner.
+
+ See https://reproducible-builds.org/docs/archives/ for more details."""
+
+ def reset(tarinfo):
+ """Helper to reset owner/group and modification time for tar entries"""
+ tarinfo.uid = tarinfo.gid = 0
+ tarinfo.uname = tarinfo.gname = "root"
+ tarinfo.mtime = timestamp
+ return tarinfo
+
+ dest_archive = os.path.abspath(dest_archive)
+ with cd(dir_to_archive):
+ current_dir = "."
+ file_list = [current_dir]
+ for root, dirs, files in os.walk(current_dir):
+ for name in itertools.chain(dirs, files):
+ file_list.append(os.path.join(root, name))
+
+ # Sort file entries with the fixed locale
+ with setlocale("C"):
+ file_list.sort(key=locale.strxfrm)
+
+ # Use a temporary file and atomic rename to avoid partially-formed
+ # packaging (in case of exceptional situations like running out of disk space).
+ temp_file = f"{dest_archive}.temp~"
+ with os.fdopen(os.open(temp_file, os.O_WRONLY | os.O_CREAT, 0o644), "wb") as out_file:
+ with gzip.GzipFile("wb", fileobj=out_file, mtime=0) as gzip_file:
+ with tarfile.open(fileobj=gzip_file, mode="w:") as tar_file:
+ for entry in file_list:
+ arcname = entry
+ if prepend_path is not None:
+ arcname = os.path.normpath(os.path.join(prepend_path, arcname))
+ tar_file.add(entry, filter=reset, recursive=False, arcname=arcname)
+ os.rename(temp_file, dest_archive)
+
+
+def main():
+ parser = ArgumentParser()
+ parser.add_argument("-d", "--dir", help="directory to archive")
+ parser.add_argument("-o", "--out", help="archive destination")
+ parser.add_argument("-p", "--prepend", help="prepend path")
+ parser.add_argument(
+ "-t", "--timestamp", help="timestamp of files", type=int, default=get_source_date_epoch()
+ )
+
+ args = parser.parse_args()
+
+ if not args.dir or not args.out:
+ error = (
+ "You should provide a directory to archive, and the "
+ f"archive file name, not {repr((args.dir, args.out))}"
+ )
+ raise ValueError(error)
+
+ archive_deterministically(args.dir, args.out, args.prepend, args.timestamp)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/images/breeze/output_release-management.svg b/images/breeze/output_release-management.svg
index f853a75b69..745601db24 100644
--- a/images/breeze/output_release-management.svg
+++ b/images/breeze/output_release-management.svg
@@ -1,4 +1,4 @@
-<svg class="rich-terminal" viewBox="0 0 1482 830.8" xmlns="http://www.w3.org/2000/svg">
+<svg class="rich-terminal" viewBox="0 0 1482 806.4" xmlns="http://www.w3.org/2000/svg">
<!-- Generated with Rich https://www.textualize.io -->
<style>
@@ -42,7 +42,7 @@
<defs>
<clipPath id="breeze-release-management-clip-terminal">
- <rect x="0" y="0" width="1463.0" height="779.8" />
+ <rect x="0" y="0" width="1463.0" height="755.4" />
</clipPath>
<clipPath id="breeze-release-management-line-0">
<rect x="0" y="1.5" width="1464" height="24.65"/>
@@ -134,12 +134,9 @@
<clipPath id="breeze-release-management-line-29">
<rect x="0" y="709.1" width="1464" height="24.65"/>
</clipPath>
-<clipPath id="breeze-release-management-line-30">
- <rect x="0" y="733.5" width="1464" height="24.65"/>
- </clipPath>
</defs>
- <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="828.8" rx="8"/><text class="breeze-release-management-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: release-management</text>
+ <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="804.4" rx="8"/><text class="breeze-release-management-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: release-management</text>
<g transform="translate(26,22)">
<circle cx="0" cy="0" r="7" fill="#ff5f57"/>
<circle cx="22" cy="0" r="7" fill="#febc2e"/>
@@ -158,29 +155,28 @@
</text><text class="breeze-release-management-r5" x="0" y="166.4" textLength="12.2" clip-path="url(#breeze-release-management-line-6)">│</text><text class="breeze-release-management-r4" x="24.4" y="166.4" textLength="12.2" clip-path="url(#breeze-release-management-line-6)">-</text><text class="breeze-release-management-r4" x="36.6" y="166.4" textLength="61" clip-path="url(#breeze-release-management-line-6)">-help</text><text class="breeze-release-management-r6" x="122" y="166.4" textLeng [...]
</text><text class="breeze-release-management-r5" x="0" y="190.8" textLength="1464" clip-path="url(#breeze-release-management-line-7)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-release-management-r1" x="1464" y="190.8" textLength="12.2" clip-path="url(#breeze-release-management-line-7)">
</text><text class="breeze-release-management-r5" x="0" y="215.2" textLength="24.4" clip-path="url(#breeze-release-management-line-8)">╭─</text><text class="breeze-release-management-r5" x="24.4" y="215.2" textLength="317.2" clip-path="url(#breeze-release-management-line-8)"> Airflow release commands </text><text class="breeze-release-management-r5" x="341.6" y="215.2" textLength="1098" clip-path="url(#breeze-release-management-line-8)">─────────────────────────────── [...]
-</text><text class="breeze-release-management-r5" x="0" y="239.6" textLength="12.2" clip-path="url(#breeze-release-management-line-9)">│</text><text class="breeze-release-management-r4" x="24.4" y="239.6" textLength="353.8" clip-path="url(#breeze-release-management-line-9)">prepare-airflow-package      </text><text class="breeze-release-management-r1" x="402.6" y="239.6" textLength="1037" clip-path="url(#breeze-release-management-line-9)">Prepare sdist/ [...]
-</text><text class="breeze-release-management-r5" x="0" y="264" textLength="12.2" clip-path="url(#breeze-release-management-line-10)">│</text><text class="breeze-release-management-r4" x="24.4" y="264" textLength="353.8" clip-path="url(#breeze-release-management-line-10)">create-minor-branch          </text><text class="breeze-release-management-r1" x="402.6" y="264" textLength="1037" clip-path="url(#breeze-release-management-line-10)">Cr [...]
-</text><text class="breeze-release-management-r5" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-release-management-line-11)">│</text><text class="breeze-release-management-r4" x="24.4" y="288.4" textLength="353.8" clip-path="url(#breeze-release-management-line-11)">start-rc-process             </text><text class="breeze-release-management-r1" x="402.6" y="288.4" textLength="1037" clip-path="url(#breeze-release-ma [...]
-</text><text class="breeze-release-management-r5" x="0" y="312.8" textLength="12.2" clip-path="url(#breeze-release-management-line-12)">│</text><text class="breeze-release-management-r4" x="24.4" y="312.8" textLength="353.8" clip-path="url(#breeze-release-management-line-12)">start-release                </text><text class="breeze-release-management-r1" x="402.6" y="312.8" textLength="1037" clip-path="url(#br [...]
-</text><text class="breeze-release-management-r5" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-release-management-line-13)">│</text><text class="breeze-release-management-r4" x="24.4" y="337.2" textLength="353.8" clip-path="url(#breeze-release-management-line-13)">release-prod-images          </text><text class="breeze-release-management-r1" x="402.6" y="337.2" textLength="1037" clip-path="url(#breeze-release-management-line-1 [...]
-</text><text class="breeze-release-management-r5" x="0" y="361.6" textLength="1464" clip-path="url(#breeze-release-management-line-14)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-release-management-r1" x="1464" y="361.6" textLength="12.2" clip-path="url(#breeze-release-management-line-14)">
-</text><text class="breeze-release-management-r5" x="0" y="386" textLength="24.4" clip-path="url(#breeze-release-management-line-15)">╭─</text><text class="breeze-release-management-r5" x="24.4" y="386" textLength="341.6" clip-path="url(#breeze-release-management-line-15)"> Providers release commands </text><text class="breeze-release-management-r5" x="366" y="386" textLength="1073.6" clip-path="url(#breeze-release-management-line-15)">──────────────────────────────── [...]
-</text><text class="breeze-release-management-r5" x="0" y="410.4" textLength="12.2" clip-path="url(#breeze-release-management-line-16)">│</text><text class="breeze-release-management-r4" x="24.4" y="410.4" textLength="439.2" clip-path="url(#breeze-release-management-line-16)">prepare-provider-documentation      </text><text class="breeze-release-management-r1" x="488" y="410.4" textLength="951.6" clip-path="url(#breeze-release-management-line-16)">Prepare [...]
-</text><text class="breeze-release-management-r5" x="0" y="434.8" textLength="12.2" clip-path="url(#breeze-release-management-line-17)">│</text><text class="breeze-release-management-r4" x="24.4" y="434.8" textLength="439.2" clip-path="url(#breeze-release-management-line-17)">prepare-provider-packages           </text><text class="breeze-release-management-r1" x="488" y="434.8" textLength="951.6" clip-path="url(#breeze-release-manage [...]
-</text><text class="breeze-release-management-r5" x="0" y="459.2" textLength="12.2" clip-path="url(#breeze-release-management-line-18)">│</text><text class="breeze-release-management-r4" x="24.4" y="459.2" textLength="439.2" clip-path="url(#breeze-release-management-line-18)">install-provider-packages           </text><text class="breeze-release-management-r1" x="488" y="459.2" textLength="951.6" clip-path="url(#breeze-release-manage [...]
-</text><text class="breeze-release-management-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-release-management-line-19)">│</text><text class="breeze-release-management-r4" x="24.4" y="483.6" textLength="439.2" clip-path="url(#breeze-release-management-line-19)">verify-provider-packages            </text><text class="breeze-release-management-r1" x="488" y="483.6" textLength="951.6" clip-path="url(#breeze-release-m [...]
-</text><text class="breeze-release-management-r5" x="0" y="508" textLength="12.2" clip-path="url(#breeze-release-management-line-20)">│</text><text class="breeze-release-management-r4" x="24.4" y="508" textLength="439.2" clip-path="url(#breeze-release-management-line-20)">generate-providers-metadata         </text><text class="breeze-release-management-r1" x="488" y="508" textLength="951.6" clip-path="url(#breeze-release-management-line-20)">G [...]
-</text><text class="breeze-release-management-r5" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-release-management-line-21)">│</text><text class="breeze-release-management-r4" x="24.4" y="532.4" textLength="439.2" clip-path="url(#breeze-release-management-line-21)">generate-issue-content-providers    </text><text class="breeze-release-management-r1" x="488" y="532.4" textLength="951.6" clip-path="url(#breeze-release-management-line-21)">Generates conte [...]
-</text><text class="breeze-release-management-r5" x="0" y="556.8" textLength="12.2" clip-path="url(#breeze-release-management-line-22)">│</text><text class="breeze-release-management-r4" x="24.4" y="556.8" textLength="439.2" clip-path="url(#breeze-release-management-line-22)">clean-old-provider-artifacts        </text><text class="breeze-release-management-r1" x="488" y="556.8" textLength="951.6" clip-path="url(#breeze-release-management-line-22)"> [...]
-</text><text class="breeze-release-management-r5" x="0" y="581.2" textLength="1464" clip-path="url(#breeze-release-management-line-23)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-release-management-r1" x="1464" y="581.2" textLength="12.2" clip-path="url(#breeze-release-management-line-23)">
-</text><text class="breeze-release-management-r5" x="0" y="605.6" textLength="24.4" clip-path="url(#breeze-release-management-line-24)">╭─</text><text class="breeze-release-management-r5" x="24.4" y="605.6" textLength="292.8" clip-path="url(#breeze-release-management-line-24)"> Other release commands </text><text class="breeze-release-management-r5" x="317.2" y="605.6" textLength="1122.4" clip-path="url(#breeze-release-management-line-24)">──────────────────────────── [...]
-</text><text class="breeze-release-management-r5" x="0" y="630" textLength="12.2" clip-path="url(#breeze-release-management-line-25)">│</text><text class="breeze-release-management-r4" x="24.4" y="630" textLength="280.6" clip-path="url(#breeze-release-management-line-25)">publish-docs           </text><text class="breeze-release-management-r1" x="329.4" y="630" textLength="1110.2" clip-path="url(#breeze-release-management-line-25)">C [...]
-</text><text class="breeze-release-management-r5" x="0" y="654.4" textLength="12.2" clip-path="url(#breeze-release-management-line-26)">│</text><text class="breeze-release-management-r4" x="24.4" y="654.4" textLength="280.6" clip-path="url(#breeze-release-management-line-26)">generate-constraints   </text><text class="breeze-release-management-r1" x="329.4" y="654.4" textLength="1110.2" clip-path="url(#breeze-release-management-line-26)">Generates pinned constrai [...]
-</text><text class="breeze-release-management-r5" x="0" y="678.8" textLength="12.2" clip-path="url(#breeze-release-management-line-27)">│</text><text class="breeze-release-management-r4" x="24.4" y="678.8" textLength="280.6" clip-path="url(#breeze-release-management-line-27)">add-back-references    </text><text class="breeze-release-management-r1" x="329.4" y="678.8" textLength="1110.2" clip-path="url(#breeze-release-management-line-27)">Command to add  [...]
-</text><text class="breeze-release-management-r5" x="0" y="703.2" textLength="1464" clip-path="url(#breeze-release-management-line-28)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-release-management-r1" x="1464" y="703.2" textLength="12.2" clip-path="url(#breeze-release-management-line-28)">
-</text><text class="breeze-release-management-r5" x="0" y="727.6" textLength="24.4" clip-path="url(#breeze-release-management-line-29)">╭─</text><text class="breeze-release-management-r5" x="24.4" y="727.6" textLength="122" clip-path="url(#breeze-release-management-line-29)"> Commands </text><text class="breeze-release-management-r5" x="146.4" y="727.6" textLength="1293.2" clip-path="url(#breeze-release-management-line-29)">────────────────────────────────────────────────────── [...]
-</text><text class="breeze-release-management-r5" x="0" y="752" textLength="12.2" clip-path="url(#breeze-release-management-line-30)">│</text><text class="breeze-release-management-r4" x="24.4" y="752" textLength="390.4" clip-path="url(#breeze-release-management-line-30)">update-constraints              </text><text class="breeze-release-management-r1" x="439.2" y="752" textLength="1000.4" clip-path="url(#breeze-releas [...]
-</text><text class="breeze-release-management-r5" x="0" y="776.4" textLength="1464" clip-path="url(#breeze-release-management-line-31)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-release-management-r1" x="1464" y="776.4" textLength="12.2" clip-path="url(#breeze-release-management-line-31)">
+</text><text class="breeze-release-management-r5" x="0" y="239.6" textLength="12.2" clip-path="url(#breeze-release-management-line-9)">│</text><text class="breeze-release-management-r4" x="24.4" y="239.6" textLength="353.8" clip-path="url(#breeze-release-management-line-9)">create-minor-branch          </text><text class="breeze-release-management-r1" x="402.6" y="239.6" textLength="1037" clip-path="url(#breeze-release-management-line-9)" [...]
+</text><text class="breeze-release-management-r5" x="0" y="264" textLength="12.2" clip-path="url(#breeze-release-management-line-10)">│</text><text class="breeze-release-management-r4" x="24.4" y="264" textLength="353.8" clip-path="url(#breeze-release-management-line-10)">prepare-airflow-package      </text><text class="breeze-release-management-r1" x="402.6" y="264" textLength="1037" clip-path="url(#breeze-release-management-line-10)">Prepare sdist/whl [...]
+</text><text class="breeze-release-management-r5" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-release-management-line-11)">│</text><text class="breeze-release-management-r4" x="24.4" y="288.4" textLength="353.8" clip-path="url(#breeze-release-management-line-11)">prepare-airflow-tarball      </text><text class="breeze-release-management-r1" x="402.6" y="288.4" textLength="1037" clip-path="url(#breeze-release-management-line-11)">Prepare air [...]
+</text><text class="breeze-release-management-r5" x="0" y="312.8" textLength="12.2" clip-path="url(#breeze-release-management-line-12)">│</text><text class="breeze-release-management-r4" x="24.4" y="312.8" textLength="353.8" clip-path="url(#breeze-release-management-line-12)">start-rc-process             </text><text class="breeze-release-management-r1" x="402.6" y="312.8" textLength="1037" clip-path="url(#breeze-release-ma [...]
+</text><text class="breeze-release-management-r5" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-release-management-line-13)">│</text><text class="breeze-release-management-r4" x="24.4" y="337.2" textLength="353.8" clip-path="url(#breeze-release-management-line-13)">start-release                </text><text class="breeze-release-management-r1" x="402.6" y="337.2" textLength="1037" clip-path="url(#br [...]
+</text><text class="breeze-release-management-r5" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-release-management-line-14)">│</text><text class="breeze-release-management-r4" x="24.4" y="361.6" textLength="353.8" clip-path="url(#breeze-release-management-line-14)">release-prod-images          </text><text class="breeze-release-management-r1" x="402.6" y="361.6" textLength="1037" clip-path="url(#breeze-release-management-line-1 [...]
+</text><text class="breeze-release-management-r5" x="0" y="386" textLength="1464" clip-path="url(#breeze-release-management-line-15)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-release-management-r1" x="1464" y="386" textLength="12.2" clip-path="url(#breeze-release-management-line-15)">
+</text><text class="breeze-release-management-r5" x="0" y="410.4" textLength="24.4" clip-path="url(#breeze-release-management-line-16)">╭─</text><text class="breeze-release-management-r5" x="24.4" y="410.4" textLength="341.6" clip-path="url(#breeze-release-management-line-16)"> Providers release commands </text><text class="breeze-release-management-r5" x="366" y="410.4" textLength="1073.6" clip-path="url(#breeze-release-management-line-16)">────────────────────────── [...]
+</text><text class="breeze-release-management-r5" x="0" y="434.8" textLength="12.2" clip-path="url(#breeze-release-management-line-17)">│</text><text class="breeze-release-management-r4" x="24.4" y="434.8" textLength="439.2" clip-path="url(#breeze-release-management-line-17)">prepare-provider-documentation      </text><text class="breeze-release-management-r1" x="488" y="434.8" textLength="951.6" clip-path="url(#breeze-release-management-line-17)">Prepare [...]
+</text><text class="breeze-release-management-r5" x="0" y="459.2" textLength="12.2" clip-path="url(#breeze-release-management-line-18)">│</text><text class="breeze-release-management-r4" x="24.4" y="459.2" textLength="439.2" clip-path="url(#breeze-release-management-line-18)">prepare-provider-packages           </text><text class="breeze-release-management-r1" x="488" y="459.2" textLength="951.6" clip-path="url(#breeze-release-manage [...]
+</text><text class="breeze-release-management-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-release-management-line-19)">│</text><text class="breeze-release-management-r4" x="24.4" y="483.6" textLength="439.2" clip-path="url(#breeze-release-management-line-19)">install-provider-packages           </text><text class="breeze-release-management-r1" x="488" y="483.6" textLength="951.6" clip-path="url(#breeze-release-manage [...]
+</text><text class="breeze-release-management-r5" x="0" y="508" textLength="12.2" clip-path="url(#breeze-release-management-line-20)">│</text><text class="breeze-release-management-r4" x="24.4" y="508" textLength="439.2" clip-path="url(#breeze-release-management-line-20)">verify-provider-packages            </text><text class="breeze-release-management-r1" x="488" y="508" textLength="951.6" clip-path="url(#breeze-release-managem [...]
+</text><text class="breeze-release-management-r5" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-release-management-line-21)">│</text><text class="breeze-release-management-r4" x="24.4" y="532.4" textLength="439.2" clip-path="url(#breeze-release-management-line-21)">generate-providers-metadata         </text><text class="breeze-release-management-r1" x="488" y="532.4" textLength="951.6" clip-path="url(#breeze-release-management-line- [...]
+</text><text class="breeze-release-management-r5" x="0" y="556.8" textLength="12.2" clip-path="url(#breeze-release-management-line-22)">│</text><text class="breeze-release-management-r4" x="24.4" y="556.8" textLength="439.2" clip-path="url(#breeze-release-management-line-22)">generate-issue-content-providers    </text><text class="breeze-release-management-r1" x="488" y="556.8" textLength="951.6" clip-path="url(#breeze-release-management-line-22)">Generates conte [...]
+</text><text class="breeze-release-management-r5" x="0" y="581.2" textLength="12.2" clip-path="url(#breeze-release-management-line-23)">│</text><text class="breeze-release-management-r4" x="24.4" y="581.2" textLength="439.2" clip-path="url(#breeze-release-management-line-23)">clean-old-provider-artifacts        </text><text class="breeze-release-management-r1" x="488" y="581.2" textLength="951.6" clip-path="url(#breeze-release-management-line-23)"> [...]
+</text><text class="breeze-release-management-r5" x="0" y="605.6" textLength="1464" clip-path="url(#breeze-release-management-line-24)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-release-management-r1" x="1464" y="605.6" textLength="12.2" clip-path="url(#breeze-release-management-line-24)">
+</text><text class="breeze-release-management-r5" x="0" y="630" textLength="24.4" clip-path="url(#breeze-release-management-line-25)">╭─</text><text class="breeze-release-management-r5" x="24.4" y="630" textLength="292.8" clip-path="url(#breeze-release-management-line-25)"> Other release commands </text><text class="breeze-release-management-r5" x="317.2" y="630" textLength="1122.4" clip-path="url(#breeze-release-management-line-25)">────────────────────────────────── [...]
+</text><text class="breeze-release-management-r5" x="0" y="654.4" textLength="12.2" clip-path="url(#breeze-release-management-line-26)">│</text><text class="breeze-release-management-r4" x="24.4" y="654.4" textLength="280.6" clip-path="url(#breeze-release-management-line-26)">add-back-references    </text><text class="breeze-release-management-r1" x="329.4" y="654.4" textLength="1110.2" clip-path="url(#breeze-release-management-line-26)">Command to add  [...]
+</text><text class="breeze-release-management-r5" x="0" y="678.8" textLength="12.2" clip-path="url(#breeze-release-management-line-27)">│</text><text class="breeze-release-management-r4" x="24.4" y="678.8" textLength="280.6" clip-path="url(#breeze-release-management-line-27)">publish-docs           </text><text class="breeze-release-management-r1" x="329.4" y="678.8" textLength="1110.2" clip-path="url(#breeze-release-management-line- [...]
+</text><text class="breeze-release-management-r5" x="0" y="703.2" textLength="12.2" clip-path="url(#breeze-release-management-line-28)">│</text><text class="breeze-release-management-r4" x="24.4" y="703.2" textLength="280.6" clip-path="url(#breeze-release-management-line-28)">generate-constraints   </text><text class="breeze-release-management-r1" x="329.4" y="703.2" textLength="1110.2" clip-path="url(#breeze-release-management-line-28)">Generates pinned constrai [...]
+</text><text class="breeze-release-management-r5" x="0" y="727.6" textLength="12.2" clip-path="url(#breeze-release-management-line-29)">│</text><text class="breeze-release-management-r4" x="24.4" y="727.6" textLength="280.6" clip-path="url(#breeze-release-management-line-29)">update-constraints     </text><text class="breeze-release-management-r1" x="329.4" y="727.6" textLength="1110.2" clip-path="url(#breeze-release-management-line-29)">Update released  [...]
+</text><text class="breeze-release-management-r5" x="0" y="752" textLength="1464" clip-path="url(#breeze-release-management-line-30)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-release-management-r1" x="1464" y="752" textLength="12.2" clip-path="url(#breeze-release-management-line-30)">
</text>
</g>
</g>
diff --git a/images/breeze/output_release-management.txt b/images/breeze/output_release-management.txt
index a0e84e6bf9..81a7a34a0e 100644
--- a/images/breeze/output_release-management.txt
+++ b/images/breeze/output_release-management.txt
@@ -1 +1 @@
-f4edc0671c6ceb04d81e936f600ba8c4
+531a745e860b739dd3e802e594db6e7e
diff --git a/images/breeze/output_release-management_prepare-airflow-package.svg b/images/breeze/output_release-management_prepare-airflow-package.svg
index c8490f5700..8ee42db614 100644
--- a/images/breeze/output_release-management_prepare-airflow-package.svg
+++ b/images/breeze/output_release-management_prepare-airflow-package.svg
@@ -1,4 +1,4 @@
-<svg class="rich-terminal" viewBox="0 0 1482 391.59999999999997" xmlns="http://www.w3.org/2000/svg">
+<svg class="rich-terminal" viewBox="0 0 1482 440.4" xmlns="http://www.w3.org/2000/svg">
<!-- Generated with Rich https://www.textualize.io -->
<style>
@@ -43,7 +43,7 @@
<defs>
<clipPath id="breeze-release-management-prepare-airflow-package-clip-terminal">
- <rect x="0" y="0" width="1463.0" height="340.59999999999997" />
+ <rect x="0" y="0" width="1463.0" height="389.4" />
</clipPath>
<clipPath id="breeze-release-management-prepare-airflow-package-line-0">
<rect x="0" y="1.5" width="1464" height="24.65"/>
@@ -84,9 +84,15 @@
<clipPath id="breeze-release-management-prepare-airflow-package-line-12">
<rect x="0" y="294.3" width="1464" height="24.65"/>
</clipPath>
+<clipPath id="breeze-release-management-prepare-airflow-package-line-13">
+ <rect x="0" y="318.7" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-prepare-airflow-package-line-14">
+ <rect x="0" y="343.1" width="1464" height="24.65"/>
+ </clipPath>
</defs>
- <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="389.6" rx="8"/><text class="breeze-release-management-prepare-airflow-package-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: release-management prepare-airflow-package</text>
+ <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="438.4" rx="8"/><text class="breeze-release-management-prepare-airflow-package-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: release-management prepare-airflow-package</text>
<g transform="translate(26,22)">
<circle cx="0" cy="0" r="7" fill="#ff5f57"/>
<circle cx="22" cy="0" r="7" fill="#febc2e"/>
@@ -103,13 +109,15 @@
</text><text class="breeze-release-management-prepare-airflow-package-r1" x="1464" y="117.6" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-4)">
</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="142" textLength="24.4" clip-path="url(#breeze-release-management-prepare-airflow-package-line-5)">╭─</text><text class="breeze-release-management-prepare-airflow-package-r5" x="24.4" y="142" textLength="183" clip-path="url(#breeze-release-management-prepare-airflow-package-line-5)"> Package flags </text><text class="breeze-release-management-prepare-airflow-package-r5" x="207.4" y="142" text [...]
</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="166.4" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-6)">│</text><text class="breeze-release-management-prepare-airflow-package-r4" x="24.4" y="166.4" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-6)">-</text><text class="breeze-release-management-prepare-airflow-package-r4" x="36.6" y="166.4" textLength="97.6" clip-path= [...]
-</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="190.8" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-7)">│</text><text class="breeze-release-management-prepare-airflow-package-r4" x="24.4" y="190.8" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-7)">-</text><text class="breeze-release-management-prepare-airflow-package-r4" x="36.6" y="190.8" textLength="97.6" clip-path= [...]
-</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="215.2" textLength="1464" clip-path="url(#breeze-release-management-prepare-airflow-package-line-8)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-release-management-prepare-airflow-package-r1" x="1464" y="215.2" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-8)">
-</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="239.6" textLength="24.4" clip-path="url(#breeze-release-management-prepare-airflow-package-line-9)">╭─</text><text class="breeze-release-management-prepare-airflow-package-r5" x="24.4" y="239.6" textLength="195.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-9)"> Common options </text><text class="breeze-release-management-prepare-airflow-package-r5" x="219.6" y="23 [...]
-</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="264" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-10)">│</text><text class="breeze-release-management-prepare-airflow-package-r4" x="24.4" y="264" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-10)">-</text><text class="breeze-release-management-prepare-airflow-package-r4" x="36.6" y="264" textLength="97.6" clip-path="url [...]
-</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-11)">│</text><text class="breeze-release-management-prepare-airflow-package-r4" x="24.4" y="288.4" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-11)">-</text><text class="breeze-release-management-prepare-airflow-package-r4" x="36.6" y="288.4" textLength="48.8" clip-pat [...]
-</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="312.8" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-12)">│</text><text class="breeze-release-management-prepare-airflow-package-r4" x="24.4" y="312.8" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-12)">-</text><text class="breeze-release-management-prepare-airflow-package-r4" x="36.6" y="312.8" textLength="61" clip-path= [...]
-</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="337.2" textLength="1464" clip-path="url(#breeze-release-management-prepare-airflow-package-line-13)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-release-management-prepare-airflow-package-r1" x="1464" y="337.2" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-13)">
+</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="190.8" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-7)">│</text><text class="breeze-release-management-prepare-airflow-package-r4" x="24.4" y="190.8" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-7)">-</text><text class="breeze-release-management-prepare-airflow-package-r4" x="36.6" y="190.8" textLength="48.8" clip-path= [...]
+</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="215.2" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-8)">│</text><text class="breeze-release-management-prepare-airflow-package-r1" x="378.2" y="215.2" textLength="1061.4" clip-path="url(#breeze-release-management-prepare-airflow-package-line-8)">installed.                  & [...]
+</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="239.6" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-9)">│</text><text class="breeze-release-management-prepare-airflow-package-r4" x="24.4" y="239.6" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-9)">-</text><text class="breeze-release-management-prepare-airflow-package-r4" x="36.6" y="239.6" textLength="97.6" clip-path= [...]
+</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="264" textLength="1464" clip-path="url(#breeze-release-management-prepare-airflow-package-line-10)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-release-management-prepare-airflow-package-r1" x="1464" y="264" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-10)">
+</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="288.4" textLength="24.4" clip-path="url(#breeze-release-management-prepare-airflow-package-line-11)">╭─</text><text class="breeze-release-management-prepare-airflow-package-r5" x="24.4" y="288.4" textLength="195.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-11)"> Common options </text><text class="breeze-release-management-prepare-airflow-package-r5" x="219.6" y=" [...]
+</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="312.8" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-12)">│</text><text class="breeze-release-management-prepare-airflow-package-r4" x="24.4" y="312.8" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-12)">-</text><text class="breeze-release-management-prepare-airflow-package-r4" x="36.6" y="312.8" textLength="97.6" clip-pat [...]
+</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-13)">│</text><text class="breeze-release-management-prepare-airflow-package-r4" x="24.4" y="337.2" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-13)">-</text><text class="breeze-release-management-prepare-airflow-package-r4" x="36.6" y="337.2" textLength="48.8" clip-pat [...]
+</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-14)">│</text><text class="breeze-release-management-prepare-airflow-package-r4" x="24.4" y="361.6" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-14)">-</text><text class="breeze-release-management-prepare-airflow-package-r4" x="36.6" y="361.6" textLength="61" clip-path= [...]
+</text><text class="breeze-release-management-prepare-airflow-package-r5" x="0" y="386" textLength="1464" clip-path="url(#breeze-release-management-prepare-airflow-package-line-15)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-release-management-prepare-airflow-package-r1" x="1464" y="386" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-package-line-15)">
</text>
</g>
</g>
diff --git a/images/breeze/output_release-management_prepare-airflow-package.txt b/images/breeze/output_release-management_prepare-airflow-package.txt
index f295947476..315a3f647a 100644
--- a/images/breeze/output_release-management_prepare-airflow-package.txt
+++ b/images/breeze/output_release-management_prepare-airflow-package.txt
@@ -1 +1 @@
-8f786828b9862f9ae63f1b691921979f
+92215dd76f7de9837fa03e88324d4be7
diff --git a/images/breeze/output_release-management_prepare-airflow-tarball.svg b/images/breeze/output_release-management_prepare-airflow-tarball.svg
new file mode 100644
index 0000000000..5a982d348d
--- /dev/null
+++ b/images/breeze/output_release-management_prepare-airflow-tarball.svg
@@ -0,0 +1,106 @@
+<svg class="rich-terminal" viewBox="0 0 1482 318.4" xmlns="http://www.w3.org/2000/svg">
+ <!-- Generated with Rich https://www.textualize.io -->
+ <style>
+
+ @font-face {
+ font-family: "Fira Code";
+ src: local("FiraCode-Regular"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff2/FiraCode-Regular.woff2") format("woff2"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff/FiraCode-Regular.woff") format("woff");
+ font-style: normal;
+ font-weight: 400;
+ }
+ @font-face {
+ font-family: "Fira Code";
+ src: local("FiraCode-Bold"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff2/FiraCode-Bold.woff2") format("woff2"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff/FiraCode-Bold.woff") format("woff");
+ font-style: bold;
+ font-weight: 700;
+ }
+
+ .breeze-release-management-prepare-airflow-tarball-matrix {
+ font-family: Fira Code, monospace;
+ font-size: 20px;
+ line-height: 24.4px;
+ font-variant-east-asian: full-width;
+ }
+
+ .breeze-release-management-prepare-airflow-tarball-title {
+ font-size: 18px;
+ font-weight: bold;
+ font-family: arial;
+ }
+
+ .breeze-release-management-prepare-airflow-tarball-r1 { fill: #c5c8c6 }
+.breeze-release-management-prepare-airflow-tarball-r2 { fill: #d0b344 }
+.breeze-release-management-prepare-airflow-tarball-r3 { fill: #c5c8c6;font-weight: bold }
+.breeze-release-management-prepare-airflow-tarball-r4 { fill: #68a0b3;font-weight: bold }
+.breeze-release-management-prepare-airflow-tarball-r5 { fill: #868887 }
+.breeze-release-management-prepare-airflow-tarball-r6 { fill: #cc555a }
+.breeze-release-management-prepare-airflow-tarball-r7 { fill: #8d7b39 }
+.breeze-release-management-prepare-airflow-tarball-r8 { fill: #8a4346 }
+.breeze-release-management-prepare-airflow-tarball-r9 { fill: #98a84b;font-weight: bold }
+ </style>
+
+ <defs>
+ <clipPath id="breeze-release-management-prepare-airflow-tarball-clip-terminal">
+ <rect x="0" y="0" width="1463.0" height="267.4" />
+ </clipPath>
+ <clipPath id="breeze-release-management-prepare-airflow-tarball-line-0">
+ <rect x="0" y="1.5" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-prepare-airflow-tarball-line-1">
+ <rect x="0" y="25.9" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-prepare-airflow-tarball-line-2">
+ <rect x="0" y="50.3" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-prepare-airflow-tarball-line-3">
+ <rect x="0" y="74.7" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-prepare-airflow-tarball-line-4">
+ <rect x="0" y="99.1" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-prepare-airflow-tarball-line-5">
+ <rect x="0" y="123.5" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-prepare-airflow-tarball-line-6">
+ <rect x="0" y="147.9" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-prepare-airflow-tarball-line-7">
+ <rect x="0" y="172.3" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-prepare-airflow-tarball-line-8">
+ <rect x="0" y="196.7" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-prepare-airflow-tarball-line-9">
+ <rect x="0" y="221.1" width="1464" height="24.65"/>
+ </clipPath>
+ </defs>
+
+ <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="316.4" rx="8"/><text class="breeze-release-management-prepare-airflow-tarball-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: release-management prepare-airflow-tarball</text>
+ <g transform="translate(26,22)">
+ <circle cx="0" cy="0" r="7" fill="#ff5f57"/>
+ <circle cx="22" cy="0" r="7" fill="#febc2e"/>
+ <circle cx="44" cy="0" r="7" fill="#28c840"/>
+ </g>
+
+ <g transform="translate(9, 41)" clip-path="url(#breeze-release-management-prepare-airflow-tarball-clip-terminal)">
+
+ <g class="breeze-release-management-prepare-airflow-tarball-matrix">
+ <text class="breeze-release-management-prepare-airflow-tarball-r1" x="1464" y="20" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-tarball-line-0)">
+</text><text class="breeze-release-management-prepare-airflow-tarball-r2" x="12.2" y="44.4" textLength="73.2" clip-path="url(#breeze-release-management-prepare-airflow-tarball-line-1)">Usage:</text><text class="breeze-release-management-prepare-airflow-tarball-r3" x="97.6" y="44.4" textLength="597.8" clip-path="url(#breeze-release-management-prepare-airflow-tarball-line-1)">breeze release-management prepare-airflow-tarball</text><text class="breeze-release-management-prepare-ai [...]
+</text><text class="breeze-release-management-prepare-airflow-tarball-r1" x="1464" y="68.8" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-tarball-line-2)">
+</text><text class="breeze-release-management-prepare-airflow-tarball-r1" x="12.2" y="93.2" textLength="402.6" clip-path="url(#breeze-release-management-prepare-airflow-tarball-line-3)">Prepare airflow's source tarball.</text><text class="breeze-release-management-prepare-airflow-tarball-r1" x="1464" y="93.2" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-tarball-line-3)">
+</text><text class="breeze-release-management-prepare-airflow-tarball-r1" x="1464" y="117.6" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-tarball-line-4)">
+</text><text class="breeze-release-management-prepare-airflow-tarball-r5" x="0" y="142" textLength="24.4" clip-path="url(#breeze-release-management-prepare-airflow-tarball-line-5)">╭─</text><text class="breeze-release-management-prepare-airflow-tarball-r5" x="24.4" y="142" textLength="183" clip-path="url(#breeze-release-management-prepare-airflow-tarball-line-5)"> Package flags </text><text class="breeze-release-management-prepare-airflow-tarball-r5" x="207.4" y="142" text [...]
+</text><text class="breeze-release-management-prepare-airflow-tarball-r5" x="0" y="166.4" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-tarball-line-6)">│</text><text class="breeze-release-management-prepare-airflow-tarball-r6" x="24.4" y="166.4" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-tarball-line-6)">*</text><text class="breeze-release-management-prepare-airflow-tarball-r4" x="61" y="166.4" textLength="12.2" clip-path="u [...]
+</text><text class="breeze-release-management-prepare-airflow-tarball-r5" x="0" y="190.8" textLength="1464" clip-path="url(#breeze-release-management-prepare-airflow-tarball-line-7)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-release-management-prepare-airflow-tarball-r1" x="1464" y="190.8" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-tarball-line-7)">
+</text><text class="breeze-release-management-prepare-airflow-tarball-r5" x="0" y="215.2" textLength="24.4" clip-path="url(#breeze-release-management-prepare-airflow-tarball-line-8)">╭─</text><text class="breeze-release-management-prepare-airflow-tarball-r5" x="24.4" y="215.2" textLength="195.2" clip-path="url(#breeze-release-management-prepare-airflow-tarball-line-8)"> Common options </text><text class="breeze-release-management-prepare-airflow-tarball-r5" x="219.6" y="21 [...]
+</text><text class="breeze-release-management-prepare-airflow-tarball-r5" x="0" y="239.6" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-tarball-line-9)">│</text><text class="breeze-release-management-prepare-airflow-tarball-r4" x="24.4" y="239.6" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-tarball-line-9)">-</text><text class="breeze-release-management-prepare-airflow-tarball-r4" x="36.6" y="239.6" textLength="61" clip-path="u [...]
+</text><text class="breeze-release-management-prepare-airflow-tarball-r5" x="0" y="264" textLength="1464" clip-path="url(#breeze-release-management-prepare-airflow-tarball-line-10)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-release-management-prepare-airflow-tarball-r1" x="1464" y="264" textLength="12.2" clip-path="url(#breeze-release-management-prepare-airflow-tarball-line-10)">
+</text>
+ </g>
+ </g>
+</svg>
diff --git a/images/breeze/output_release-management_prepare-airflow-tarball.txt b/images/breeze/output_release-management_prepare-airflow-tarball.txt
new file mode 100644
index 0000000000..12bc4ebe94
--- /dev/null
+++ b/images/breeze/output_release-management_prepare-airflow-tarball.txt
@@ -0,0 +1 @@
+7cf8b45e0b23837e8bf99d5a4907d48c
diff --git a/images/breeze/output_setup_check-all-params-in-groups.svg b/images/breeze/output_setup_check-all-params-in-groups.svg
index f8b7ce58c0..05beab1bb3 100644
--- a/images/breeze/output_setup_check-all-params-in-groups.svg
+++ b/images/breeze/output_setup_check-all-params-in-groups.svg
@@ -170,16 +170,16 @@
</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-14)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="361.6" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-14)">release-management:clean-old-provider-artifacts | release-management:create-minor-branch |          [...]
</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="386" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-15)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="386" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-15)">release-management:generate-constraints | release-management:generate-issue-content-providers |        </tex [...]
</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="410.4" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-16)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="410.4" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-16)">release-management:generate-providers-metadata | release-management:install-provider-packages |        < [...]
-</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="434.8" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-17)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="434.8" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-17)">release-management:prepare-airflow-package | release-management:prepare-provider-documentation |       </text [...]
-</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="459.2" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-18)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="459.2" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-18)">release-management:prepare-provider-packages | release-management:publish-docs |           [...]
-</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-19)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="483.6" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-19)">release-management:release-prod-images | release-management:start-rc-process |            [...]
-</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="508" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-20)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="508" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-20)">release-management:start-release | release-management:update-constraints |            &# [...]
-</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-21)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="532.4" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-21)">release-management:verify-provider-packages | sbom | sbom:build-all-airflow-images |        &# [...]
-</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="556.8" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-22)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="556.8" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-22)">sbom:generate-providers-requirements | sbom:update-sbom-information | setup | setup:autocomplete |    &# [...]
-</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="581.2" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-23)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="581.2" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-23)">setup:check-all-params-in-groups | setup:config | setup:regenerate-command-images | setup:self-upgrade </text><text class="b [...]
-</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="605.6" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-24)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="605.6" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-24)">| setup:synchronize-local-mounts | setup:version | shell | start-airflow | static-checks | testing  [...]
-</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="630" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-25)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="630" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-25)">testing:db-tests | testing:docker-compose-tests | testing:helm-tests | testing:integration-tests |      [...]
-</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="654.4" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-26)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="654.4" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-26)">testing:non-db-tests | testing:tests)                   [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="434.8" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-17)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="434.8" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-17)">release-management:prepare-airflow-package | release-management:prepare-airflow-tarball |         & [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="459.2" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-18)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="459.2" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-18)">release-management:prepare-provider-documentation | release-management:prepare-provider-packages |     </text><text cla [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-19)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="483.6" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-19)">release-management:publish-docs | release-management:release-prod-images |            [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="508" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-20)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="508" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-20)">release-management:start-rc-process | release-management:start-release |             [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-21)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="532.4" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-21)">release-management:update-constraints | release-management:verify-provider-packages | sbom |        [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="556.8" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-22)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="556.8" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-22)">sbom:build-all-airflow-images | sbom:generate-providers-requirements | sbom:update-sbom-information |  </text><text class="b [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="581.2" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-23)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="581.2" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-23)">setup | setup:autocomplete | setup:check-all-params-in-groups | setup:config |        [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="605.6" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-24)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="605.6" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-24)">setup:regenerate-command-images | setup:self-upgrade | setup:synchronize-local-mounts | setup:version |</text><text class="b [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="630" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-25)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="630" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-25)">shell | start-airflow | static-checks | testing | testing:db-tests | testing:docker-compose-tests |  [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="654.4" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-26)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="654.4" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-26)">testing:helm-tests | testing:integration-tests | testing:non-db-tests | testing:tests)       & [...]
</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="678.8" textLength="1464" clip-path="url(#breeze-setup-check-all-params-in-groups-line-27)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-setup-check-all-params-in-groups-r1" x="1464" y="678.8" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-27)">
</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="703.2" textLength="24.4" clip-path="url(#breeze-setup-check-all-params-in-groups-line-28)">╭─</text><text class="breeze-setup-check-all-params-in-groups-r5" x="24.4" y="703.2" textLength="195.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-28)"> Common options </text><text class="breeze-setup-check-all-params-in-groups-r5" x="219.6" y="703.2" textLength="1220" clip-path="url(#breeze-se [...]
</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="727.6" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-29)">│</text><text class="breeze-setup-check-all-params-in-groups-r4" x="24.4" y="727.6" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-29)">-</text><text class="breeze-setup-check-all-params-in-groups-r4" x="36.6" y="727.6" textLength="97.6" clip-path="url(#breeze-setup-check-all-params-in-groups-li [...]
diff --git a/images/breeze/output_setup_check-all-params-in-groups.txt b/images/breeze/output_setup_check-all-params-in-groups.txt
index a785973bff..243d3eca55 100644
--- a/images/breeze/output_setup_check-all-params-in-groups.txt
+++ b/images/breeze/output_setup_check-all-params-in-groups.txt
@@ -1 +1 @@
-ff7ba7e2ad2a27deacc72227a3584b6b
+69ae9158510708ca0793c715a62368b2
diff --git a/images/breeze/output_setup_regenerate-command-images.svg b/images/breeze/output_setup_regenerate-command-images.svg
index 5057857969..fc59c7c211 100644
--- a/images/breeze/output_setup_regenerate-command-images.svg
+++ b/images/breeze/output_setup_regenerate-command-images.svg
@@ -1,4 +1,4 @@
-<svg class="rich-terminal" viewBox="0 0 1482 928.4" xmlns="http://www.w3.org/2000/svg">
+<svg class="rich-terminal" viewBox="0 0 1482 952.8" xmlns="http://www.w3.org/2000/svg">
<!-- Generated with Rich https://www.textualize.io -->
<style>
@@ -43,7 +43,7 @@
<defs>
<clipPath id="breeze-setup-regenerate-command-images-clip-terminal">
- <rect x="0" y="0" width="1463.0" height="877.4" />
+ <rect x="0" y="0" width="1463.0" height="901.8" />
</clipPath>
<clipPath id="breeze-setup-regenerate-command-images-line-0">
<rect x="0" y="1.5" width="1464" height="24.65"/>
@@ -150,9 +150,12 @@
<clipPath id="breeze-setup-regenerate-command-images-line-34">
<rect x="0" y="831.1" width="1464" height="24.65"/>
</clipPath>
+<clipPath id="breeze-setup-regenerate-command-images-line-35">
+ <rect x="0" y="855.5" width="1464" height="24.65"/>
+ </clipPath>
</defs>
- <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="926.4" rx="8"/><text class="breeze-setup-regenerate-command-images-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: setup regenerate-command-images</text>
+ <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="950.8" rx="8"/><text class="breeze-setup-regenerate-command-images-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: setup regenerate-command-images</text>
<g transform="translate(26,22)">
<circle cx="0" cy="0" r="7" fill="#ff5f57"/>
<circle cx="22" cy="0" r="7" fill="#febc2e"/>
@@ -180,24 +183,25 @@
</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="386" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-15)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="386" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-15)">release-management:clean-old-provider-artifacts | release-management:create-minor-branch |          </ [...]
</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="410.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-16)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="410.4" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-16)">release-management:generate-constraints | release-management:generate-issue-content-providers |     </text><text class="bre [...]
</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="434.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-17)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="434.8" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-17)">release-management:generate-providers-metadata | release-management:install-provider-packages |     </text><text class="bre [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="459.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-18)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="459.2" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-18)">release-management:prepare-airflow-package | release-management:prepare-provider-documentation |    </text><text class="breeze-s [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-19)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="483.6" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-19)">release-management:prepare-provider-packages | release-management:publish-docs |           &# [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="508" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-20)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="508" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-20)">release-management:release-prod-images | release-management:start-rc-process |            &# [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-21)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="532.4" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-21)">release-management:start-release | release-management:update-constraints |            &# [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="556.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-22)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="556.8" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-22)">release-management:verify-provider-packages | sbom | sbom:build-all-airflow-images |          [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="581.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-23)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="581.2" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-23)">sbom:generate-providers-requirements | sbom:update-sbom-information | setup | setup:autocomplete |  </text><text class [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="605.6" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-24)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="605.6" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-24)">setup:check-all-params-in-groups | setup:config | setup:regenerate-command-images |         & [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="630" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-25)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="630" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-25)">setup:self-upgrade | setup:synchronize-local-mounts | setup:version | shell | start-airflow |      [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="654.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-26)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="654.4" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-26)">static-checks | testing | testing:db-tests | testing:docker-compose-tests | testing:helm-tests |    [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="678.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-27)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="678.8" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-27)">testing:integration-tests | testing:non-db-tests | testing:tests)              [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="703.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-28)">│</text><text class="breeze-setup-regenerate-command-images-r4" x="24.4" y="703.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-28)">-</text><text class="breeze-setup-regenerate-command-images-r4" x="36.6" y="703.2" textLength="73.2" clip-path="url(#breeze-setup-regenerate-command-images-line-28) [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="727.6" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-29)">│</text><text class="breeze-setup-regenerate-command-images-r1" x="219.6" y="727.6" textLength="170.8" clip-path="url(#breeze-setup-regenerate-command-images-line-29)">together with </text><text class="breeze-setup-regenerate-command-images-r4" x="390.4" y="727.6" textLength="12.2" clip-path="url(#breeze-setup-regenera [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="752" textLength="1464" clip-path="url(#breeze-setup-regenerate-command-images-line-30)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-setup-regenerate-command-images-r1" x="1464" y="752" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-30)">
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="776.4" textLength="24.4" clip-path="url(#breeze-setup-regenerate-command-images-line-31)">╭─</text><text class="breeze-setup-regenerate-command-images-r5" x="24.4" y="776.4" textLength="195.2" clip-path="url(#breeze-setup-regenerate-command-images-line-31)"> Common options </text><text class="breeze-setup-regenerate-command-images-r5" x="219.6" y="776.4" textLength="1220" clip-path="url(#breeze-setup-r [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="800.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-32)">│</text><text class="breeze-setup-regenerate-command-images-r4" x="24.4" y="800.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-32)">-</text><text class="breeze-setup-regenerate-command-images-r4" x="36.6" y="800.8" textLength="97.6" clip-path="url(#breeze-setup-regenerate-command-images-line-32) [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="825.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-33)">│</text><text class="breeze-setup-regenerate-command-images-r4" x="24.4" y="825.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-33)">-</text><text class="breeze-setup-regenerate-command-images-r4" x="36.6" y="825.2" textLength="48.8" clip-path="url(#breeze-setup-regenerate-command-images-line-33) [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="849.6" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-34)">│</text><text class="breeze-setup-regenerate-command-images-r4" x="24.4" y="849.6" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-34)">-</text><text class="breeze-setup-regenerate-command-images-r4" x="36.6" y="849.6" textLength="61" clip-path="url(#breeze-setup-regenerate-command-images-line-34)"> [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="874" textLength="1464" clip-path="url(#breeze-setup-regenerate-command-images-line-35)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-setup-regenerate-command-images-r1" x="1464" y="874" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-35)">
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="459.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-18)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="459.2" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-18)">release-management:prepare-airflow-package | release-management:prepare-airflow-tarball |           [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-19)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="483.6" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-19)">release-management:prepare-provider-documentation | release-management:prepare-provider-packages |  </text><text class="breeze-setup-regen [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="508" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-20)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="508" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-20)">release-management:publish-docs | release-management:release-prod-images |              [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-21)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="532.4" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-21)">release-management:start-rc-process | release-management:start-release |             [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="556.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-22)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="556.8" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-22)">release-management:update-constraints | release-management:verify-provider-packages | sbom |        [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="581.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-23)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="581.2" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-23)">sbom:build-all-airflow-images | sbom:generate-providers-requirements | sbom:update-sbom-information </text><text class="breeze-setup-regen [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="605.6" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-24)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="605.6" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-24)">| setup | setup:autocomplete | setup:check-all-params-in-groups | setup:config |       & [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="630" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-25)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="630" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-25)">setup:regenerate-command-images | setup:self-upgrade | setup:synchronize-local-mounts |         & [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="654.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-26)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="654.4" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-26)">setup:version | shell | start-airflow | static-checks | testing | testing:db-tests |     [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="678.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-27)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="678.8" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-27)">testing:docker-compose-tests | testing:helm-tests | testing:integration-tests | testing:non-db-tests</text><text class="breeze-setup- [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="703.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-28)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="703.2" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-28)">| testing:tests)                        [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="727.6" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-29)">│</text><text class="breeze-setup-regenerate-command-images-r4" x="24.4" y="727.6" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-29)">-</text><text class="breeze-setup-regenerate-command-images-r4" x="36.6" y="727.6" textLength="73.2" clip-path="url(#breeze-setup-regenerate-command-images-line-29) [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="752" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-30)">│</text><text class="breeze-setup-regenerate-command-images-r1" x="219.6" y="752" textLength="170.8" clip-path="url(#breeze-setup-regenerate-command-images-line-30)">together with </text><text class="breeze-setup-regenerate-command-images-r4" x="390.4" y="752" textLength="12.2" clip-path="url(#breeze-setup-regenerate-com [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="776.4" textLength="1464" clip-path="url(#breeze-setup-regenerate-command-images-line-31)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-setup-regenerate-command-images-r1" x="1464" y="776.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-31)">
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="800.8" textLength="24.4" clip-path="url(#breeze-setup-regenerate-command-images-line-32)">╭─</text><text class="breeze-setup-regenerate-command-images-r5" x="24.4" y="800.8" textLength="195.2" clip-path="url(#breeze-setup-regenerate-command-images-line-32)"> Common options </text><text class="breeze-setup-regenerate-command-images-r5" x="219.6" y="800.8" textLength="1220" clip-path="url(#breeze-setup-r [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="825.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-33)">│</text><text class="breeze-setup-regenerate-command-images-r4" x="24.4" y="825.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-33)">-</text><text class="breeze-setup-regenerate-command-images-r4" x="36.6" y="825.2" textLength="97.6" clip-path="url(#breeze-setup-regenerate-command-images-line-33) [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="849.6" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-34)">│</text><text class="breeze-setup-regenerate-command-images-r4" x="24.4" y="849.6" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-34)">-</text><text class="breeze-setup-regenerate-command-images-r4" x="36.6" y="849.6" textLength="48.8" clip-path="url(#breeze-setup-regenerate-command-images-line-34) [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="874" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-35)">│</text><text class="breeze-setup-regenerate-command-images-r4" x="24.4" y="874" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-35)">-</text><text class="breeze-setup-regenerate-command-images-r4" x="36.6" y="874" textLength="61" clip-path="url(#breeze-setup-regenerate-command-images-line-35)">-help< [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="898.4" textLength="1464" clip-path="url(#breeze-setup-regenerate-command-images-line-36)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-setup-regenerate-command-images-r1" x="1464" y="898.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-36)">
</text>
</g>
</g>
diff --git a/images/breeze/output_setup_regenerate-command-images.txt b/images/breeze/output_setup_regenerate-command-images.txt
index c1fea28696..87d87112ba 100644
--- a/images/breeze/output_setup_regenerate-command-images.txt
+++ b/images/breeze/output_setup_regenerate-command-images.txt
@@ -1 +1 @@
-2dd686ef33844512aed7032351111991
+4f1d4b74c81fd81e8c6f32ec137ff480
diff --git a/scripts/in_container/run_prepare_airflow_packages.py b/scripts/in_container/run_prepare_airflow_packages.py
index 66aae93ca4..9305d0dbd4 100755
--- a/scripts/in_container/run_prepare_airflow_packages.py
+++ b/scripts/in_container/run_prepare_airflow_packages.py
@@ -77,11 +77,10 @@ def get_current_airflow_version() -> str:
def build_airflow_packages(package_format: str):
build_command = [sys.executable, "-m", "hatch", "build", "-t", "custom"]
-
- if package_format in ["both", "wheel"]:
- build_command.extend(["-t", "wheel"])
if package_format in ["both", "sdist"]:
build_command.extend(["-t", "sdist"])
+ if package_format in ["both", "wheel"]:
+ build_command.extend(["-t", "wheel"])
reproducible_date = yaml.safe_load(REPRODUCIBLE_BUILD_FILE.read_text())["source-date-epoch"]
(airflow) 13/20: Add version check for k8s setup venv command (#36673)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 8ab07ce525508ce26df1fef264d89844364294d7
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Mon Jan 8 23:23:41 2024 +0100
Add version check for k8s setup venv command (#36673)
This command install airflow in k8s venv and in case version of
Python is not yet supported by Airflow, it might fail.
We do not have check it lower-bound because breeze supports the
same minimum version of Airflow as Airflow itself.
The command prints instructions on how to reinstall breeze with
different Python version in such case.
(cherry picked from commit 9264a4b4e21702a2bc71bb77ee3cc4ada9dfd5e7)
---
.../src/airflow_breeze/utils/kubernetes_utils.py | 28 +++++++++++++++++++++-
1 file changed, 27 insertions(+), 1 deletion(-)
diff --git a/dev/breeze/src/airflow_breeze/utils/kubernetes_utils.py b/dev/breeze/src/airflow_breeze/utils/kubernetes_utils.py
index b7002add2a..8a8db025d5 100644
--- a/dev/breeze/src/airflow_breeze/utils/kubernetes_utils.py
+++ b/dev/breeze/src/airflow_breeze/utils/kubernetes_utils.py
@@ -32,7 +32,13 @@ from time import sleep
from typing import Any, NamedTuple
from urllib import request
-from airflow_breeze.global_constants import ALLOWED_ARCHITECTURES, HELM_VERSION, KIND_VERSION, PIP_VERSION
+from airflow_breeze.global_constants import (
+ ALLOWED_ARCHITECTURES,
+ ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS,
+ HELM_VERSION,
+ KIND_VERSION,
+ PIP_VERSION,
+)
from airflow_breeze.utils.console import Output, get_console
from airflow_breeze.utils.host_info_utils import Architecture, get_host_architecture, get_host_os
from airflow_breeze.utils.path_utils import AIRFLOW_SOURCES_ROOT, BUILD_CACHE_DIR
@@ -330,6 +336,26 @@ def create_virtualenv(force_venv_setup: bool) -> RunCommandResult:
get_console().print(f"[info]Dry run - would be removing {K8S_ENV_PATH}")
else:
shutil.rmtree(K8S_ENV_PATH, ignore_errors=True)
+ max_python_version = ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS[-1]
+ max_python_version_tuple = tuple(int(x) for x in max_python_version.split("."))
+ higher_python_version_tuple = max_python_version_tuple[0], max_python_version_tuple[1] + 1
+ if sys.version_info >= higher_python_version_tuple:
+ get_console().print(
+ f"[red]This is not supported in Python {higher_python_version_tuple} and above[/]\n"
+ )
+ get_console().print(f"[warning]Please use Python version before {higher_python_version_tuple}[/]\n")
+ get_console().print(
+ "[info]You can uninstall breeze and install it again with earlier Python "
+ "version. For example:[/]\n"
+ )
+ get_console().print("pipx uninstall apache-airflow-breeze")
+ get_console().print("pipx install --python PYTHON_PATH -e ./dev/breeze\n")
+ get_console().print(
+ f"[info]PYTHON_PATH - path to your Python binary(< {higher_python_version_tuple})[/]\n"
+ )
+ get_console().print("[info]Then recreate your k8s virtualenv with:[/]\n")
+ get_console().print("breeze k8s setup-env --force-venv-setup\n")
+ sys.exit(1)
venv_command_result = run_command(
[sys.executable, "-m", "venv", str(K8S_ENV_PATH)],
check=False,
(airflow) 07/20: Upgrade to latest ruff and remove ISC001 warning from output (#36649)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit e447cae1c52a28e59dcefce25af42c39348c0b11
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sun Jan 7 20:02:08 2024 +0100
Upgrade to latest ruff and remove ISC001 warning from output (#36649)
This PR upgrades to latest ruff, and removes the ISC001 warning that
warns us against potential conflict between ruff and ruff-formatter
when two strings in one line get concatenated.
This warning makes sense if you run both ruff and formatting at the
same time, but in our case we are doing it in two separate
steps - one step is to run ruff linting and the second step is to
run formatting and running formatting already runs after linting
is complete.
This warnign is pretty misleading as it distracts from real formatting
issues you might have.
There is - unfortunately - no standard way to remove the warning
so we have to do it a little "around" - rather than running
the pre-commit directly from ruff website, we run our local pre-commit
with few lines of Python code that runs ruff through shell and
greps out the ISC001 warning. We also force color to make
sure the output is still coloured.
(cherry picked from commit 11c46fd2ec165da32202b464be7c2df5cca4d6c0)
---
.pre-commit-config.yaml | 32 ++--
STATIC_CODE_CHECKS.rst | 4 +-
airflow/contrib/operators/__init__.py | 189 ++++++++++--------------
airflow/contrib/secrets/__init__.py | 2 +-
airflow/contrib/sensors/__init__.py | 10 +-
airflow/contrib/utils/__init__.py | 4 +-
pyproject.toml | 2 +-
scripts/ci/pre_commit/pre_commit_ruff_format.py | 26 ++++
setup.py | 2 +-
9 files changed, 131 insertions(+), 140 deletions(-)
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 8676195a42..51aba2e09f 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -173,18 +173,6 @@ repos:
additional_dependencies: ['pyyaml']
pass_filenames: false
require_serial: true
- - repo: https://github.com/astral-sh/ruff-pre-commit
- rev: v0.1.9
- hooks:
- # Since ruff makes use of multiple cores we _purposefully_ don't run this in docker so it can use the
- # host CPU to it's fullest
- - id: ruff
- name: ruff-lint
- args: [--fix]
- exclude: ^.*/.*_vendor/|^tests/dags/test_imports.py
- - id: ruff-format
- name: ruff-format
- exclude: ^.*/.*_vendor/|^tests/dags/test_imports.py|^airflow/contrib/
- repo: https://github.com/asottile/blacken-docs
rev: 1.16.0
hooks:
@@ -300,6 +288,26 @@ repos:
# changes quickly - especially when we want the early modifications from the first local group
# to be applied before the non-local pre-commits are run
hooks:
+ - id: ruff
+ name: Run 'ruff' for extremely fast Python linting
+ description: "Run 'ruff' for extremely fast Python linting"
+ entry: ruff check --force-exclude
+ language: python
+ types_or: [python, pyi]
+ args: [--fix]
+ require_serial: true
+ additional_dependencies: ["ruff==0.1.11"]
+ exclude: ^.*/.*_vendor/|^tests/dags/test_imports.py
+ - id: ruff-format
+ name: Run 'ruff format' for extremely fast Python formatting
+ description: "Run 'ruff format' for extremely fast Python formatting"
+ entry: ./scripts/ci/pre_commit/pre_commit_ruff_format.py
+ language: python
+ types_or: [python, pyi]
+ args: []
+ require_serial: true
+ additional_dependencies: ["ruff==0.1.11"]
+ exclude: ^.*/.*_vendor/|^tests/dags/test_imports.py|^airflow/contrib/
- id: replace-bad-characters
name: Replace bad characters
entry: ./scripts/ci/pre_commit/pre_commit_replace_bad_characters.py
diff --git a/STATIC_CODE_CHECKS.rst b/STATIC_CODE_CHECKS.rst
index 57d11e42c2..81911d7375 100644
--- a/STATIC_CODE_CHECKS.rst
+++ b/STATIC_CODE_CHECKS.rst
@@ -360,9 +360,9 @@ require Breeze Docker image to be built locally.
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
| rst-backticks | Check if RST files use double backticks for code | |
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| ruff | ruff-lint | |
+| ruff | Run 'ruff' for extremely fast Python linting | |
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| ruff-format | ruff-format | |
+| ruff-format | Run 'ruff format' for extremely fast Python formatting | |
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
| shellcheck | Check Shell scripts syntax correctness | |
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
diff --git a/airflow/contrib/operators/__init__.py b/airflow/contrib/operators/__init__.py
index faf1899606..86351d05a1 100644
--- a/airflow/contrib/operators/__init__.py
+++ b/airflow/contrib/operators/__init__.py
@@ -180,9 +180,8 @@ __deprecated_classes = {
"DataprocDeleteClusterOperator": (
"airflow.providers.google.cloud.operators.dataproc.DataprocDeleteClusterOperator"
),
- "DataprocInstantiateInlineWorkflowTemplateOperator":
- "airflow.providers.google.cloud.operators.dataproc."
- "DataprocInstantiateInlineWorkflowTemplateOperator",
+ "DataprocInstantiateInlineWorkflowTemplateOperator": "airflow.providers.google.cloud.operators.dataproc."
+ "DataprocInstantiateInlineWorkflowTemplateOperator",
"DataprocInstantiateWorkflowTemplateOperator": (
"airflow.providers.google.cloud.operators.dataproc.DataprocInstantiateWorkflowTemplateOperator"
),
@@ -240,9 +239,8 @@ __deprecated_classes = {
"DataProcSparkSqlOperator": (
"airflow.providers.google.cloud.operators.dataproc.DataprocSubmitSparkSqlJobOperator"
),
- "DataprocWorkflowTemplateInstantiateInlineOperator":
- "airflow.providers.google.cloud.operators.dataproc."
- "DataprocInstantiateInlineWorkflowTemplateOperator",
+ "DataprocWorkflowTemplateInstantiateInlineOperator": "airflow.providers.google.cloud.operators.dataproc."
+ "DataprocInstantiateInlineWorkflowTemplateOperator",
"DataprocWorkflowTemplateInstantiateOperator": (
"airflow.providers.google.cloud.operators.dataproc.DataprocInstantiateWorkflowTemplateOperator"
),
@@ -351,9 +349,8 @@ __deprecated_classes = {
"ComputeEngineCopyInstanceTemplateOperator": (
"airflow.providers.google.cloud.operators.compute.ComputeEngineCopyInstanceTemplateOperator"
),
- "ComputeEngineInstanceGroupUpdateManagerTemplateOperator":
- "airflow.providers.google.cloud.operators.compute."
- "ComputeEngineInstanceGroupUpdateManagerTemplateOperator",
+ "ComputeEngineInstanceGroupUpdateManagerTemplateOperator": "airflow.providers.google.cloud.operators.compute."
+ "ComputeEngineInstanceGroupUpdateManagerTemplateOperator",
"ComputeEngineSetMachineTypeOperator": (
"airflow.providers.google.cloud.operators.compute.ComputeEngineSetMachineTypeOperator"
),
@@ -364,9 +361,8 @@ __deprecated_classes = {
"airflow.providers.google.cloud.operators.compute.ComputeEngineStopInstanceOperator"
),
"GceBaseOperator": "airflow.providers.google.cloud.operators.compute.ComputeEngineBaseOperator",
- "GceInstanceGroupManagerUpdateTemplateOperator":
- "airflow.providers.google.cloud.operators.compute."
- "ComputeEngineInstanceGroupUpdateManagerTemplateOperator",
+ "GceInstanceGroupManagerUpdateTemplateOperator": "airflow.providers.google.cloud.operators.compute."
+ "ComputeEngineInstanceGroupUpdateManagerTemplateOperator",
"GceInstanceStartOperator": (
"airflow.providers.google.cloud.operators.compute.ComputeEngineStartInstanceOperator"
),
@@ -513,30 +509,22 @@ __deprecated_classes = {
),
},
"gcp_natural_language_operator": {
- "CloudNaturalLanguageAnalyzeEntitiesOperator":
- "airflow.providers.google.cloud.operators.natural_language."
- "CloudNaturalLanguageAnalyzeEntitiesOperator",
- "CloudNaturalLanguageAnalyzeEntitySentimentOperator":
- "airflow.providers.google.cloud.operators.natural_language."
- "CloudNaturalLanguageAnalyzeEntitySentimentOperator",
- "CloudNaturalLanguageAnalyzeSentimentOperator":
- "airflow.providers.google.cloud.operators.natural_language."
- "CloudNaturalLanguageAnalyzeSentimentOperator",
- "CloudNaturalLanguageClassifyTextOperator":
- "airflow.providers.google.cloud.operators.natural_language."
- "CloudNaturalLanguageClassifyTextOperator",
- "CloudLanguageAnalyzeEntitiesOperator":
- "airflow.providers.google.cloud.operators.natural_language."
- "CloudNaturalLanguageAnalyzeEntitiesOperator",
- "CloudLanguageAnalyzeEntitySentimentOperator":
- "airflow.providers.google.cloud.operators.natural_language."
- "CloudNaturalLanguageAnalyzeEntitySentimentOperator",
- "CloudLanguageAnalyzeSentimentOperator":
- "airflow.providers.google.cloud.operators.natural_language."
- "CloudNaturalLanguageAnalyzeSentimentOperator",
- "CloudLanguageClassifyTextOperator":
- "airflow.providers.google.cloud.operators.natural_language."
- "CloudNaturalLanguageClassifyTextOperator",
+ "CloudNaturalLanguageAnalyzeEntitiesOperator": "airflow.providers.google.cloud.operators.natural_language."
+ "CloudNaturalLanguageAnalyzeEntitiesOperator",
+ "CloudNaturalLanguageAnalyzeEntitySentimentOperator": "airflow.providers.google.cloud.operators.natural_language."
+ "CloudNaturalLanguageAnalyzeEntitySentimentOperator",
+ "CloudNaturalLanguageAnalyzeSentimentOperator": "airflow.providers.google.cloud.operators.natural_language."
+ "CloudNaturalLanguageAnalyzeSentimentOperator",
+ "CloudNaturalLanguageClassifyTextOperator": "airflow.providers.google.cloud.operators.natural_language."
+ "CloudNaturalLanguageClassifyTextOperator",
+ "CloudLanguageAnalyzeEntitiesOperator": "airflow.providers.google.cloud.operators.natural_language."
+ "CloudNaturalLanguageAnalyzeEntitiesOperator",
+ "CloudLanguageAnalyzeEntitySentimentOperator": "airflow.providers.google.cloud.operators.natural_language."
+ "CloudNaturalLanguageAnalyzeEntitySentimentOperator",
+ "CloudLanguageAnalyzeSentimentOperator": "airflow.providers.google.cloud.operators.natural_language."
+ "CloudNaturalLanguageAnalyzeSentimentOperator",
+ "CloudLanguageClassifyTextOperator": "airflow.providers.google.cloud.operators.natural_language."
+ "CloudNaturalLanguageClassifyTextOperator",
},
"gcp_spanner_operator": {
"SpannerDeleteDatabaseInstanceOperator": (
@@ -692,66 +680,46 @@ __deprecated_classes = {
),
},
"gcp_transfer_operator": {
- "CloudDataTransferServiceCancelOperationOperator":
- "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
- "CloudDataTransferServiceCancelOperationOperator",
- "CloudDataTransferServiceCreateJobOperator":
- "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
- "CloudDataTransferServiceCreateJobOperator",
- "CloudDataTransferServiceDeleteJobOperator":
- "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
- "CloudDataTransferServiceDeleteJobOperator",
- "CloudDataTransferServiceGCSToGCSOperator":
- "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
- "CloudDataTransferServiceGCSToGCSOperator",
- "CloudDataTransferServiceGetOperationOperator":
- "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
- "CloudDataTransferServiceGetOperationOperator",
- "CloudDataTransferServiceListOperationsOperator":
- "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
- "CloudDataTransferServiceListOperationsOperator",
- "CloudDataTransferServicePauseOperationOperator":
- "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
- "CloudDataTransferServicePauseOperationOperator",
- "CloudDataTransferServiceResumeOperationOperator":
- "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
- "CloudDataTransferServiceResumeOperationOperator",
- "CloudDataTransferServiceS3ToGCSOperator":
- "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
- "CloudDataTransferServiceS3ToGCSOperator",
- "CloudDataTransferServiceUpdateJobOperator":
- "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
- "CloudDataTransferServiceUpdateJobOperator",
- "GcpTransferServiceJobCreateOperator":
- "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
- "CloudDataTransferServiceCreateJobOperator",
- "GcpTransferServiceJobDeleteOperator":
- "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
- "CloudDataTransferServiceDeleteJobOperator",
- "GcpTransferServiceJobUpdateOperator":
- "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
- "CloudDataTransferServiceUpdateJobOperator",
- "GcpTransferServiceOperationCancelOperator":
- "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
- "CloudDataTransferServiceCancelOperationOperator",
- "GcpTransferServiceOperationGetOperator":
- "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
- "CloudDataTransferServiceGetOperationOperator",
- "GcpTransferServiceOperationPauseOperator":
- "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
- "CloudDataTransferServicePauseOperationOperator",
- "GcpTransferServiceOperationResumeOperator":
- "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
- "CloudDataTransferServiceResumeOperationOperator",
- "GcpTransferServiceOperationsListOperator":
- "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
- "CloudDataTransferServiceListOperationsOperator",
- "GoogleCloudStorageToGoogleCloudStorageTransferOperator":
- "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
- "CloudDataTransferServiceGCSToGCSOperator",
- "S3ToGoogleCloudStorageTransferOperator":
- "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
- "CloudDataTransferServiceS3ToGCSOperator",
+ "CloudDataTransferServiceCancelOperationOperator": "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
+ "CloudDataTransferServiceCancelOperationOperator",
+ "CloudDataTransferServiceCreateJobOperator": "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
+ "CloudDataTransferServiceCreateJobOperator",
+ "CloudDataTransferServiceDeleteJobOperator": "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
+ "CloudDataTransferServiceDeleteJobOperator",
+ "CloudDataTransferServiceGCSToGCSOperator": "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
+ "CloudDataTransferServiceGCSToGCSOperator",
+ "CloudDataTransferServiceGetOperationOperator": "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
+ "CloudDataTransferServiceGetOperationOperator",
+ "CloudDataTransferServiceListOperationsOperator": "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
+ "CloudDataTransferServiceListOperationsOperator",
+ "CloudDataTransferServicePauseOperationOperator": "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
+ "CloudDataTransferServicePauseOperationOperator",
+ "CloudDataTransferServiceResumeOperationOperator": "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
+ "CloudDataTransferServiceResumeOperationOperator",
+ "CloudDataTransferServiceS3ToGCSOperator": "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
+ "CloudDataTransferServiceS3ToGCSOperator",
+ "CloudDataTransferServiceUpdateJobOperator": "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
+ "CloudDataTransferServiceUpdateJobOperator",
+ "GcpTransferServiceJobCreateOperator": "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
+ "CloudDataTransferServiceCreateJobOperator",
+ "GcpTransferServiceJobDeleteOperator": "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
+ "CloudDataTransferServiceDeleteJobOperator",
+ "GcpTransferServiceJobUpdateOperator": "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
+ "CloudDataTransferServiceUpdateJobOperator",
+ "GcpTransferServiceOperationCancelOperator": "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
+ "CloudDataTransferServiceCancelOperationOperator",
+ "GcpTransferServiceOperationGetOperator": "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
+ "CloudDataTransferServiceGetOperationOperator",
+ "GcpTransferServiceOperationPauseOperator": "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
+ "CloudDataTransferServicePauseOperationOperator",
+ "GcpTransferServiceOperationResumeOperator": "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
+ "CloudDataTransferServiceResumeOperationOperator",
+ "GcpTransferServiceOperationsListOperator": "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
+ "CloudDataTransferServiceListOperationsOperator",
+ "GoogleCloudStorageToGoogleCloudStorageTransferOperator": "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
+ "CloudDataTransferServiceGCSToGCSOperator",
+ "S3ToGoogleCloudStorageTransferOperator": "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
+ "CloudDataTransferServiceS3ToGCSOperator",
},
"gcp_translate_operator": {
"CloudTranslateTextOperator": (
@@ -767,15 +735,12 @@ __deprecated_classes = {
),
},
"gcp_video_intelligence_operator": {
- "CloudVideoIntelligenceDetectVideoExplicitContentOperator":
- "airflow.providers.google.cloud.operators.video_intelligence."
- "CloudVideoIntelligenceDetectVideoExplicitContentOperator",
- "CloudVideoIntelligenceDetectVideoLabelsOperator":
- "airflow.providers.google.cloud.operators.video_intelligence."
- "CloudVideoIntelligenceDetectVideoLabelsOperator",
- "CloudVideoIntelligenceDetectVideoShotsOperator":
- "airflow.providers.google.cloud.operators.video_intelligence."
- "CloudVideoIntelligenceDetectVideoShotsOperator",
+ "CloudVideoIntelligenceDetectVideoExplicitContentOperator": "airflow.providers.google.cloud.operators.video_intelligence."
+ "CloudVideoIntelligenceDetectVideoExplicitContentOperator",
+ "CloudVideoIntelligenceDetectVideoLabelsOperator": "airflow.providers.google.cloud.operators.video_intelligence."
+ "CloudVideoIntelligenceDetectVideoLabelsOperator",
+ "CloudVideoIntelligenceDetectVideoShotsOperator": "airflow.providers.google.cloud.operators.video_intelligence."
+ "CloudVideoIntelligenceDetectVideoShotsOperator",
},
"gcp_vision_operator": {
"CloudVisionAddProductToProductSetOperator": (
@@ -945,9 +910,7 @@ __deprecated_classes = {
"JiraOperator": "airflow.providers.atlassian.jira.operators.jira.JiraOperator",
},
"kubernetes_pod_operator": {
- "KubernetesPodOperator": (
- "airflow.providers.cncf.kubernetes.operators.pod.KubernetesPodOperator"
- ),
+ "KubernetesPodOperator": ("airflow.providers.cncf.kubernetes.operators.pod.KubernetesPodOperator"),
},
"mlengine_operator": {
"MLEngineManageModelOperator": (
@@ -997,9 +960,8 @@ __deprecated_classes = {
"OpsgenieAlertOperator": "airflow.providers.opsgenie.operators.opsgenie.OpsgenieCreateAlertOperator",
},
"oracle_to_azure_data_lake_transfer": {
- "OracleToAzureDataLakeOperator":
- "airflow.providers.microsoft.azure.transfers."
- "oracle_to_azure_data_lake.OracleToAzureDataLakeOperator",
+ "OracleToAzureDataLakeOperator": "airflow.providers.microsoft.azure.transfers."
+ "oracle_to_azure_data_lake.OracleToAzureDataLakeOperator",
},
"oracle_to_oracle_transfer": {
"OracleToOracleOperator": (
@@ -1056,9 +1018,8 @@ __deprecated_classes = {
"S3ToGCSOperator": "airflow.providers.google.cloud.transfers.s3_to_gcs.S3ToGCSOperator",
},
"s3_to_gcs_transfer_operator": {
- "CloudDataTransferServiceS3ToGCSOperator":
- "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
- "CloudDataTransferServiceS3ToGCSOperator",
+ "CloudDataTransferServiceS3ToGCSOperator": "airflow.providers.google.cloud.operators.cloud_storage_transfer_service."
+ "CloudDataTransferServiceS3ToGCSOperator",
},
"s3_to_sftp_operator": {
"S3ToSFTPOperator": "airflow.providers.amazon.aws.transfers.s3_to_sftp.S3ToSFTPOperator",
diff --git a/airflow/contrib/secrets/__init__.py b/airflow/contrib/secrets/__init__.py
index 7498cf0fc5..d97107bfed 100644
--- a/airflow/contrib/secrets/__init__.py
+++ b/airflow/contrib/secrets/__init__.py
@@ -26,7 +26,7 @@ from airflow.utils.deprecation_tools import add_deprecated_classes
warnings.warn(
"This module is deprecated. Please use airflow.providers.*.secrets.",
RemovedInAirflow3Warning,
- stacklevel=2
+ stacklevel=2,
)
__deprecated_classes = {
diff --git a/airflow/contrib/sensors/__init__.py b/airflow/contrib/sensors/__init__.py
index 9d9d1c1557..811263de75 100644
--- a/airflow/contrib/sensors/__init__.py
+++ b/airflow/contrib/sensors/__init__.py
@@ -85,12 +85,10 @@ __deprecated_classes = {
"FTPSSensor": "airflow.providers.ftp.sensors.ftp.FTPSSensor",
},
"gcp_transfer_sensor": {
- "CloudDataTransferServiceJobStatusSensor":
- "airflow.providers.google.cloud.sensors.cloud_storage_transfer_service."
- "CloudDataTransferServiceJobStatusSensor",
- "GCPTransferServiceWaitForJobStatusSensor":
- "airflow.providers.google.cloud.sensors.cloud_storage_transfer_service."
- "CloudDataTransferServiceJobStatusSensor",
+ "CloudDataTransferServiceJobStatusSensor": "airflow.providers.google.cloud.sensors.cloud_storage_transfer_service."
+ "CloudDataTransferServiceJobStatusSensor",
+ "GCPTransferServiceWaitForJobStatusSensor": "airflow.providers.google.cloud.sensors.cloud_storage_transfer_service."
+ "CloudDataTransferServiceJobStatusSensor",
},
"gcs_sensor": {
"GCSObjectExistenceSensor": "airflow.providers.google.cloud.sensors.gcs.GCSObjectExistenceSensor",
diff --git a/airflow/contrib/utils/__init__.py b/airflow/contrib/utils/__init__.py
index c24b7a24e9..0202a1c7c1 100644
--- a/airflow/contrib/utils/__init__.py
+++ b/airflow/contrib/utils/__init__.py
@@ -24,9 +24,7 @@ from airflow.exceptions import RemovedInAirflow3Warning
from airflow.utils.deprecation_tools import add_deprecated_classes
warnings.warn(
- "This module is deprecated. Please use `airflow.utils`.",
- RemovedInAirflow3Warning,
- stacklevel=2
+ "This module is deprecated. Please use `airflow.utils`.", RemovedInAirflow3Warning, stacklevel=2
)
__deprecated_classes = {
diff --git a/pyproject.toml b/pyproject.toml
index 7a305243a1..a8f4bbdeec 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -63,7 +63,7 @@ extend-select = [
"TID251", # Specific modules or module members that may not be imported or accessed
"TID253", # Ban certain modules from being imported at module level
]
-extend-ignore = [
+ignore = [
"D203",
"D212",
"D213",
diff --git a/scripts/ci/pre_commit/pre_commit_ruff_format.py b/scripts/ci/pre_commit/pre_commit_ruff_format.py
new file mode 100755
index 0000000000..f7f9f0f599
--- /dev/null
+++ b/scripts/ci/pre_commit/pre_commit_ruff_format.py
@@ -0,0 +1,26 @@
+#!/usr/bin/env python
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+import subprocess
+
+ruff_format_cmd = "ruff format --force-exclude 2>&1 | grep -v '`ISC001`. To avoid unexpected behavior'"
+envcopy = os.environ.copy()
+envcopy["CLICOLOR_FORCE"] = "1"
+subprocess.run(ruff_format_cmd, shell=True, check=True, env=envcopy)
diff --git a/setup.py b/setup.py
index 2b01c28a6b..0628f3aacf 100644
--- a/setup.py
+++ b/setup.py
@@ -466,7 +466,7 @@ _devel_only_sentry = [
_devel_only_static_checks = [
"pre-commit",
"black",
- "ruff>=0.0.219",
+ "ruff==0.1.11",
"yamllint",
]
(airflow) 19/20: Add support of Pendulum 3 (#36281)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 0c93669556d7eb051a38ec56b8278752a391da0d
Author: Andrey Anshin <An...@taragol.is>
AuthorDate: Fri Jan 12 14:24:12 2024 +0400
Add support of Pendulum 3 (#36281)
* Add support of Pendulum 3
* Add backcompat to pendulum 2
* Update airflow/serialization/serialized_objects.py
Co-authored-by: Tzu-ping Chung <ur...@gmail.com>
* Add newsfragments
---------
Co-authored-by: Tzu-ping Chung <ur...@gmail.com>
(cherry picked from commit 2ffa6e4c4c9dc129daa54491d5af8f535cd0d479)
---
.github/workflows/ci.yml | 92 ++++++++++++-
Dockerfile.ci | 13 ++
airflow/models/dag.py | 6 +-
.../cncf/kubernetes/pod_launcher_deprecated.py | 6 +-
airflow/serialization/serialized_objects.py | 9 +-
airflow/serialization/serializers/datetime.py | 14 +-
airflow/serialization/serializers/timezone.py | 7 +-
airflow/settings.py | 11 +-
airflow/timetables/_cron.py | 9 +-
airflow/timetables/trigger.py | 18 ++-
airflow/utils/sqlalchemy.py | 5 +-
airflow/utils/timezone.py | 50 +++++--
.../src/airflow_breeze/commands/common_options.py | 6 +
.../airflow_breeze/commands/developer_commands.py | 4 +
.../commands/developer_commands_config.py | 1 +
.../airflow_breeze/commands/testing_commands.py | 6 +
.../commands/testing_commands_config.py | 3 +
.../src/airflow_breeze/params/shell_params.py | 2 +
images/breeze/output_shell.svg | 44 +++---
images/breeze/output_shell.txt | 2 +-
images/breeze/output_testing_db-tests.svg | 26 ++--
images/breeze/output_testing_db-tests.txt | 2 +-
images/breeze/output_testing_non-db-tests.svg | 26 ++--
images/breeze/output_testing_non-db-tests.txt | 2 +-
images/breeze/output_testing_tests.svg | 26 ++--
images/breeze/output_testing_tests.txt | 2 +-
kubernetes_tests/test_kubernetes_pod_operator.py | 5 +-
newsfragments/36281.significant.rst | 4 +
pyproject.toml | 4 +-
scripts/ci/docker-compose/devcontainer.env | 1 +
scripts/docker/entrypoint_ci.sh | 14 ++
tests/api_connexion/endpoints/test_dag_endpoint.py | 12 +-
tests/api_connexion/schemas/test_dag_schema.py | 5 +-
tests/cli/commands/test_dag_command.py | 13 +-
tests/models/test_dag.py | 9 +-
tests/providers/openlineage/plugins/test_utils.py | 6 +-
tests/sensors/test_time_sensor.py | 19 ++-
.../serialization/serializers/test_serializers.py | 152 +++++++++++++++++++++
tests/serialization/test_serialized_objects.py | 4 +-
tests/triggers/test_temporal.py | 6 +-
tests/utils/test_timezone.py | 47 ++++++-
41 files changed, 540 insertions(+), 153 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 6ab4213153..922efd9854 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -1169,11 +1169,61 @@ jobs:
breeze testing db-tests
--parallel-test-types "${{needs.build-info.outputs.parallel-test-types-list-as-string}}"
- name: >
- Post Tests success: ${{needs.build-info.outputs.default-python-version}}:Boto"
+ Post Tests success: ${{needs.build-info.outputs.default-python-version}}:MinSQLAlchemy"
uses: ./.github/actions/post_tests_success
if: success()
- name: >
- Post Tests failure: ${{needs.build-info.outputs.default-python-version}}:Boto"
+ Post Tests failure: ${{needs.build-info.outputs.default-python-version}}:MinSQLAlchemy"
+ uses: ./.github/actions/post_tests_failure
+ if: failure()
+
+ tests-postgres-pendulum-2:
+ timeout-minutes: 130
+ name: >
+ DB:Postgres${{needs.build-info.outputs.default-postgres-version}},
+ Pendulum2,Py${{needs.build-info.outputs.default-python-version}}:
+ ${{needs.build-info.outputs.parallel-test-types-list-as-string}}
+ runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
+ needs: [build-info, wait-for-ci-images]
+ env:
+ RUNS_ON: "${{needs.build-info.outputs.runs-on}}"
+ PARALLEL_TEST_TYPES: "${{needs.build-info.outputs.parallel-test-types-list-as-string}}"
+ PR_LABELS: "${{needs.build-info.outputs.pull-request-labels}}"
+ FULL_TESTS_NEEDED: "${{needs.build-info.outputs.full-tests-needed}}"
+ DEBUG_RESOURCES: "${{needs.build-info.outputs.debug-resources}}"
+ BACKEND: "postgres"
+ ENABLE_COVERAGE: "${{needs.build-info.outputs.run-coverage}}"
+ PYTHON_MAJOR_MINOR_VERSION: "${{needs.build-info.outputs.default-python-version}}"
+ PYTHON_VERSION: "${needs.build-info.outputs.default-python-version}}"
+ POSTGRES_VERSION: "${{needs.build-info.outputs.default-postgres-version}}"
+ BACKEND_VERSION: "${{needs.build-info.outputs.default-postgres-version}}"
+ DOWNGRADE_PENDULUM: "true"
+ JOB_ID: >
+ postgres-pendulum-2-${{needs.build-info.outputs.default-python-version}}-
+ ${{needs.build-info.outputs.default-postgres-version}}
+ if: needs.build-info.outputs.run-tests == 'true'
+ steps:
+ - name: Cleanup repo
+ shell: bash
+ run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
+ uses: actions/checkout@v4
+ with:
+ persist-credentials: false
+ - name: >
+ Prepare breeze & CI image: ${{needs.build-info.outputs.default-python-version}}:${{env.IMAGE_TAG}}
+ uses: ./.github/actions/prepare_breeze_and_image
+ - name: >
+ Tests: ${{matrix.python-version}}:${{needs.build-info.outputs.parallel-test-types-list-as-string}}
+ run: >
+ breeze testing db-tests
+ --parallel-test-types "${{needs.build-info.outputs.parallel-test-types-list-as-string}}"
+ - name: >
+ Post Tests success: ${{needs.build-info.outputs.default-python-version}}:Pendulum2"
+ uses: ./.github/actions/post_tests_success
+ if: success()
+ - name: >
+ Post Tests failure: ${{needs.build-info.outputs.default-python-version}}:Pendulum2"
uses: ./.github/actions/post_tests_failure
if: failure()
@@ -1616,6 +1666,44 @@ jobs:
uses: ./.github/actions/post_tests_failure
if: failure()
+ tests-no-db-pendulum-2:
+ timeout-minutes: 60
+ name: >
+ Non-DB: Pendulum2, Py${{needs.build-info.outputs.default-python-version}}:
+ ${{needs.build-info.outputs.parallel-test-types-list-as-string}}
+ runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
+ needs: [build-info, wait-for-ci-images]
+ env:
+ RUNS_ON: "${{needs.build-info.outputs.runs-on}}"
+ PR_LABELS: "${{needs.build-info.outputs.pull-request-labels}}"
+ PYTHON_MAJOR_MINOR_VERSION: "${{needs.build-info.outputs.default-python-version}}"
+ DEBUG_RESOURCES: "${{needs.build-info.outputs.debug-resources}}"
+ JOB_ID: "quarantined-${{needs.build-info.outputs.default-python-version}}"
+ ENABLE_COVERAGE: "${{needs.build-info.outputs.run-coverage}}"
+ DOWNGRADE_PENDULUM: "true"
+ if: needs.build-info.outputs.run-tests == 'true'
+ steps:
+ - name: Cleanup repo
+ shell: bash
+ run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
+ uses: actions/checkout@v4
+ with:
+ persist-credentials: false
+ - name: >
+ Prepare breeze & CI image: ${{needs.build-info.outputs.default-python-version}}:${{env.IMAGE_TAG}}
+ uses: ./.github/actions/prepare_breeze_and_image
+ - name: "Tests: ${{matrix.python-version}}:Non-DB-Pendulum2"
+ run: >
+ breeze testing non-db-tests
+ --parallel-test-types "${{needs.build-info.outputs.parallel-test-types-list-as-string}}"
+ - name: "Post Tests success: Non-DB-Pendulum2"
+ uses: ./.github/actions/post_tests_success
+ if: success()
+ - name: "Post Tests failure: Non-DB-Pendulum2"
+ uses: ./.github/actions/post_tests_failure
+ if: failure()
+
summarize-warnings:
timeout-minutes: 15
name: "Summarize warnings"
diff --git a/Dockerfile.ci b/Dockerfile.ci
index 0f10757de5..5487e32af0 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -908,6 +908,18 @@ function check_download_sqlalchemy() {
pip check
}
+function check_download_pendulum() {
+ if [[ ${DOWNGRADE_PENDULUM=} != "true" ]]; then
+ return
+ fi
+ min_pendulum_version=$(grep "\"pendulum>=" pyproject.toml | sed "s/.*>=\([0-9\.]*\).*/\1/" | xargs)
+ echo
+ echo "${COLOR_BLUE}Downgrading pendulum to minimum supported version: ${min_pendulum_version}${COLOR_RESET}"
+ echo
+ pip install --root-user-action ignore "pendulum==${min_pendulum_version}"
+ pip check
+}
+
function check_run_tests() {
if [[ ${RUN_TESTS=} != "true" ]]; then
return
@@ -937,6 +949,7 @@ determine_airflow_to_use
environment_initialization
check_boto_upgrade
check_download_sqlalchemy
+check_download_pendulum
check_run_tests "${@}"
exec /bin/bash "${@}"
diff --git a/airflow/models/dag.py b/airflow/models/dag.py
index d0f46feed2..c0abadf339 100644
--- a/airflow/models/dag.py
+++ b/airflow/models/dag.py
@@ -138,7 +138,7 @@ from airflow.utils.types import NOTSET, ArgNotSet, DagRunType, EdgeInfoType
if TYPE_CHECKING:
from types import ModuleType
- from pendulum.tz.timezone import Timezone
+ from pendulum.tz.timezone import FixedTimezone, Timezone
from sqlalchemy.orm.query import Query
from sqlalchemy.orm.session import Session
@@ -213,7 +213,7 @@ def _get_model_data_interval(
return DataInterval(start, end)
-def create_timetable(interval: ScheduleIntervalArg, timezone: Timezone) -> Timetable:
+def create_timetable(interval: ScheduleIntervalArg, timezone: Timezone | FixedTimezone) -> Timetable:
"""Create a Timetable instance from a ``schedule_interval`` argument."""
if interval is NOTSET:
return DeltaDataIntervalTimetable(DEFAULT_SCHEDULE_INTERVAL)
@@ -529,7 +529,7 @@ class DAG(LoggingMixin):
tzinfo = None if date.tzinfo else settings.TIMEZONE
tz = pendulum.instance(date, tz=tzinfo).timezone
- self.timezone: Timezone = tz or settings.TIMEZONE
+ self.timezone: Timezone | FixedTimezone = tz or settings.TIMEZONE
# Apply the timezone we settled on to end_date if it wasn't supplied
if "end_date" in self.default_args and self.default_args["end_date"]:
diff --git a/airflow/providers/cncf/kubernetes/pod_launcher_deprecated.py b/airflow/providers/cncf/kubernetes/pod_launcher_deprecated.py
index 18799ed920..6c5f038b0a 100644
--- a/airflow/providers/cncf/kubernetes/pod_launcher_deprecated.py
+++ b/airflow/providers/cncf/kubernetes/pod_launcher_deprecated.py
@@ -21,7 +21,7 @@ import json
import math
import time
import warnings
-from typing import TYPE_CHECKING
+from typing import TYPE_CHECKING, cast
import pendulum
import tenacity
@@ -148,13 +148,13 @@ class PodLauncher(LoggingMixin):
"""
if get_logs:
read_logs_since_sec = None
- last_log_time = None
+ last_log_time: pendulum.DateTime | None = None
while True:
logs = self.read_pod_logs(pod, timestamps=True, since_seconds=read_logs_since_sec)
for line in logs:
timestamp, message = self.parse_log_line(line.decode("utf-8"))
if timestamp:
- last_log_time = pendulum.parse(timestamp)
+ last_log_time = cast(pendulum.DateTime, pendulum.parse(timestamp))
self.log.info(message)
time.sleep(1)
diff --git a/airflow/serialization/serialized_objects.py b/airflow/serialization/serialized_objects.py
index 48aa595933..87ee4d4a73 100644
--- a/airflow/serialization/serialized_objects.py
+++ b/airflow/serialization/serialized_objects.py
@@ -65,6 +65,7 @@ from airflow.utils.docs import get_docs_url
from airflow.utils.module_loading import import_string, qualname
from airflow.utils.operator_resources import Resources
from airflow.utils.task_group import MappedTaskGroup, TaskGroup
+from airflow.utils.timezone import parse_timezone
from airflow.utils.types import NOTSET, ArgNotSet
if TYPE_CHECKING:
@@ -144,7 +145,7 @@ def decode_relativedelta(var: dict[str, Any]) -> relativedelta.relativedelta:
return relativedelta.relativedelta(**var)
-def encode_timezone(var: Timezone) -> str | int:
+def encode_timezone(var: Timezone | FixedTimezone) -> str | int:
"""
Encode a Pendulum Timezone for serialization.
@@ -167,9 +168,9 @@ def encode_timezone(var: Timezone) -> str | int:
)
-def decode_timezone(var: str | int) -> Timezone:
+def decode_timezone(var: str | int) -> Timezone | FixedTimezone:
"""Decode a previously serialized Pendulum Timezone."""
- return pendulum.tz.timezone(var)
+ return parse_timezone(var)
def _get_registered_timetable(importable_string: str) -> type[Timetable] | None:
@@ -607,7 +608,7 @@ class BaseSerialization:
raise TypeError(f"Invalid type {type_!s} in deserialization.")
_deserialize_datetime = pendulum.from_timestamp
- _deserialize_timezone = pendulum.tz.timezone
+ _deserialize_timezone = parse_timezone
@classmethod
def _deserialize_timedelta(cls, seconds: int) -> datetime.timedelta:
diff --git a/airflow/serialization/serializers/datetime.py b/airflow/serialization/serializers/datetime.py
index d32dd8897b..69058b8c02 100644
--- a/airflow/serialization/serializers/datetime.py
+++ b/airflow/serialization/serializers/datetime.py
@@ -24,6 +24,7 @@ from airflow.serialization.serializers.timezone import (
serialize as serialize_timezone,
)
from airflow.utils.module_loading import qualname
+from airflow.utils.timezone import parse_timezone
if TYPE_CHECKING:
import datetime
@@ -62,23 +63,22 @@ def deserialize(classname: str, version: int, data: dict | str) -> datetime.date
import datetime
from pendulum import DateTime
- from pendulum.tz import fixed_timezone, timezone
tz: datetime.tzinfo | None = None
if isinstance(data, dict) and TIMEZONE in data:
if version == 1:
# try to deserialize unsupported timezones
timezone_mapping = {
- "EDT": fixed_timezone(-4 * 3600),
- "CDT": fixed_timezone(-5 * 3600),
- "MDT": fixed_timezone(-6 * 3600),
- "PDT": fixed_timezone(-7 * 3600),
- "CEST": timezone("CET"),
+ "EDT": parse_timezone(-4 * 3600),
+ "CDT": parse_timezone(-5 * 3600),
+ "MDT": parse_timezone(-6 * 3600),
+ "PDT": parse_timezone(-7 * 3600),
+ "CEST": parse_timezone("CET"),
}
if data[TIMEZONE] in timezone_mapping:
tz = timezone_mapping[data[TIMEZONE]]
else:
- tz = timezone(data[TIMEZONE])
+ tz = parse_timezone(data[TIMEZONE])
else:
tz = (
deserialize_timezone(data[TIMEZONE][1], data[TIMEZONE][2], data[TIMEZONE][0])
diff --git a/airflow/serialization/serializers/timezone.py b/airflow/serialization/serializers/timezone.py
index 23901b9d44..0f580adef8 100644
--- a/airflow/serialization/serializers/timezone.py
+++ b/airflow/serialization/serializers/timezone.py
@@ -74,7 +74,7 @@ def serialize(o: object) -> tuple[U, str, int, bool]:
def deserialize(classname: str, version: int, data: object) -> Any:
- from pendulum.tz import fixed_timezone, timezone
+ from airflow.utils.timezone import parse_timezone
if not isinstance(data, (str, int)):
raise TypeError(f"{data} is not of type int or str but of {type(data)}")
@@ -82,9 +82,6 @@ def deserialize(classname: str, version: int, data: object) -> Any:
if version > __version__:
raise TypeError(f"serialized {version} of {classname} > {__version__}")
- if isinstance(data, int):
- return fixed_timezone(data)
-
if "zoneinfo.ZoneInfo" in classname:
try:
from zoneinfo import ZoneInfo
@@ -93,7 +90,7 @@ def deserialize(classname: str, version: int, data: object) -> Any:
return ZoneInfo(data)
- return timezone(data)
+ return parse_timezone(data)
# ported from pendulum.tz.timezone._get_tzinfo_name
diff --git a/airflow/settings.py b/airflow/settings.py
index 1a38a59ed3..53c5cc6aa4 100644
--- a/airflow/settings.py
+++ b/airflow/settings.py
@@ -26,7 +26,6 @@ import sys
import warnings
from typing import TYPE_CHECKING, Any, Callable
-import pendulum
import pluggy
import sqlalchemy
from sqlalchemy import create_engine, exc, text
@@ -40,6 +39,7 @@ from airflow.executors import executor_constants
from airflow.logging_config import configure_logging
from airflow.utils.orm_event_handlers import setup_event_handlers
from airflow.utils.state import State
+from airflow.utils.timezone import local_timezone, parse_timezone, utc
if TYPE_CHECKING:
from sqlalchemy.engine import Engine
@@ -50,13 +50,12 @@ if TYPE_CHECKING:
log = logging.getLogger(__name__)
try:
- tz = conf.get_mandatory_value("core", "default_timezone")
- if tz == "system":
- TIMEZONE = pendulum.tz.local_timezone()
+ if (tz := conf.get_mandatory_value("core", "default_timezone")) != "system":
+ TIMEZONE = parse_timezone(tz)
else:
- TIMEZONE = pendulum.tz.timezone(tz)
+ TIMEZONE = local_timezone()
except Exception:
- TIMEZONE = pendulum.tz.timezone("UTC")
+ TIMEZONE = utc
log.info("Configured default timezone %s", TIMEZONE)
diff --git a/airflow/timetables/_cron.py b/airflow/timetables/_cron.py
index b0e6e256ee..fa2fb1266f 100644
--- a/airflow/timetables/_cron.py
+++ b/airflow/timetables/_cron.py
@@ -19,17 +19,16 @@ from __future__ import annotations
import datetime
from typing import TYPE_CHECKING, Any
-import pendulum
from cron_descriptor import CasingTypeEnum, ExpressionDescriptor, FormatException, MissingFieldException
from croniter import CroniterBadCronError, CroniterBadDateError, croniter
from airflow.exceptions import AirflowTimetableInvalid
from airflow.utils.dates import cron_presets
-from airflow.utils.timezone import convert_to_utc, make_aware, make_naive
+from airflow.utils.timezone import convert_to_utc, make_aware, make_naive, parse_timezone
if TYPE_CHECKING:
from pendulum import DateTime
- from pendulum.tz.timezone import Timezone
+ from pendulum.tz.timezone import FixedTimezone, Timezone
def _covers_every_hour(cron: croniter) -> bool:
@@ -63,11 +62,11 @@ def _covers_every_hour(cron: croniter) -> bool:
class CronMixin:
"""Mixin to provide interface to work with croniter."""
- def __init__(self, cron: str, timezone: str | Timezone) -> None:
+ def __init__(self, cron: str, timezone: str | Timezone | FixedTimezone) -> None:
self._expression = cron_presets.get(cron, cron)
if isinstance(timezone, str):
- timezone = pendulum.tz.timezone(timezone)
+ timezone = parse_timezone(timezone)
self._timezone = timezone
try:
diff --git a/airflow/timetables/trigger.py b/airflow/timetables/trigger.py
index 95d2923803..2a0df645da 100644
--- a/airflow/timetables/trigger.py
+++ b/airflow/timetables/trigger.py
@@ -26,7 +26,7 @@ from airflow.timetables.base import DagRunInfo, DataInterval, Timetable
if TYPE_CHECKING:
from dateutil.relativedelta import relativedelta
- from pendulum.tz.timezone import Timezone
+ from pendulum.tz.timezone import FixedTimezone, Timezone
from airflow.timetables.base import TimeRestriction
@@ -48,7 +48,7 @@ class CronTriggerTimetable(CronMixin, Timetable):
self,
cron: str,
*,
- timezone: str | Timezone,
+ timezone: str | Timezone | FixedTimezone,
interval: datetime.timedelta | relativedelta = datetime.timedelta(),
) -> None:
super().__init__(cron, timezone)
@@ -77,7 +77,12 @@ class CronTriggerTimetable(CronMixin, Timetable):
return {"expression": self._expression, "timezone": timezone, "interval": interval}
def infer_manual_data_interval(self, *, run_after: DateTime) -> DataInterval:
- return DataInterval(run_after - self._interval, run_after)
+ return DataInterval(
+ # pendulum.Datetime ± timedelta should return pendulum.Datetime
+ # however mypy decide that output would be datetime.datetime
+ run_after - self._interval, # type: ignore[arg-type]
+ run_after,
+ )
def next_dagrun_info(
self,
@@ -101,4 +106,9 @@ class CronTriggerTimetable(CronMixin, Timetable):
next_start_time = max(start_time_candidates)
if restriction.latest is not None and restriction.latest < next_start_time:
return None
- return DagRunInfo.interval(next_start_time - self._interval, next_start_time)
+ return DagRunInfo.interval(
+ # pendulum.Datetime ± timedelta should return pendulum.Datetime
+ # however mypy decide that output would be datetime.datetime
+ next_start_time - self._interval, # type: ignore[arg-type]
+ next_start_time,
+ )
diff --git a/airflow/utils/sqlalchemy.py b/airflow/utils/sqlalchemy.py
index a042d4e902..fb241f482f 100644
--- a/airflow/utils/sqlalchemy.py
+++ b/airflow/utils/sqlalchemy.py
@@ -24,7 +24,6 @@ import json
import logging
from typing import TYPE_CHECKING, Any, Generator, Iterable, overload
-import pendulum
from dateutil import relativedelta
from sqlalchemy import TIMESTAMP, PickleType, and_, event, false, nullsfirst, or_, true, tuple_
from sqlalchemy.dialects import mssql, mysql
@@ -34,7 +33,7 @@ from sqlalchemy.types import JSON, Text, TypeDecorator, UnicodeText
from airflow import settings
from airflow.configuration import conf
from airflow.serialization.enums import Encoding
-from airflow.utils.timezone import make_naive
+from airflow.utils.timezone import make_naive, utc
if TYPE_CHECKING:
from kubernetes.client.models.v1_pod import V1Pod
@@ -46,8 +45,6 @@ if TYPE_CHECKING:
log = logging.getLogger(__name__)
-utc = pendulum.tz.timezone("UTC")
-
class UtcDateTime(TypeDecorator):
"""
diff --git a/airflow/utils/timezone.py b/airflow/utils/timezone.py
index 12c75bef59..8ac9a49e0e 100644
--- a/airflow/utils/timezone.py
+++ b/airflow/utils/timezone.py
@@ -18,14 +18,20 @@
from __future__ import annotations
import datetime as dt
-from typing import overload
+from typing import TYPE_CHECKING, overload
import pendulum
from dateutil.relativedelta import relativedelta
from pendulum.datetime import DateTime
-# UTC time zone as a tzinfo instance.
-utc = pendulum.tz.timezone("UTC")
+if TYPE_CHECKING:
+ from pendulum.tz.timezone import FixedTimezone, Timezone
+
+_PENDULUM3 = pendulum.__version__.startswith("3")
+# UTC Timezone as a tzinfo instance. Actual value depends on pendulum version:
+# - Timezone("UTC") in pendulum 3
+# - FixedTimezone(0, "UTC") in pendulum 2
+utc = pendulum.UTC
def is_localized(value):
@@ -135,12 +141,10 @@ def make_aware(value: dt.datetime | None, timezone: dt.tzinfo | None = None) ->
# Check that we won't overwrite the timezone of an aware datetime.
if is_localized(value):
raise ValueError(f"make_aware expects a naive datetime, got {value}")
- if hasattr(value, "fold"):
- # In case of python 3.6 we want to do the same that pendulum does for python3.5
- # i.e in case we move clock back we want to schedule the run at the time of the second
- # instance of the same clock time rather than the first one.
- # Fold parameter has no impact in other cases so we can safely set it to 1 here
- value = value.replace(fold=1)
+ # In case we move clock back we want to schedule the run at the time of the second
+ # instance of the same clock time rather than the first one.
+ # Fold parameter has no impact in other cases, so we can safely set it to 1 here
+ value = value.replace(fold=1)
localized = getattr(timezone, "localize", None)
if localized is not None:
# This method is available for pytz time zones
@@ -273,3 +277,31 @@ def td_format(td_object: None | dt.timedelta | float | int) -> str | None:
if not joined:
return "<1s"
return joined
+
+
+def parse_timezone(name: str | int) -> FixedTimezone | Timezone:
+ """
+ Parse timezone and return one of the pendulum Timezone.
+
+ Provide the same interface as ``pendulum.timezone(name)``
+
+ :param name: Either IANA timezone or offset to UTC in seconds.
+
+ :meta private:
+ """
+ if _PENDULUM3:
+ # This only presented in pendulum 3 and code do not reached into the pendulum 2
+ return pendulum.timezone(name) # type: ignore[operator]
+ # In pendulum 2 this refers to the function, in pendulum 3 refers to the module
+ return pendulum.tz.timezone(name) # type: ignore[operator]
+
+
+def local_timezone() -> FixedTimezone | Timezone:
+ """
+ Return local timezone.
+
+ Provide the same interface as ``pendulum.tz.local_timezone()``
+
+ :meta private:
+ """
+ return pendulum.tz.local_timezone()
diff --git a/dev/breeze/src/airflow_breeze/commands/common_options.py b/dev/breeze/src/airflow_breeze/commands/common_options.py
index 12d0ee77b8..a280db4e2c 100644
--- a/dev/breeze/src/airflow_breeze/commands/common_options.py
+++ b/dev/breeze/src/airflow_breeze/commands/common_options.py
@@ -151,6 +151,12 @@ option_downgrade_sqlalchemy = click.option(
is_flag=True,
envvar="DOWNGRADE_SQLALCHEMY",
)
+option_downgrade_pendulum = click.option(
+ "--downgrade-pendulum",
+ help="Downgrade Pendulum to minimum supported version.",
+ is_flag=True,
+ envvar="DOWNGRADE_PENDULUM",
+)
option_dry_run = click.option(
"-D",
"--dry-run",
diff --git a/dev/breeze/src/airflow_breeze/commands/developer_commands.py b/dev/breeze/src/airflow_breeze/commands/developer_commands.py
index ede6dfd933..27c43fc11b 100644
--- a/dev/breeze/src/airflow_breeze/commands/developer_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/developer_commands.py
@@ -39,6 +39,7 @@ from airflow_breeze.commands.common_options import (
option_database_isolation,
option_db_reset,
option_docker_host,
+ option_downgrade_pendulum,
option_downgrade_sqlalchemy,
option_dry_run,
option_forward_credentials,
@@ -248,6 +249,7 @@ option_warn_image_upgrade_needed = click.option(
@option_db_reset
@option_docker_host
@option_downgrade_sqlalchemy
+@option_downgrade_pendulum
@option_dry_run
@option_executor_shell
@option_force_build
@@ -294,6 +296,7 @@ def shell(
database_isolation: bool,
db_reset: bool,
downgrade_sqlalchemy: bool,
+ downgrade_pendulum: bool,
docker_host: str | None,
executor: str,
extra_args: tuple,
@@ -354,6 +357,7 @@ def shell(
database_isolation=database_isolation,
db_reset=db_reset,
downgrade_sqlalchemy=downgrade_sqlalchemy,
+ downgrade_pendulum=downgrade_pendulum,
docker_host=docker_host,
executor=executor,
extra_args=extra_args if not max_time else ["exit"],
diff --git a/dev/breeze/src/airflow_breeze/commands/developer_commands_config.py b/dev/breeze/src/airflow_breeze/commands/developer_commands_config.py
index 88b734f513..911ed9ebb5 100644
--- a/dev/breeze/src/airflow_breeze/commands/developer_commands_config.py
+++ b/dev/breeze/src/airflow_breeze/commands/developer_commands_config.py
@@ -159,6 +159,7 @@ DEVELOPER_PARAMETERS: dict[str, list[dict[str, str | list[str]]]] = {
"options": [
"--upgrade-boto",
"--downgrade-sqlalchemy",
+ "--downgrade-pendulum",
],
},
{
diff --git a/dev/breeze/src/airflow_breeze/commands/testing_commands.py b/dev/breeze/src/airflow_breeze/commands/testing_commands.py
index c4aff8797e..f826d9bdff 100644
--- a/dev/breeze/src/airflow_breeze/commands/testing_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/testing_commands.py
@@ -29,6 +29,7 @@ from airflow_breeze.commands.common_options import (
option_backend,
option_db_reset,
option_debug_resources,
+ option_downgrade_pendulum,
option_downgrade_sqlalchemy,
option_dry_run,
option_forward_credentials,
@@ -471,6 +472,7 @@ option_remove_arm_packages = click.option(
@option_excluded_parallel_test_types
@option_upgrade_boto
@option_downgrade_sqlalchemy
+@option_downgrade_pendulum
@option_collect_only
@option_remove_arm_packages
@option_skip_docker_compose_down
@@ -513,6 +515,7 @@ def command_for_tests(**kwargs):
@option_excluded_parallel_test_types
@option_upgrade_boto
@option_downgrade_sqlalchemy
+@option_downgrade_pendulum
@option_collect_only
@option_remove_arm_packages
@option_skip_docker_compose_down
@@ -548,6 +551,7 @@ def command_for_db_tests(**kwargs):
@option_collect_only
@option_debug_resources
@option_downgrade_sqlalchemy
+@option_downgrade_pendulum
@option_dry_run
@option_enable_coverage
@option_excluded_parallel_test_types
@@ -589,6 +593,7 @@ def _run_test_command(
db_reset: bool,
debug_resources: bool,
downgrade_sqlalchemy: bool,
+ downgrade_pendulum: bool,
enable_coverage: bool,
excluded_parallel_test_types: str,
extra_pytest_args: tuple,
@@ -632,6 +637,7 @@ def _run_test_command(
backend=backend,
collect_only=collect_only,
downgrade_sqlalchemy=downgrade_sqlalchemy,
+ downgrade_pendulum=downgrade_pendulum,
enable_coverage=enable_coverage,
forward_credentials=forward_credentials,
forward_ports=False,
diff --git a/dev/breeze/src/airflow_breeze/commands/testing_commands_config.py b/dev/breeze/src/airflow_breeze/commands/testing_commands_config.py
index 404e0cabe0..370cdad91f 100644
--- a/dev/breeze/src/airflow_breeze/commands/testing_commands_config.py
+++ b/dev/breeze/src/airflow_breeze/commands/testing_commands_config.py
@@ -79,6 +79,7 @@ TESTING_PARAMETERS: dict[str, list[dict[str, str | list[str]]]] = {
"--mount-sources",
"--upgrade-boto",
"--downgrade-sqlalchemy",
+ "--downgrade-pendulum",
"--remove-arm-packages",
"--skip-docker-compose-down",
],
@@ -126,6 +127,7 @@ TESTING_PARAMETERS: dict[str, list[dict[str, str | list[str]]]] = {
"--mount-sources",
"--upgrade-boto",
"--downgrade-sqlalchemy",
+ "--downgrade-pendulum",
"--remove-arm-packages",
"--skip-docker-compose-down",
],
@@ -177,6 +179,7 @@ TESTING_PARAMETERS: dict[str, list[dict[str, str | list[str]]]] = {
"--mount-sources",
"--upgrade-boto",
"--downgrade-sqlalchemy",
+ "--downgrade-pendulum",
"--remove-arm-packages",
"--skip-docker-compose-down",
],
diff --git a/dev/breeze/src/airflow_breeze/params/shell_params.py b/dev/breeze/src/airflow_breeze/params/shell_params.py
index 85fbecd6ca..fbc02b7922 100644
--- a/dev/breeze/src/airflow_breeze/params/shell_params.py
+++ b/dev/breeze/src/airflow_breeze/params/shell_params.py
@@ -148,6 +148,7 @@ class ShellParams:
dev_mode: bool = False
docker_host: str | None = os.environ.get("DOCKER_HOST")
downgrade_sqlalchemy: bool = False
+ downgrade_pendulum: bool = False
dry_run: bool = False
enable_coverage: bool = False
executor: str = START_AIRFLOW_DEFAULT_ALLOWED_EXECUTOR
@@ -516,6 +517,7 @@ class ShellParams:
_set_var(_env, "DEV_MODE", self.dev_mode)
_set_var(_env, "DOCKER_IS_ROOTLESS", self.rootless_docker)
_set_var(_env, "DOWNGRADE_SQLALCHEMY", self.downgrade_sqlalchemy)
+ _set_var(_env, "DOWNGRADE_PENDULUM", self.downgrade_pendulum)
_set_var(_env, "ENABLED_SYSTEMS", None, "")
_set_var(_env, "FLOWER_HOST_PORT", None, FLOWER_HOST_PORT)
_set_var(_env, "GITHUB_ACTIONS", self.github_actions)
diff --git a/images/breeze/output_shell.svg b/images/breeze/output_shell.svg
index 8d9d7350d5..84151382c7 100644
--- a/images/breeze/output_shell.svg
+++ b/images/breeze/output_shell.svg
@@ -1,4 +1,4 @@
-<svg class="rich-terminal" viewBox="0 0 1482 3026.7999999999997" xmlns="http://www.w3.org/2000/svg">
+<svg class="rich-terminal" viewBox="0 0 1482 3051.2" xmlns="http://www.w3.org/2000/svg">
<!-- Generated with Rich https://www.textualize.io -->
<style>
@@ -43,7 +43,7 @@
<defs>
<clipPath id="breeze-shell-clip-terminal">
- <rect x="0" y="0" width="1463.0" height="2975.7999999999997" />
+ <rect x="0" y="0" width="1463.0" height="3000.2" />
</clipPath>
<clipPath id="breeze-shell-line-0">
<rect x="0" y="1.5" width="1464" height="24.65"/>
@@ -408,9 +408,12 @@
<clipPath id="breeze-shell-line-120">
<rect x="0" y="2929.5" width="1464" height="24.65"/>
</clipPath>
+<clipPath id="breeze-shell-line-121">
+ <rect x="0" y="2953.9" width="1464" height="24.65"/>
+ </clipPath>
</defs>
- <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="3024.8" rx="8"/><text class="breeze-shell-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: shell</text>
+ <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="3049.2" rx="8"/><text class="breeze-shell-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: shell</text>
<g transform="translate(26,22)">
<circle cx="0" cy="0" r="7" fill="#ff5f57"/>
<circle cx="22" cy="0" r="7" fill="#febc2e"/>
@@ -525,23 +528,24 @@
</text><text class="breeze-shell-r5" x="0" y="2508.8" textLength="24.4" clip-path="url(#breeze-shell-line-102)">╭─</text><text class="breeze-shell-r5" x="24.4" y="2508.8" textLength="500.2" clip-path="url(#breeze-shell-line-102)"> Upgrading/downgrading selected packages </text><text class="breeze-shell-r5" x="524.6" y="2508.8" textLength="915" clip-path="url(#breeze-shell-line-102)">───────────────────────────────────────────────────────────────────────────</text><tex [...]
</text><text class="breeze-shell-r5" x="0" y="2533.2" textLength="12.2" clip-path="url(#breeze-shell-line-103)">│</text><text class="breeze-shell-r4" x="24.4" y="2533.2" textLength="12.2" clip-path="url(#breeze-shell-line-103)">-</text><text class="breeze-shell-r4" x="36.6" y="2533.2" textLength="97.6" clip-path="url(#breeze-shell-line-103)">-upgrade</text><text class="breeze-shell-r4" x="134.2" y="2533.2" textLength="61" clip-path="url(#breeze-shell-line-103)">-boto</text><text class="b [...]
</text><text class="breeze-shell-r5" x="0" y="2557.6" textLength="12.2" clip-path="url(#breeze-shell-line-104)">│</text><text class="breeze-shell-r4" x="24.4" y="2557.6" textLength="12.2" clip-path="url(#breeze-shell-line-104)">-</text><text class="breeze-shell-r4" x="36.6" y="2557.6" textLength="122" clip-path="url(#breeze-shell-line-104)">-downgrade</text><text class="breeze-shell-r4" x="158.6" y="2557.6" textLength="134.2" clip-path="url(#breeze-shell-line-104)">-sqlalchemy</text><tex [...]
-</text><text class="breeze-shell-r5" x="0" y="2582" textLength="1464" clip-path="url(#breeze-shell-line-105)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-shell-r1" x="1464" y="2582" textLength="12.2" clip-path="url(#breeze-shell-line-105)">
-</text><text class="breeze-shell-r5" x="0" y="2606.4" textLength="24.4" clip-path="url(#breeze-shell-line-106)">╭─</text><text class="breeze-shell-r5" x="24.4" y="2606.4" textLength="183" clip-path="url(#breeze-shell-line-106)"> DB test flags </text><text class="breeze-shell-r5" x="207.4" y="2606.4" textLength="1232.2" clip-path="url(#breeze-shell-line-106)">─────────────────────────────────────────────────────────────────────────────────────────────────────</text><te [...]
-</text><text class="breeze-shell-r5" x="0" y="2630.8" textLength="12.2" clip-path="url(#breeze-shell-line-107)">│</text><text class="breeze-shell-r4" x="24.4" y="2630.8" textLength="12.2" clip-path="url(#breeze-shell-line-107)">-</text><text class="breeze-shell-r4" x="36.6" y="2630.8" textLength="48.8" clip-path="url(#breeze-shell-line-107)">-run</text><text class="breeze-shell-r4" x="85.4" y="2630.8" textLength="170.8" clip-path="url(#breeze-shell-line-107)">-db-tests-only</text><text c [...]
-</text><text class="breeze-shell-r5" x="0" y="2655.2" textLength="12.2" clip-path="url(#breeze-shell-line-108)">│</text><text class="breeze-shell-r4" x="24.4" y="2655.2" textLength="12.2" clip-path="url(#breeze-shell-line-108)">-</text><text class="breeze-shell-r4" x="36.6" y="2655.2" textLength="61" clip-path="url(#breeze-shell-line-108)">-skip</text><text class="breeze-shell-r4" x="97.6" y="2655.2" textLength="109.8" clip-path="url(#breeze-shell-line-108)">-db-tests</text><text class=" [...]
-</text><text class="breeze-shell-r5" x="0" y="2679.6" textLength="1464" clip-path="url(#breeze-shell-line-109)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-shell-r1" x="1464" y="2679.6" textLength="12.2" clip-path="url(#breeze-shell-line-109)">
-</text><text class="breeze-shell-r5" x="0" y="2704" textLength="24.4" clip-path="url(#breeze-shell-line-110)">╭─</text><text class="breeze-shell-r5" x="24.4" y="2704" textLength="183" clip-path="url(#breeze-shell-line-110)"> Other options </text><text class="breeze-shell-r5" x="207.4" y="2704" textLength="1232.2" clip-path="url(#breeze-shell-line-110)">─────────────────────────────────────────────────────────────────────────────────────────────────────</text><text class="b [...]
-</text><text class="breeze-shell-r5" x="0" y="2728.4" textLength="12.2" clip-path="url(#breeze-shell-line-111)">│</text><text class="breeze-shell-r4" x="24.4" y="2728.4" textLength="12.2" clip-path="url(#breeze-shell-line-111)">-</text><text class="breeze-shell-r4" x="36.6" y="2728.4" textLength="97.6" clip-path="url(#breeze-shell-line-111)">-forward</text><text class="breeze-shell-r4" x="134.2" y="2728.4" textLength="146.4" clip-path="url(#breeze-shell-line-111)">-credentials</text><tex [...]
-</text><text class="breeze-shell-r5" x="0" y="2752.8" textLength="12.2" clip-path="url(#breeze-shell-line-112)">│</text><text class="breeze-shell-r4" x="24.4" y="2752.8" textLength="12.2" clip-path="url(#breeze-shell-line-112)">-</text><text class="breeze-shell-r4" x="36.6" y="2752.8" textLength="48.8" clip-path="url(#breeze-shell-line-112)">-max</text><text class="breeze-shell-r4" x="85.4" y="2752.8" textLength="61" clip-path="url(#breeze-shell-line-112)">-time</text><text class="breeze [...]
-</text><text class="breeze-shell-r5" x="0" y="2777.2" textLength="12.2" clip-path="url(#breeze-shell-line-113)">│</text><text class="breeze-shell-r7" x="353.8" y="2777.2" textLength="1049.2" clip-path="url(#breeze-shell-line-113)">(INTEGER RANGE)                                        & [...]
-</text><text class="breeze-shell-r5" x="0" y="2801.6" textLength="12.2" clip-path="url(#breeze-shell-line-114)">│</text><text class="breeze-shell-r4" x="24.4" y="2801.6" textLength="12.2" clip-path="url(#breeze-shell-line-114)">-</text><text class="breeze-shell-r4" x="36.6" y="2801.6" textLength="97.6" clip-path="url(#breeze-shell-line-114)">-verbose</text><text class="breeze-shell-r4" x="134.2" y="2801.6" textLength="109.8" clip-path="url(#breeze-shell-line-114)">-commands</text><text c [...]
-</text><text class="breeze-shell-r5" x="0" y="2826" textLength="1464" clip-path="url(#breeze-shell-line-115)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-shell-r1" x="1464" y="2826" textLength="12.2" clip-path="url(#breeze-shell-line-115)">
-</text><text class="breeze-shell-r5" x="0" y="2850.4" textLength="24.4" clip-path="url(#breeze-shell-line-116)">╭─</text><text class="breeze-shell-r5" x="24.4" y="2850.4" textLength="195.2" clip-path="url(#breeze-shell-line-116)"> Common options </text><text class="breeze-shell-r5" x="219.6" y="2850.4" textLength="1220" clip-path="url(#breeze-shell-line-116)">────────────────────────────────────────────────────────────────────────────────────────────────────</text><text cl [...]
-</text><text class="breeze-shell-r5" x="0" y="2874.8" textLength="12.2" clip-path="url(#breeze-shell-line-117)">│</text><text class="breeze-shell-r4" x="24.4" y="2874.8" textLength="12.2" clip-path="url(#breeze-shell-line-117)">-</text><text class="breeze-shell-r4" x="36.6" y="2874.8" textLength="85.4" clip-path="url(#breeze-shell-line-117)">-answer</text><text class="breeze-shell-r6" x="158.6" y="2874.8" textLength="24.4" clip-path="url(#breeze-shell-line-117)">-a</text><text class="bre [...]
-</text><text class="breeze-shell-r5" x="0" y="2899.2" textLength="12.2" clip-path="url(#breeze-shell-line-118)">│</text><text class="breeze-shell-r4" x="24.4" y="2899.2" textLength="12.2" clip-path="url(#breeze-shell-line-118)">-</text><text class="breeze-shell-r4" x="36.6" y="2899.2" textLength="48.8" clip-path="url(#breeze-shell-line-118)">-dry</text><text class="breeze-shell-r4" x="85.4" y="2899.2" textLength="48.8" clip-path="url(#breeze-shell-line-118)">-run</text><text class="breez [...]
-</text><text class="breeze-shell-r5" x="0" y="2923.6" textLength="12.2" clip-path="url(#breeze-shell-line-119)">│</text><text class="breeze-shell-r4" x="24.4" y="2923.6" textLength="12.2" clip-path="url(#breeze-shell-line-119)">-</text><text class="breeze-shell-r4" x="36.6" y="2923.6" textLength="97.6" clip-path="url(#breeze-shell-line-119)">-verbose</text><text class="breeze-shell-r6" x="158.6" y="2923.6" textLength="24.4" clip-path="url(#breeze-shell-line-119)">-v</text><text class="br [...]
-</text><text class="breeze-shell-r5" x="0" y="2948" textLength="12.2" clip-path="url(#breeze-shell-line-120)">│</text><text class="breeze-shell-r4" x="24.4" y="2948" textLength="12.2" clip-path="url(#breeze-shell-line-120)">-</text><text class="breeze-shell-r4" x="36.6" y="2948" textLength="61" clip-path="url(#breeze-shell-line-120)">-help</text><text class="breeze-shell-r6" x="158.6" y="2948" textLength="24.4" clip-path="url(#breeze-shell-line-120)">-h</text><text class="breeze-shell-r1 [...]
-</text><text class="breeze-shell-r5" x="0" y="2972.4" textLength="1464" clip-path="url(#breeze-shell-line-121)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-shell-r1" x="1464" y="2972.4" textLength="12.2" clip-path="url(#breeze-shell-line-121)">
+</text><text class="breeze-shell-r5" x="0" y="2582" textLength="12.2" clip-path="url(#breeze-shell-line-105)">│</text><text class="breeze-shell-r4" x="24.4" y="2582" textLength="12.2" clip-path="url(#breeze-shell-line-105)">-</text><text class="breeze-shell-r4" x="36.6" y="2582" textLength="122" clip-path="url(#breeze-shell-line-105)">-downgrade</text><text class="breeze-shell-r4" x="158.6" y="2582" textLength="109.8" clip-path="url(#breeze-shell-line-105)">-pendulum</text><text class="b [...]
+</text><text class="breeze-shell-r5" x="0" y="2606.4" textLength="1464" clip-path="url(#breeze-shell-line-106)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-shell-r1" x="1464" y="2606.4" textLength="12.2" clip-path="url(#breeze-shell-line-106)">
+</text><text class="breeze-shell-r5" x="0" y="2630.8" textLength="24.4" clip-path="url(#breeze-shell-line-107)">╭─</text><text class="breeze-shell-r5" x="24.4" y="2630.8" textLength="183" clip-path="url(#breeze-shell-line-107)"> DB test flags </text><text class="breeze-shell-r5" x="207.4" y="2630.8" textLength="1232.2" clip-path="url(#breeze-shell-line-107)">─────────────────────────────────────────────────────────────────────────────────────────────────────</text><te [...]
+</text><text class="breeze-shell-r5" x="0" y="2655.2" textLength="12.2" clip-path="url(#breeze-shell-line-108)">│</text><text class="breeze-shell-r4" x="24.4" y="2655.2" textLength="12.2" clip-path="url(#breeze-shell-line-108)">-</text><text class="breeze-shell-r4" x="36.6" y="2655.2" textLength="48.8" clip-path="url(#breeze-shell-line-108)">-run</text><text class="breeze-shell-r4" x="85.4" y="2655.2" textLength="170.8" clip-path="url(#breeze-shell-line-108)">-db-tests-only</text><text c [...]
+</text><text class="breeze-shell-r5" x="0" y="2679.6" textLength="12.2" clip-path="url(#breeze-shell-line-109)">│</text><text class="breeze-shell-r4" x="24.4" y="2679.6" textLength="12.2" clip-path="url(#breeze-shell-line-109)">-</text><text class="breeze-shell-r4" x="36.6" y="2679.6" textLength="61" clip-path="url(#breeze-shell-line-109)">-skip</text><text class="breeze-shell-r4" x="97.6" y="2679.6" textLength="109.8" clip-path="url(#breeze-shell-line-109)">-db-tests</text><text class=" [...]
+</text><text class="breeze-shell-r5" x="0" y="2704" textLength="1464" clip-path="url(#breeze-shell-line-110)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-shell-r1" x="1464" y="2704" textLength="12.2" clip-path="url(#breeze-shell-line-110)">
+</text><text class="breeze-shell-r5" x="0" y="2728.4" textLength="24.4" clip-path="url(#breeze-shell-line-111)">╭─</text><text class="breeze-shell-r5" x="24.4" y="2728.4" textLength="183" clip-path="url(#breeze-shell-line-111)"> Other options </text><text class="breeze-shell-r5" x="207.4" y="2728.4" textLength="1232.2" clip-path="url(#breeze-shell-line-111)">─────────────────────────────────────────────────────────────────────────────────────────────────────</text><text cl [...]
+</text><text class="breeze-shell-r5" x="0" y="2752.8" textLength="12.2" clip-path="url(#breeze-shell-line-112)">│</text><text class="breeze-shell-r4" x="24.4" y="2752.8" textLength="12.2" clip-path="url(#breeze-shell-line-112)">-</text><text class="breeze-shell-r4" x="36.6" y="2752.8" textLength="97.6" clip-path="url(#breeze-shell-line-112)">-forward</text><text class="breeze-shell-r4" x="134.2" y="2752.8" textLength="146.4" clip-path="url(#breeze-shell-line-112)">-credentials</text><tex [...]
+</text><text class="breeze-shell-r5" x="0" y="2777.2" textLength="12.2" clip-path="url(#breeze-shell-line-113)">│</text><text class="breeze-shell-r4" x="24.4" y="2777.2" textLength="12.2" clip-path="url(#breeze-shell-line-113)">-</text><text class="breeze-shell-r4" x="36.6" y="2777.2" textLength="48.8" clip-path="url(#breeze-shell-line-113)">-max</text><text class="breeze-shell-r4" x="85.4" y="2777.2" textLength="61" clip-path="url(#breeze-shell-line-113)">-time</text><text class="breeze [...]
+</text><text class="breeze-shell-r5" x="0" y="2801.6" textLength="12.2" clip-path="url(#breeze-shell-line-114)">│</text><text class="breeze-shell-r7" x="353.8" y="2801.6" textLength="1049.2" clip-path="url(#breeze-shell-line-114)">(INTEGER RANGE)                                        & [...]
+</text><text class="breeze-shell-r5" x="0" y="2826" textLength="12.2" clip-path="url(#breeze-shell-line-115)">│</text><text class="breeze-shell-r4" x="24.4" y="2826" textLength="12.2" clip-path="url(#breeze-shell-line-115)">-</text><text class="breeze-shell-r4" x="36.6" y="2826" textLength="97.6" clip-path="url(#breeze-shell-line-115)">-verbose</text><text class="breeze-shell-r4" x="134.2" y="2826" textLength="109.8" clip-path="url(#breeze-shell-line-115)">-commands</text><text class="br [...]
+</text><text class="breeze-shell-r5" x="0" y="2850.4" textLength="1464" clip-path="url(#breeze-shell-line-116)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-shell-r1" x="1464" y="2850.4" textLength="12.2" clip-path="url(#breeze-shell-line-116)">
+</text><text class="breeze-shell-r5" x="0" y="2874.8" textLength="24.4" clip-path="url(#breeze-shell-line-117)">╭─</text><text class="breeze-shell-r5" x="24.4" y="2874.8" textLength="195.2" clip-path="url(#breeze-shell-line-117)"> Common options </text><text class="breeze-shell-r5" x="219.6" y="2874.8" textLength="1220" clip-path="url(#breeze-shell-line-117)">────────────────────────────────────────────────────────────────────────────────────────────────────</text><text cl [...]
+</text><text class="breeze-shell-r5" x="0" y="2899.2" textLength="12.2" clip-path="url(#breeze-shell-line-118)">│</text><text class="breeze-shell-r4" x="24.4" y="2899.2" textLength="12.2" clip-path="url(#breeze-shell-line-118)">-</text><text class="breeze-shell-r4" x="36.6" y="2899.2" textLength="85.4" clip-path="url(#breeze-shell-line-118)">-answer</text><text class="breeze-shell-r6" x="158.6" y="2899.2" textLength="24.4" clip-path="url(#breeze-shell-line-118)">-a</text><text class="bre [...]
+</text><text class="breeze-shell-r5" x="0" y="2923.6" textLength="12.2" clip-path="url(#breeze-shell-line-119)">│</text><text class="breeze-shell-r4" x="24.4" y="2923.6" textLength="12.2" clip-path="url(#breeze-shell-line-119)">-</text><text class="breeze-shell-r4" x="36.6" y="2923.6" textLength="48.8" clip-path="url(#breeze-shell-line-119)">-dry</text><text class="breeze-shell-r4" x="85.4" y="2923.6" textLength="48.8" clip-path="url(#breeze-shell-line-119)">-run</text><text class="breez [...]
+</text><text class="breeze-shell-r5" x="0" y="2948" textLength="12.2" clip-path="url(#breeze-shell-line-120)">│</text><text class="breeze-shell-r4" x="24.4" y="2948" textLength="12.2" clip-path="url(#breeze-shell-line-120)">-</text><text class="breeze-shell-r4" x="36.6" y="2948" textLength="97.6" clip-path="url(#breeze-shell-line-120)">-verbose</text><text class="breeze-shell-r6" x="158.6" y="2948" textLength="24.4" clip-path="url(#breeze-shell-line-120)">-v</text><text class="breeze-she [...]
+</text><text class="breeze-shell-r5" x="0" y="2972.4" textLength="12.2" clip-path="url(#breeze-shell-line-121)">│</text><text class="breeze-shell-r4" x="24.4" y="2972.4" textLength="12.2" clip-path="url(#breeze-shell-line-121)">-</text><text class="breeze-shell-r4" x="36.6" y="2972.4" textLength="61" clip-path="url(#breeze-shell-line-121)">-help</text><text class="breeze-shell-r6" x="158.6" y="2972.4" textLength="24.4" clip-path="url(#breeze-shell-line-121)">-h</text><text class="breeze- [...]
+</text><text class="breeze-shell-r5" x="0" y="2996.8" textLength="1464" clip-path="url(#breeze-shell-line-122)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-shell-r1" x="1464" y="2996.8" textLength="12.2" clip-path="url(#breeze-shell-line-122)">
</text>
</g>
</g>
diff --git a/images/breeze/output_shell.txt b/images/breeze/output_shell.txt
index e71507684a..0702a3041e 100644
--- a/images/breeze/output_shell.txt
+++ b/images/breeze/output_shell.txt
@@ -1 +1 @@
-0ceaf6dd335e09cc9ecb42cedd18c064
+35d2198ba2086d6f11da105483505fe1
diff --git a/images/breeze/output_testing_db-tests.svg b/images/breeze/output_testing_db-tests.svg
index a4c191f601..9a8c28fceb 100644
--- a/images/breeze/output_testing_db-tests.svg
+++ b/images/breeze/output_testing_db-tests.svg
@@ -1,4 +1,4 @@
-<svg class="rich-terminal" viewBox="0 0 1482 1831.1999999999998" xmlns="http://www.w3.org/2000/svg">
+<svg class="rich-terminal" viewBox="0 0 1482 1855.6" xmlns="http://www.w3.org/2000/svg">
<!-- Generated with Rich https://www.textualize.io -->
<style>
@@ -43,7 +43,7 @@
<defs>
<clipPath id="breeze-testing-db-tests-clip-terminal">
- <rect x="0" y="0" width="1463.0" height="1780.1999999999998" />
+ <rect x="0" y="0" width="1463.0" height="1804.6" />
</clipPath>
<clipPath id="breeze-testing-db-tests-line-0">
<rect x="0" y="1.5" width="1464" height="24.65"/>
@@ -261,9 +261,12 @@
<clipPath id="breeze-testing-db-tests-line-71">
<rect x="0" y="1733.9" width="1464" height="24.65"/>
</clipPath>
+<clipPath id="breeze-testing-db-tests-line-72">
+ <rect x="0" y="1758.3" width="1464" height="24.65"/>
+ </clipPath>
</defs>
- <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="1829.2" rx="8"/><text class="breeze-testing-db-tests-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: testing db-tests</text>
+ <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="1853.6" rx="8"/><text class="breeze-testing-db-tests-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: testing db-tests</text>
<g transform="translate(26,22)">
<circle cx="0" cy="0" r="7" fill="#ff5f57"/>
<circle cx="22" cy="0" r="7" fill="#febc2e"/>
@@ -338,14 +341,15 @@
</text><text class="breeze-testing-db-tests-r5" x="0" y="1532.8" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-62)">│</text><text class="breeze-testing-db-tests-r5" x="414.8" y="1532.8" textLength="1024.8" clip-path="url(#breeze-testing-db-tests-line-62)">[default: selected]                                 [...]
</text><text class="breeze-testing-db-tests-r5" x="0" y="1557.2" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-63)">│</text><text class="breeze-testing-db-tests-r4" x="24.4" y="1557.2" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-63)">-</text><text class="breeze-testing-db-tests-r4" x="36.6" y="1557.2" textLength="97.6" clip-path="url(#breeze-testing-db-tests-line-63)">-upgrade</text><text class="breeze-testing-db-tests-r4" x="134.2" y="1557.2" textLeng [...]
</text><text class="breeze-testing-db-tests-r5" x="0" y="1581.6" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-64)">│</text><text class="breeze-testing-db-tests-r4" x="24.4" y="1581.6" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-64)">-</text><text class="breeze-testing-db-tests-r4" x="36.6" y="1581.6" textLength="122" clip-path="url(#breeze-testing-db-tests-line-64)">-downgrade</text><text class="breeze-testing-db-tests-r4" x="158.6" y="1581.6" textLen [...]
-</text><text class="breeze-testing-db-tests-r5" x="0" y="1606" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-65)">│</text><text class="breeze-testing-db-tests-r4" x="24.4" y="1606" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-65)">-</text><text class="breeze-testing-db-tests-r4" x="36.6" y="1606" textLength="85.4" clip-path="url(#breeze-testing-db-tests-line-65)">-remove</text><text class="breeze-testing-db-tests-r4" x="122" y="1606" textLength="158.6" [...]
-</text><text class="breeze-testing-db-tests-r5" x="0" y="1630.4" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-66)">│</text><text class="breeze-testing-db-tests-r4" x="24.4" y="1630.4" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-66)">-</text><text class="breeze-testing-db-tests-r4" x="36.6" y="1630.4" textLength="61" clip-path="url(#breeze-testing-db-tests-line-66)">-skip</text><text class="breeze-testing-db-tests-r4" x="97.6" y="1630.4" textLength="24 [...]
-</text><text class="breeze-testing-db-tests-r5" x="0" y="1654.8" textLength="1464" clip-path="url(#breeze-testing-db-tests-line-67)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-testing-db-tests-r1" x="1464" y="1654.8" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-67)">
-</text><text class="breeze-testing-db-tests-r5" x="0" y="1679.2" textLength="24.4" clip-path="url(#breeze-testing-db-tests-line-68)">╭─</text><text class="breeze-testing-db-tests-r5" x="24.4" y="1679.2" textLength="195.2" clip-path="url(#breeze-testing-db-tests-line-68)"> Common options </text><text class="breeze-testing-db-tests-r5" x="219.6" y="1679.2" textLength="1220" clip-path="url(#breeze-testing-db-tests-line-68)">──────────────────────────────────────────────────── [...]
-</text><text class="breeze-testing-db-tests-r5" x="0" y="1703.6" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-69)">│</text><text class="breeze-testing-db-tests-r4" x="24.4" y="1703.6" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-69)">-</text><text class="breeze-testing-db-tests-r4" x="36.6" y="1703.6" textLength="97.6" clip-path="url(#breeze-testing-db-tests-line-69)">-verbose</text><text class="breeze-testing-db-tests-r7" x="158.6" y="1703.6" textLeng [...]
-</text><text class="breeze-testing-db-tests-r5" x="0" y="1728" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-70)">│</text><text class="breeze-testing-db-tests-r4" x="24.4" y="1728" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-70)">-</text><text class="breeze-testing-db-tests-r4" x="36.6" y="1728" textLength="48.8" clip-path="url(#breeze-testing-db-tests-line-70)">-dry</text><text class="breeze-testing-db-tests-r4" x="85.4" y="1728" textLength="48.8" cli [...]
-</text><text class="breeze-testing-db-tests-r5" x="0" y="1752.4" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-71)">│</text><text class="breeze-testing-db-tests-r4" x="24.4" y="1752.4" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-71)">-</text><text class="breeze-testing-db-tests-r4" x="36.6" y="1752.4" textLength="61" clip-path="url(#breeze-testing-db-tests-line-71)">-help</text><text class="breeze-testing-db-tests-r7" x="158.6" y="1752.4" textLength="2 [...]
-</text><text class="breeze-testing-db-tests-r5" x="0" y="1776.8" textLength="1464" clip-path="url(#breeze-testing-db-tests-line-72)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-testing-db-tests-r1" x="1464" y="1776.8" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-72)">
+</text><text class="breeze-testing-db-tests-r5" x="0" y="1606" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-65)">│</text><text class="breeze-testing-db-tests-r4" x="24.4" y="1606" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-65)">-</text><text class="breeze-testing-db-tests-r4" x="36.6" y="1606" textLength="122" clip-path="url(#breeze-testing-db-tests-line-65)">-downgrade</text><text class="breeze-testing-db-tests-r4" x="158.6" y="1606" textLength="109 [...]
+</text><text class="breeze-testing-db-tests-r5" x="0" y="1630.4" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-66)">│</text><text class="breeze-testing-db-tests-r4" x="24.4" y="1630.4" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-66)">-</text><text class="breeze-testing-db-tests-r4" x="36.6" y="1630.4" textLength="85.4" clip-path="url(#breeze-testing-db-tests-line-66)">-remove</text><text class="breeze-testing-db-tests-r4" x="122" y="1630.4" textLength= [...]
+</text><text class="breeze-testing-db-tests-r5" x="0" y="1654.8" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-67)">│</text><text class="breeze-testing-db-tests-r4" x="24.4" y="1654.8" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-67)">-</text><text class="breeze-testing-db-tests-r4" x="36.6" y="1654.8" textLength="61" clip-path="url(#breeze-testing-db-tests-line-67)">-skip</text><text class="breeze-testing-db-tests-r4" x="97.6" y="1654.8" textLength="24 [...]
+</text><text class="breeze-testing-db-tests-r5" x="0" y="1679.2" textLength="1464" clip-path="url(#breeze-testing-db-tests-line-68)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-testing-db-tests-r1" x="1464" y="1679.2" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-68)">
+</text><text class="breeze-testing-db-tests-r5" x="0" y="1703.6" textLength="24.4" clip-path="url(#breeze-testing-db-tests-line-69)">╭─</text><text class="breeze-testing-db-tests-r5" x="24.4" y="1703.6" textLength="195.2" clip-path="url(#breeze-testing-db-tests-line-69)"> Common options </text><text class="breeze-testing-db-tests-r5" x="219.6" y="1703.6" textLength="1220" clip-path="url(#breeze-testing-db-tests-line-69)">──────────────────────────────────────────────────── [...]
+</text><text class="breeze-testing-db-tests-r5" x="0" y="1728" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-70)">│</text><text class="breeze-testing-db-tests-r4" x="24.4" y="1728" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-70)">-</text><text class="breeze-testing-db-tests-r4" x="36.6" y="1728" textLength="97.6" clip-path="url(#breeze-testing-db-tests-line-70)">-verbose</text><text class="breeze-testing-db-tests-r7" x="158.6" y="1728" textLength="24.4 [...]
+</text><text class="breeze-testing-db-tests-r5" x="0" y="1752.4" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-71)">│</text><text class="breeze-testing-db-tests-r4" x="24.4" y="1752.4" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-71)">-</text><text class="breeze-testing-db-tests-r4" x="36.6" y="1752.4" textLength="48.8" clip-path="url(#breeze-testing-db-tests-line-71)">-dry</text><text class="breeze-testing-db-tests-r4" x="85.4" y="1752.4" textLength="4 [...]
+</text><text class="breeze-testing-db-tests-r5" x="0" y="1776.8" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-72)">│</text><text class="breeze-testing-db-tests-r4" x="24.4" y="1776.8" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-72)">-</text><text class="breeze-testing-db-tests-r4" x="36.6" y="1776.8" textLength="61" clip-path="url(#breeze-testing-db-tests-line-72)">-help</text><text class="breeze-testing-db-tests-r7" x="158.6" y="1776.8" textLength="2 [...]
+</text><text class="breeze-testing-db-tests-r5" x="0" y="1801.2" textLength="1464" clip-path="url(#breeze-testing-db-tests-line-73)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-testing-db-tests-r1" x="1464" y="1801.2" textLength="12.2" clip-path="url(#breeze-testing-db-tests-line-73)">
</text>
</g>
</g>
diff --git a/images/breeze/output_testing_db-tests.txt b/images/breeze/output_testing_db-tests.txt
index 24c7f336cf..092397ae05 100644
--- a/images/breeze/output_testing_db-tests.txt
+++ b/images/breeze/output_testing_db-tests.txt
@@ -1 +1 @@
-825d30b396fe55ffd0862c0d85441f36
+bbff9b2f8394bf13cb4916c15c0ba967
diff --git a/images/breeze/output_testing_non-db-tests.svg b/images/breeze/output_testing_non-db-tests.svg
index e7686cd36c..3d64e0989c 100644
--- a/images/breeze/output_testing_non-db-tests.svg
+++ b/images/breeze/output_testing_non-db-tests.svg
@@ -1,4 +1,4 @@
-<svg class="rich-terminal" viewBox="0 0 1482 1636.0" xmlns="http://www.w3.org/2000/svg">
+<svg class="rich-terminal" viewBox="0 0 1482 1660.3999999999999" xmlns="http://www.w3.org/2000/svg">
<!-- Generated with Rich https://www.textualize.io -->
<style>
@@ -43,7 +43,7 @@
<defs>
<clipPath id="breeze-testing-non-db-tests-clip-terminal">
- <rect x="0" y="0" width="1463.0" height="1585.0" />
+ <rect x="0" y="0" width="1463.0" height="1609.3999999999999" />
</clipPath>
<clipPath id="breeze-testing-non-db-tests-line-0">
<rect x="0" y="1.5" width="1464" height="24.65"/>
@@ -237,9 +237,12 @@
<clipPath id="breeze-testing-non-db-tests-line-63">
<rect x="0" y="1538.7" width="1464" height="24.65"/>
</clipPath>
+<clipPath id="breeze-testing-non-db-tests-line-64">
+ <rect x="0" y="1563.1" width="1464" height="24.65"/>
+ </clipPath>
</defs>
- <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="1634" rx="8"/><text class="breeze-testing-non-db-tests-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: testing non-db-tests</text>
+ <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="1658.4" rx="8"/><text class="breeze-testing-non-db-tests-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: testing non-db-tests</text>
<g transform="translate(26,22)">
<circle cx="0" cy="0" r="7" fill="#ff5f57"/>
<circle cx="22" cy="0" r="7" fill="#febc2e"/>
@@ -306,14 +309,15 @@
</text><text class="breeze-testing-non-db-tests-r5" x="0" y="1337.6" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-54)">│</text><text class="breeze-testing-non-db-tests-r5" x="414.8" y="1337.6" textLength="1024.8" clip-path="url(#breeze-testing-non-db-tests-line-54)">[default: selected]                               [...]
</text><text class="breeze-testing-non-db-tests-r5" x="0" y="1362" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-55)">│</text><text class="breeze-testing-non-db-tests-r4" x="24.4" y="1362" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-55)">-</text><text class="breeze-testing-non-db-tests-r4" x="36.6" y="1362" textLength="97.6" clip-path="url(#breeze-testing-non-db-tests-line-55)">-upgrade</text><text class="breeze-testing-non-db-tests-r4" x="134. [...]
</text><text class="breeze-testing-non-db-tests-r5" x="0" y="1386.4" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-56)">│</text><text class="breeze-testing-non-db-tests-r4" x="24.4" y="1386.4" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-56)">-</text><text class="breeze-testing-non-db-tests-r4" x="36.6" y="1386.4" textLength="122" clip-path="url(#breeze-testing-non-db-tests-line-56)">-downgrade</text><text class="breeze-testing-non-db-tests-r4" [...]
-</text><text class="breeze-testing-non-db-tests-r5" x="0" y="1410.8" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-57)">│</text><text class="breeze-testing-non-db-tests-r4" x="24.4" y="1410.8" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-57)">-</text><text class="breeze-testing-non-db-tests-r4" x="36.6" y="1410.8" textLength="85.4" clip-path="url(#breeze-testing-non-db-tests-line-57)">-remove</text><text class="breeze-testing-non-db-tests-r4" x= [...]
-</text><text class="breeze-testing-non-db-tests-r5" x="0" y="1435.2" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-58)">│</text><text class="breeze-testing-non-db-tests-r4" x="24.4" y="1435.2" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-58)">-</text><text class="breeze-testing-non-db-tests-r4" x="36.6" y="1435.2" textLength="61" clip-path="url(#breeze-testing-non-db-tests-line-58)">-skip</text><text class="breeze-testing-non-db-tests-r4" x="97. [...]
-</text><text class="breeze-testing-non-db-tests-r5" x="0" y="1459.6" textLength="1464" clip-path="url(#breeze-testing-non-db-tests-line-59)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-testing-non-db-tests-r1" x="1464" y="1459.6" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-59)">
-</text><text class="breeze-testing-non-db-tests-r5" x="0" y="1484" textLength="24.4" clip-path="url(#breeze-testing-non-db-tests-line-60)">╭─</text><text class="breeze-testing-non-db-tests-r5" x="24.4" y="1484" textLength="195.2" clip-path="url(#breeze-testing-non-db-tests-line-60)"> Common options </text><text class="breeze-testing-non-db-tests-r5" x="219.6" y="1484" textLength="1220" clip-path="url(#breeze-testing-non-db-tests-line-60)">────────────────────────────────── [...]
-</text><text class="breeze-testing-non-db-tests-r5" x="0" y="1508.4" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-61)">│</text><text class="breeze-testing-non-db-tests-r4" x="24.4" y="1508.4" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-61)">-</text><text class="breeze-testing-non-db-tests-r4" x="36.6" y="1508.4" textLength="48.8" clip-path="url(#breeze-testing-non-db-tests-line-61)">-dry</text><text class="breeze-testing-non-db-tests-r4" x="85 [...]
-</text><text class="breeze-testing-non-db-tests-r5" x="0" y="1532.8" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-62)">│</text><text class="breeze-testing-non-db-tests-r4" x="24.4" y="1532.8" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-62)">-</text><text class="breeze-testing-non-db-tests-r4" x="36.6" y="1532.8" textLength="97.6" clip-path="url(#breeze-testing-non-db-tests-line-62)">-verbose</text><text class="breeze-testing-non-db-tests-r7" x [...]
-</text><text class="breeze-testing-non-db-tests-r5" x="0" y="1557.2" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-63)">│</text><text class="breeze-testing-non-db-tests-r4" x="24.4" y="1557.2" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-63)">-</text><text class="breeze-testing-non-db-tests-r4" x="36.6" y="1557.2" textLength="61" clip-path="url(#breeze-testing-non-db-tests-line-63)">-help</text><text class="breeze-testing-non-db-tests-r7" x="158 [...]
-</text><text class="breeze-testing-non-db-tests-r5" x="0" y="1581.6" textLength="1464" clip-path="url(#breeze-testing-non-db-tests-line-64)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-testing-non-db-tests-r1" x="1464" y="1581.6" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-64)">
+</text><text class="breeze-testing-non-db-tests-r5" x="0" y="1410.8" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-57)">│</text><text class="breeze-testing-non-db-tests-r4" x="24.4" y="1410.8" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-57)">-</text><text class="breeze-testing-non-db-tests-r4" x="36.6" y="1410.8" textLength="122" clip-path="url(#breeze-testing-non-db-tests-line-57)">-downgrade</text><text class="breeze-testing-non-db-tests-r4" [...]
+</text><text class="breeze-testing-non-db-tests-r5" x="0" y="1435.2" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-58)">│</text><text class="breeze-testing-non-db-tests-r4" x="24.4" y="1435.2" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-58)">-</text><text class="breeze-testing-non-db-tests-r4" x="36.6" y="1435.2" textLength="85.4" clip-path="url(#breeze-testing-non-db-tests-line-58)">-remove</text><text class="breeze-testing-non-db-tests-r4" x= [...]
+</text><text class="breeze-testing-non-db-tests-r5" x="0" y="1459.6" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-59)">│</text><text class="breeze-testing-non-db-tests-r4" x="24.4" y="1459.6" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-59)">-</text><text class="breeze-testing-non-db-tests-r4" x="36.6" y="1459.6" textLength="61" clip-path="url(#breeze-testing-non-db-tests-line-59)">-skip</text><text class="breeze-testing-non-db-tests-r4" x="97. [...]
+</text><text class="breeze-testing-non-db-tests-r5" x="0" y="1484" textLength="1464" clip-path="url(#breeze-testing-non-db-tests-line-60)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-testing-non-db-tests-r1" x="1464" y="1484" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-60)">
+</text><text class="breeze-testing-non-db-tests-r5" x="0" y="1508.4" textLength="24.4" clip-path="url(#breeze-testing-non-db-tests-line-61)">╭─</text><text class="breeze-testing-non-db-tests-r5" x="24.4" y="1508.4" textLength="195.2" clip-path="url(#breeze-testing-non-db-tests-line-61)"> Common options </text><text class="breeze-testing-non-db-tests-r5" x="219.6" y="1508.4" textLength="1220" clip-path="url(#breeze-testing-non-db-tests-line-61)">──────────────────────────── [...]
+</text><text class="breeze-testing-non-db-tests-r5" x="0" y="1532.8" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-62)">│</text><text class="breeze-testing-non-db-tests-r4" x="24.4" y="1532.8" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-62)">-</text><text class="breeze-testing-non-db-tests-r4" x="36.6" y="1532.8" textLength="48.8" clip-path="url(#breeze-testing-non-db-tests-line-62)">-dry</text><text class="breeze-testing-non-db-tests-r4" x="85 [...]
+</text><text class="breeze-testing-non-db-tests-r5" x="0" y="1557.2" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-63)">│</text><text class="breeze-testing-non-db-tests-r4" x="24.4" y="1557.2" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-63)">-</text><text class="breeze-testing-non-db-tests-r4" x="36.6" y="1557.2" textLength="97.6" clip-path="url(#breeze-testing-non-db-tests-line-63)">-verbose</text><text class="breeze-testing-non-db-tests-r7" x [...]
+</text><text class="breeze-testing-non-db-tests-r5" x="0" y="1581.6" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-64)">│</text><text class="breeze-testing-non-db-tests-r4" x="24.4" y="1581.6" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-64)">-</text><text class="breeze-testing-non-db-tests-r4" x="36.6" y="1581.6" textLength="61" clip-path="url(#breeze-testing-non-db-tests-line-64)">-help</text><text class="breeze-testing-non-db-tests-r7" x="158 [...]
+</text><text class="breeze-testing-non-db-tests-r5" x="0" y="1606" textLength="1464" clip-path="url(#breeze-testing-non-db-tests-line-65)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-testing-non-db-tests-r1" x="1464" y="1606" textLength="12.2" clip-path="url(#breeze-testing-non-db-tests-line-65)">
</text>
</g>
</g>
diff --git a/images/breeze/output_testing_non-db-tests.txt b/images/breeze/output_testing_non-db-tests.txt
index 6bc1946e1a..7d9ef31069 100644
--- a/images/breeze/output_testing_non-db-tests.txt
+++ b/images/breeze/output_testing_non-db-tests.txt
@@ -1 +1 @@
-70b970c5c754371d9d5d80234d130615
+8aa7803c0bbff622175c01d94d22290b
diff --git a/images/breeze/output_testing_tests.svg b/images/breeze/output_testing_tests.svg
index 9e06eefda1..ed7c062406 100644
--- a/images/breeze/output_testing_tests.svg
+++ b/images/breeze/output_testing_tests.svg
@@ -1,4 +1,4 @@
-<svg class="rich-terminal" viewBox="0 0 1482 2246.0" xmlns="http://www.w3.org/2000/svg">
+<svg class="rich-terminal" viewBox="0 0 1482 2270.4" xmlns="http://www.w3.org/2000/svg">
<!-- Generated with Rich https://www.textualize.io -->
<style>
@@ -43,7 +43,7 @@
<defs>
<clipPath id="breeze-testing-tests-clip-terminal">
- <rect x="0" y="0" width="1463.0" height="2195.0" />
+ <rect x="0" y="0" width="1463.0" height="2219.4" />
</clipPath>
<clipPath id="breeze-testing-tests-line-0">
<rect x="0" y="1.5" width="1464" height="24.65"/>
@@ -312,9 +312,12 @@
<clipPath id="breeze-testing-tests-line-88">
<rect x="0" y="2148.7" width="1464" height="24.65"/>
</clipPath>
+<clipPath id="breeze-testing-tests-line-89">
+ <rect x="0" y="2173.1" width="1464" height="24.65"/>
+ </clipPath>
</defs>
- <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="2244" rx="8"/><text class="breeze-testing-tests-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: testing tests</text>
+ <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="2268.4" rx="8"/><text class="breeze-testing-tests-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: testing tests</text>
<g transform="translate(26,22)">
<circle cx="0" cy="0" r="7" fill="#ff5f57"/>
<circle cx="22" cy="0" r="7" fill="#febc2e"/>
@@ -406,14 +409,15 @@
</text><text class="breeze-testing-tests-r5" x="0" y="1947.6" textLength="12.2" clip-path="url(#breeze-testing-tests-line-79)">│</text><text class="breeze-testing-tests-r5" x="414.8" y="1947.6" textLength="1024.8" clip-path="url(#breeze-testing-tests-line-79)">[default: selected]                                   [...]
</text><text class="breeze-testing-tests-r5" x="0" y="1972" textLength="12.2" clip-path="url(#breeze-testing-tests-line-80)">│</text><text class="breeze-testing-tests-r4" x="24.4" y="1972" textLength="12.2" clip-path="url(#breeze-testing-tests-line-80)">-</text><text class="breeze-testing-tests-r4" x="36.6" y="1972" textLength="97.6" clip-path="url(#breeze-testing-tests-line-80)">-upgrade</text><text class="breeze-testing-tests-r4" x="134.2" y="1972" textLength="61" clip-path="url(#breez [...]
</text><text class="breeze-testing-tests-r5" x="0" y="1996.4" textLength="12.2" clip-path="url(#breeze-testing-tests-line-81)">│</text><text class="breeze-testing-tests-r4" x="24.4" y="1996.4" textLength="12.2" clip-path="url(#breeze-testing-tests-line-81)">-</text><text class="breeze-testing-tests-r4" x="36.6" y="1996.4" textLength="122" clip-path="url(#breeze-testing-tests-line-81)">-downgrade</text><text class="breeze-testing-tests-r4" x="158.6" y="1996.4" textLength="134.2" clip-path [...]
-</text><text class="breeze-testing-tests-r5" x="0" y="2020.8" textLength="12.2" clip-path="url(#breeze-testing-tests-line-82)">│</text><text class="breeze-testing-tests-r4" x="24.4" y="2020.8" textLength="12.2" clip-path="url(#breeze-testing-tests-line-82)">-</text><text class="breeze-testing-tests-r4" x="36.6" y="2020.8" textLength="85.4" clip-path="url(#breeze-testing-tests-line-82)">-remove</text><text class="breeze-testing-tests-r4" x="122" y="2020.8" textLength="158.6" clip-path="ur [...]
-</text><text class="breeze-testing-tests-r5" x="0" y="2045.2" textLength="12.2" clip-path="url(#breeze-testing-tests-line-83)">│</text><text class="breeze-testing-tests-r4" x="24.4" y="2045.2" textLength="12.2" clip-path="url(#breeze-testing-tests-line-83)">-</text><text class="breeze-testing-tests-r4" x="36.6" y="2045.2" textLength="61" clip-path="url(#breeze-testing-tests-line-83)">-skip</text><text class="breeze-testing-tests-r4" x="97.6" y="2045.2" textLength="244" clip-path="url(#br [...]
-</text><text class="breeze-testing-tests-r5" x="0" y="2069.6" textLength="1464" clip-path="url(#breeze-testing-tests-line-84)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-testing-tests-r1" x="1464" y="2069.6" textLength="12.2" clip-path="url(#breeze-testing-tests-line-84)">
-</text><text class="breeze-testing-tests-r5" x="0" y="2094" textLength="24.4" clip-path="url(#breeze-testing-tests-line-85)">╭─</text><text class="breeze-testing-tests-r5" x="24.4" y="2094" textLength="195.2" clip-path="url(#breeze-testing-tests-line-85)"> Common options </text><text class="breeze-testing-tests-r5" x="219.6" y="2094" textLength="1220" clip-path="url(#breeze-testing-tests-line-85)">──────────────────────────────────────────────────────────────────────────── [...]
-</text><text class="breeze-testing-tests-r5" x="0" y="2118.4" textLength="12.2" clip-path="url(#breeze-testing-tests-line-86)">│</text><text class="breeze-testing-tests-r4" x="24.4" y="2118.4" textLength="12.2" clip-path="url(#breeze-testing-tests-line-86)">-</text><text class="breeze-testing-tests-r4" x="36.6" y="2118.4" textLength="97.6" clip-path="url(#breeze-testing-tests-line-86)">-verbose</text><text class="breeze-testing-tests-r6" x="158.6" y="2118.4" textLength="24.4" clip-path=" [...]
-</text><text class="breeze-testing-tests-r5" x="0" y="2142.8" textLength="12.2" clip-path="url(#breeze-testing-tests-line-87)">│</text><text class="breeze-testing-tests-r4" x="24.4" y="2142.8" textLength="12.2" clip-path="url(#breeze-testing-tests-line-87)">-</text><text class="breeze-testing-tests-r4" x="36.6" y="2142.8" textLength="48.8" clip-path="url(#breeze-testing-tests-line-87)">-dry</text><text class="breeze-testing-tests-r4" x="85.4" y="2142.8" textLength="48.8" clip-path="url(# [...]
-</text><text class="breeze-testing-tests-r5" x="0" y="2167.2" textLength="12.2" clip-path="url(#breeze-testing-tests-line-88)">│</text><text class="breeze-testing-tests-r4" x="24.4" y="2167.2" textLength="12.2" clip-path="url(#breeze-testing-tests-line-88)">-</text><text class="breeze-testing-tests-r4" x="36.6" y="2167.2" textLength="61" clip-path="url(#breeze-testing-tests-line-88)">-help</text><text class="breeze-testing-tests-r6" x="158.6" y="2167.2" textLength="24.4" clip-path="url(# [...]
-</text><text class="breeze-testing-tests-r5" x="0" y="2191.6" textLength="1464" clip-path="url(#breeze-testing-tests-line-89)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-testing-tests-r1" x="1464" y="2191.6" textLength="12.2" clip-path="url(#breeze-testing-tests-line-89)">
+</text><text class="breeze-testing-tests-r5" x="0" y="2020.8" textLength="12.2" clip-path="url(#breeze-testing-tests-line-82)">│</text><text class="breeze-testing-tests-r4" x="24.4" y="2020.8" textLength="12.2" clip-path="url(#breeze-testing-tests-line-82)">-</text><text class="breeze-testing-tests-r4" x="36.6" y="2020.8" textLength="122" clip-path="url(#breeze-testing-tests-line-82)">-downgrade</text><text class="breeze-testing-tests-r4" x="158.6" y="2020.8" textLength="109.8" clip-path [...]
+</text><text class="breeze-testing-tests-r5" x="0" y="2045.2" textLength="12.2" clip-path="url(#breeze-testing-tests-line-83)">│</text><text class="breeze-testing-tests-r4" x="24.4" y="2045.2" textLength="12.2" clip-path="url(#breeze-testing-tests-line-83)">-</text><text class="breeze-testing-tests-r4" x="36.6" y="2045.2" textLength="85.4" clip-path="url(#breeze-testing-tests-line-83)">-remove</text><text class="breeze-testing-tests-r4" x="122" y="2045.2" textLength="158.6" clip-path="ur [...]
+</text><text class="breeze-testing-tests-r5" x="0" y="2069.6" textLength="12.2" clip-path="url(#breeze-testing-tests-line-84)">│</text><text class="breeze-testing-tests-r4" x="24.4" y="2069.6" textLength="12.2" clip-path="url(#breeze-testing-tests-line-84)">-</text><text class="breeze-testing-tests-r4" x="36.6" y="2069.6" textLength="61" clip-path="url(#breeze-testing-tests-line-84)">-skip</text><text class="breeze-testing-tests-r4" x="97.6" y="2069.6" textLength="244" clip-path="url(#br [...]
+</text><text class="breeze-testing-tests-r5" x="0" y="2094" textLength="1464" clip-path="url(#breeze-testing-tests-line-85)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-testing-tests-r1" x="1464" y="2094" textLength="12.2" clip-path="url(#breeze-testing-tests-line-85)">
+</text><text class="breeze-testing-tests-r5" x="0" y="2118.4" textLength="24.4" clip-path="url(#breeze-testing-tests-line-86)">╭─</text><text class="breeze-testing-tests-r5" x="24.4" y="2118.4" textLength="195.2" clip-path="url(#breeze-testing-tests-line-86)"> Common options </text><text class="breeze-testing-tests-r5" x="219.6" y="2118.4" textLength="1220" clip-path="url(#breeze-testing-tests-line-86)">────────────────────────────────────────────────────────────────────── [...]
+</text><text class="breeze-testing-tests-r5" x="0" y="2142.8" textLength="12.2" clip-path="url(#breeze-testing-tests-line-87)">│</text><text class="breeze-testing-tests-r4" x="24.4" y="2142.8" textLength="12.2" clip-path="url(#breeze-testing-tests-line-87)">-</text><text class="breeze-testing-tests-r4" x="36.6" y="2142.8" textLength="97.6" clip-path="url(#breeze-testing-tests-line-87)">-verbose</text><text class="breeze-testing-tests-r6" x="158.6" y="2142.8" textLength="24.4" clip-path=" [...]
+</text><text class="breeze-testing-tests-r5" x="0" y="2167.2" textLength="12.2" clip-path="url(#breeze-testing-tests-line-88)">│</text><text class="breeze-testing-tests-r4" x="24.4" y="2167.2" textLength="12.2" clip-path="url(#breeze-testing-tests-line-88)">-</text><text class="breeze-testing-tests-r4" x="36.6" y="2167.2" textLength="48.8" clip-path="url(#breeze-testing-tests-line-88)">-dry</text><text class="breeze-testing-tests-r4" x="85.4" y="2167.2" textLength="48.8" clip-path="url(# [...]
+</text><text class="breeze-testing-tests-r5" x="0" y="2191.6" textLength="12.2" clip-path="url(#breeze-testing-tests-line-89)">│</text><text class="breeze-testing-tests-r4" x="24.4" y="2191.6" textLength="12.2" clip-path="url(#breeze-testing-tests-line-89)">-</text><text class="breeze-testing-tests-r4" x="36.6" y="2191.6" textLength="61" clip-path="url(#breeze-testing-tests-line-89)">-help</text><text class="breeze-testing-tests-r6" x="158.6" y="2191.6" textLength="24.4" clip-path="url(# [...]
+</text><text class="breeze-testing-tests-r5" x="0" y="2216" textLength="1464" clip-path="url(#breeze-testing-tests-line-90)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-testing-tests-r1" x="1464" y="2216" textLength="12.2" clip-path="url(#breeze-testing-tests-line-90)">
</text>
</g>
</g>
diff --git a/images/breeze/output_testing_tests.txt b/images/breeze/output_testing_tests.txt
index b243afade7..6b81ec51d0 100644
--- a/images/breeze/output_testing_tests.txt
+++ b/images/breeze/output_testing_tests.txt
@@ -1 +1 @@
-4c38b2815f6a7606d05fe58fd9a8550a
+f63db76413092aa6f822957edbf3fc8c
diff --git a/kubernetes_tests/test_kubernetes_pod_operator.py b/kubernetes_tests/test_kubernetes_pod_operator.py
index 249cf667e1..8d7dad9d12 100644
--- a/kubernetes_tests/test_kubernetes_pod_operator.py
+++ b/kubernetes_tests/test_kubernetes_pod_operator.py
@@ -26,7 +26,6 @@ from unittest import mock
from unittest.mock import ANY, MagicMock
from uuid import uuid4
-import pendulum
import pytest
from kubernetes import client
from kubernetes.client import V1EnvVar, V1PodSecurityContext, V1SecurityContext, models as k8s
@@ -53,7 +52,9 @@ POD_MANAGER_CLASS = "airflow.providers.cncf.kubernetes.utils.pod_manager.PodMana
def create_context(task) -> Context:
dag = DAG(dag_id="dag")
- execution_date = timezone.datetime(2016, 1, 1, 1, 0, 0, tzinfo=pendulum.tz.timezone("Europe/Amsterdam"))
+ execution_date = timezone.datetime(
+ 2016, 1, 1, 1, 0, 0, tzinfo=timezone.parse_timezone("Europe/Amsterdam")
+ )
dag_run = DagRun(
dag_id=dag.dag_id,
execution_date=execution_date,
diff --git a/newsfragments/36281.significant.rst b/newsfragments/36281.significant.rst
new file mode 100644
index 0000000000..1207be0782
--- /dev/null
+++ b/newsfragments/36281.significant.rst
@@ -0,0 +1,4 @@
+Target version for core dependency ``pendulum`` package set to 3
+
+Support for pendulum 2.1.2 will be saved for a while, presumably until the next feature version of Airflow.
+It is advised to upgrade user code to use pendulum 3 as soon as possible.
diff --git a/pyproject.toml b/pyproject.toml
index b1c40d4b40..be12fcba93 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -123,9 +123,7 @@ dependencies = [
"opentelemetry-exporter-otlp",
"packaging>=14.0",
"pathspec>=0.9.0",
- # When (if) pendulum 3 released it would introduce changes in module/objects imports,
- # since we are tightly coupled with pendulum library internally it will breaks Airflow functionality.
- "pendulum>=2.0,<3.0",
+ "pendulum>=2.1.2,<4.0",
"pluggy>=1.0",
"psutil>=4.2.0",
"pydantic>=2.3.0",
diff --git a/scripts/ci/docker-compose/devcontainer.env b/scripts/ci/docker-compose/devcontainer.env
index 4ef3fd0459..5c627c9ebf 100644
--- a/scripts/ci/docker-compose/devcontainer.env
+++ b/scripts/ci/docker-compose/devcontainer.env
@@ -68,6 +68,7 @@ START_AIRFLOW="false"
SUSPENDED_PROVIDERS_FOLDERS=""
TEST_TYPE=
UPGRADE_BOTO="false"
+DOWNGRADE_PENDULUM="false"
DOWNGRADE_SQLALCHEMY="false"
UPGRADE_TO_NEWER_DEPENDENCIES="false"
VERBOSE="false"
diff --git a/scripts/docker/entrypoint_ci.sh b/scripts/docker/entrypoint_ci.sh
index 52d79f4154..b5bbe777ac 100755
--- a/scripts/docker/entrypoint_ci.sh
+++ b/scripts/docker/entrypoint_ci.sh
@@ -239,6 +239,19 @@ function check_download_sqlalchemy() {
pip check
}
+# Download minimum supported version of pendulum to run tests with it
+function check_download_pendulum() {
+ if [[ ${DOWNGRADE_PENDULUM=} != "true" ]]; then
+ return
+ fi
+ min_pendulum_version=$(grep "\"pendulum>=" pyproject.toml | sed "s/.*>=\([0-9\.]*\).*/\1/" | xargs)
+ echo
+ echo "${COLOR_BLUE}Downgrading pendulum to minimum supported version: ${min_pendulum_version}${COLOR_RESET}"
+ echo
+ pip install --root-user-action ignore "pendulum==${min_pendulum_version}"
+ pip check
+}
+
# Check if we should run tests and run them if needed
function check_run_tests() {
if [[ ${RUN_TESTS=} != "true" ]]; then
@@ -269,6 +282,7 @@ determine_airflow_to_use
environment_initialization
check_boto_upgrade
check_download_sqlalchemy
+check_download_pendulum
check_run_tests "${@}"
# If we are not running tests - just exec to bash shell
diff --git a/tests/api_connexion/endpoints/test_dag_endpoint.py b/tests/api_connexion/endpoints/test_dag_endpoint.py
index c02e8b0ff3..a29b86f4a6 100644
--- a/tests/api_connexion/endpoints/test_dag_endpoint.py
+++ b/tests/api_connexion/endpoints/test_dag_endpoint.py
@@ -20,6 +20,7 @@ import os
import unittest.mock
from datetime import datetime
+import pendulum
import pytest
from airflow.api_connexion.exceptions import EXCEPTIONS_LINK_MAP
@@ -46,6 +47,7 @@ DAG_ID = "test_dag"
TASK_ID = "op1"
DAG2_ID = "test_dag2"
DAG3_ID = "test_dag3"
+UTC_JSON_REPR = "UTC" if pendulum.__version__.startswith("3") else "Timezone('UTC')"
@pytest.fixture(scope="module")
@@ -316,7 +318,7 @@ class TestGetDagDetails(TestDagEndpoint):
"tags": [],
"template_searchpath": None,
"timetable_description": None,
- "timezone": "Timezone('UTC')",
+ "timezone": UTC_JSON_REPR,
}
assert response.json == expected
@@ -367,7 +369,7 @@ class TestGetDagDetails(TestDagEndpoint):
"tags": [],
"template_searchpath": None,
"timetable_description": None,
- "timezone": "Timezone('UTC')",
+ "timezone": UTC_JSON_REPR,
}
assert response.json == expected
@@ -418,7 +420,7 @@ class TestGetDagDetails(TestDagEndpoint):
"tags": [],
"template_searchpath": None,
"timetable_description": None,
- "timezone": "Timezone('UTC')",
+ "timezone": UTC_JSON_REPR,
}
assert response.json == expected
@@ -478,7 +480,7 @@ class TestGetDagDetails(TestDagEndpoint):
"tags": [],
"template_searchpath": None,
"timetable_description": None,
- "timezone": "Timezone('UTC')",
+ "timezone": UTC_JSON_REPR,
}
response = self.client.get(
f"/api/v1/dags/{self.dag_id}/details", environ_overrides={"REMOTE_USER": "test"}
@@ -539,7 +541,7 @@ class TestGetDagDetails(TestDagEndpoint):
"tags": [],
"template_searchpath": None,
"timetable_description": None,
- "timezone": "Timezone('UTC')",
+ "timezone": UTC_JSON_REPR,
}
expected.update({"last_parsed": response.json["last_parsed"]})
assert response.json == expected
diff --git a/tests/api_connexion/schemas/test_dag_schema.py b/tests/api_connexion/schemas/test_dag_schema.py
index f3e54c0a96..df227eb5c6 100644
--- a/tests/api_connexion/schemas/test_dag_schema.py
+++ b/tests/api_connexion/schemas/test_dag_schema.py
@@ -18,6 +18,7 @@ from __future__ import annotations
from datetime import datetime
+import pendulum
import pytest
from airflow.api_connexion.schemas.dag_schema import (
@@ -29,6 +30,8 @@ from airflow.api_connexion.schemas.dag_schema import (
from airflow.models import DagModel, DagTag
from airflow.models.dag import DAG
+UTC_JSON_REPR = "UTC" if pendulum.__version__.startswith("3") else "Timezone('UTC')"
+
def test_serialize_test_dag_schema(url_safe_serializer):
dag_model = DagModel(
@@ -184,7 +187,7 @@ def test_serialize_test_dag_detail_schema(url_safe_serializer):
"start_date": "2020-06-19T00:00:00+00:00",
"tags": [{"name": "example1"}, {"name": "example2"}],
"template_searchpath": None,
- "timezone": "Timezone('UTC')",
+ "timezone": UTC_JSON_REPR,
"max_active_runs": 16,
"pickle_id": None,
"end_date": None,
diff --git a/tests/cli/commands/test_dag_command.py b/tests/cli/commands/test_dag_command.py
index 4f16c381ad..1c1a7ff650 100644
--- a/tests/cli/commands/test_dag_command.py
+++ b/tests/cli/commands/test_dag_command.py
@@ -50,6 +50,10 @@ from tests.test_utils.config import conf_vars
from tests.test_utils.db import clear_db_dags, clear_db_runs
DEFAULT_DATE = timezone.make_aware(datetime(2015, 1, 1), timezone=timezone.utc)
+if pendulum.__version__.startswith("3"):
+ DEFAULT_DATE_REPR = DEFAULT_DATE.isoformat(sep=" ")
+else:
+ DEFAULT_DATE_REPR = DEFAULT_DATE.isoformat()
# TODO: Check if tests needs side effects - locally there's missing DAG
@@ -162,7 +166,7 @@ class TestCliDags:
)
output = stdout.getvalue()
- assert f"Dry run of DAG example_bash_operator on {DEFAULT_DATE.isoformat()}\n" in output
+ assert f"Dry run of DAG example_bash_operator on {DEFAULT_DATE_REPR}\n" in output
assert "Task runme_0 located in DAG example_bash_operator\n" in output
mock_run.assert_not_called() # Dry run shouldn't run the backfill
@@ -235,12 +239,9 @@ class TestCliDags:
output = stdout.getvalue()
- assert (
- f"Dry run of DAG example_branch_python_operator_decorator on "
- f"{DEFAULT_DATE.isoformat()}\n" in output
- )
+ assert f"Dry run of DAG example_branch_python_operator_decorator on {DEFAULT_DATE_REPR}\n" in output
assert "Task run_this_first located in DAG example_branch_python_operator_decorator\n" in output
- assert f"Dry run of DAG example_branch_operator on {DEFAULT_DATE.isoformat()}\n" in output
+ assert f"Dry run of DAG example_branch_operator on {DEFAULT_DATE_REPR}\n" in output
assert "Task run_this_first located in DAG example_branch_operator\n" in output
@mock.patch("airflow.cli.commands.dag_command.get_dag")
diff --git a/tests/models/test_dag.py b/tests/models/test_dag.py
index f7bf1ad6d0..de7fa87d5a 100644
--- a/tests/models/test_dag.py
+++ b/tests/models/test_dag.py
@@ -37,6 +37,7 @@ import pendulum
import pytest
import time_machine
from dateutil.relativedelta import relativedelta
+from pendulum.tz.timezone import Timezone
from sqlalchemy import inspect
from airflow import settings
@@ -676,8 +677,8 @@ class TestDag:
"""
Make sure DST transitions are properly observed
"""
- local_tz = pendulum.timezone("Europe/Zurich")
- start = local_tz.convert(datetime.datetime(2018, 10, 28, 2, 55), dst_rule=pendulum.PRE_TRANSITION)
+ local_tz = Timezone("Europe/Zurich")
+ start = local_tz.convert(datetime.datetime(2018, 10, 28, 2, 55, fold=0))
assert start.isoformat() == "2018-10-28T02:55:00+02:00", "Pre-condition: start date is in DST"
utc = timezone.convert_to_utc(start)
@@ -706,7 +707,7 @@ class TestDag:
Make sure DST transitions are properly observed
"""
local_tz = pendulum.timezone("Europe/Zurich")
- start = local_tz.convert(datetime.datetime(2018, 10, 27, 3), dst_rule=pendulum.PRE_TRANSITION)
+ start = local_tz.convert(datetime.datetime(2018, 10, 27, 3, fold=0))
utc = timezone.convert_to_utc(start)
@@ -735,7 +736,7 @@ class TestDag:
Make sure DST transitions are properly observed
"""
local_tz = pendulum.timezone("Europe/Zurich")
- start = local_tz.convert(datetime.datetime(2018, 3, 25, 2), dst_rule=pendulum.PRE_TRANSITION)
+ start = local_tz.convert(datetime.datetime(2018, 3, 25, 2, fold=0))
utc = timezone.convert_to_utc(start)
diff --git a/tests/providers/openlineage/plugins/test_utils.py b/tests/providers/openlineage/plugins/test_utils.py
index 54710bcd9e..b7ced7a37c 100644
--- a/tests/providers/openlineage/plugins/test_utils.py
+++ b/tests/providers/openlineage/plugins/test_utils.py
@@ -23,7 +23,6 @@ import uuid
from json import JSONEncoder
from typing import Any
-import pendulum
import pytest
from attrs import define
from openlineage.client.utils import RedactMixin
@@ -39,6 +38,7 @@ from airflow.providers.openlineage.utils.utils import (
to_json_encodable,
url_to_https,
)
+from airflow.utils import timezone
from airflow.utils.log.secrets_masker import _secrets_masker
from airflow.utils.state import State
@@ -86,8 +86,8 @@ def test_get_dagrun_start_end():
state=State.NONE, run_id=run_id, data_interval=dag.get_next_data_interval(dag_model)
)
assert dagrun.data_interval_start is not None
- start_date_tz = datetime.datetime(2022, 1, 1, tzinfo=pendulum.tz.timezone("UTC"))
- end_date_tz = datetime.datetime(2022, 1, 1, hour=2, tzinfo=pendulum.tz.timezone("UTC"))
+ start_date_tz = datetime.datetime(2022, 1, 1, tzinfo=timezone.utc)
+ end_date_tz = datetime.datetime(2022, 1, 1, hour=2, tzinfo=timezone.utc)
assert dagrun.data_interval_start, dagrun.data_interval_end == (start_date_tz, end_date_tz)
diff --git a/tests/sensors/test_time_sensor.py b/tests/sensors/test_time_sensor.py
index 935d1cb128..54a0212a24 100644
--- a/tests/sensors/test_time_sensor.py
+++ b/tests/sensors/test_time_sensor.py
@@ -18,12 +18,10 @@
from __future__ import annotations
from datetime import datetime, time
-from unittest.mock import patch
import pendulum
import pytest
import time_machine
-from pendulum.tz.timezone import UTC
from airflow.exceptions import TaskDeferred
from airflow.models.dag import DAG
@@ -33,7 +31,7 @@ from airflow.utils import timezone
DEFAULT_TIMEZONE = "Asia/Singapore" # UTC+08:00
DEFAULT_DATE_WO_TZ = datetime(2015, 1, 1)
-DEFAULT_DATE_WITH_TZ = datetime(2015, 1, 1, tzinfo=pendulum.tz.timezone(DEFAULT_TIMEZONE))
+DEFAULT_DATE_WITH_TZ = datetime(2015, 1, 1, tzinfo=timezone.parse_timezone(DEFAULT_TIMEZONE))
class TestTimeSensor:
@@ -46,11 +44,11 @@ class TestTimeSensor:
],
)
@time_machine.travel(timezone.datetime(2020, 1, 1, 23, 0).replace(tzinfo=timezone.utc))
- def test_timezone(self, default_timezone, start_date, expected):
- with patch("airflow.settings.TIMEZONE", pendulum.timezone(default_timezone)):
- dag = DAG("test", default_args={"start_date": start_date})
- op = TimeSensor(task_id="test", target_time=time(10, 0), dag=dag)
- assert op.poke(None) == expected
+ def test_timezone(self, default_timezone, start_date, expected, monkeypatch):
+ monkeypatch.setattr("airflow.settings.TIMEZONE", timezone.parse_timezone(default_timezone))
+ dag = DAG("test", default_args={"start_date": start_date})
+ op = TimeSensor(task_id="test", target_time=time(10, 0), dag=dag)
+ assert op.poke(None) == expected
class TestTimeSensorAsync:
@@ -72,8 +70,7 @@ class TestTimeSensorAsync:
with DAG("test_target_time_aware", start_date=timezone.datetime(2020, 1, 1, 23, 0)):
aware_time = time(0, 1).replace(tzinfo=pendulum.local_timezone())
op = TimeSensorAsync(task_id="test", target_time=aware_time)
- assert hasattr(op.target_datetime.tzinfo, "offset")
- assert op.target_datetime.tzinfo.offset == 0
+ assert op.target_datetime.tzinfo == timezone.utc
def test_target_time_naive_dag_timezone(self):
"""
@@ -85,4 +82,4 @@ class TestTimeSensorAsync:
):
op = TimeSensorAsync(task_id="test", target_time=pendulum.time(9, 0))
assert op.target_datetime.time() == pendulum.time(1, 0)
- assert op.target_datetime.tzinfo == UTC
+ assert op.target_datetime.tzinfo == timezone.utc
diff --git a/tests/serialization/serializers/test_serializers.py b/tests/serialization/serializers/test_serializers.py
index 32e9787ccf..26027fdbf0 100644
--- a/tests/serialization/serializers/test_serializers.py
+++ b/tests/serialization/serializers/test_serializers.py
@@ -21,11 +21,13 @@ import decimal
from unittest.mock import patch
import numpy as np
+import pendulum
import pendulum.tz
import pytest
from dateutil.tz import tzutc
from deltalake import DeltaTable
from pendulum import DateTime
+from pendulum.tz.timezone import FixedTimezone, Timezone
from pyiceberg.catalog import Catalog
from pyiceberg.io import FileIO
from pyiceberg.table import Table
@@ -39,6 +41,8 @@ if PY39:
else:
from backports.zoneinfo import ZoneInfo
+PENDULUM3 = pendulum.__version__.startswith("3")
+
class TestSerializers:
def test_datetime(self):
@@ -227,3 +231,151 @@ class TestSerializers:
assert i.version() == d.version()
assert i._storage_options == d._storage_options
assert d._storage_options is None
+
+ @pytest.mark.skipif(not PENDULUM3, reason="Test case for pendulum~=3")
+ @pytest.mark.parametrize(
+ "ser_value, expected",
+ [
+ pytest.param(
+ {
+ "__classname__": "pendulum.datetime.DateTime",
+ "__version__": 2,
+ "__data__": {
+ "timestamp": 1680307200.0,
+ "tz": {
+ "__classname__": "builtins.tuple",
+ "__version__": 1,
+ "__data__": ["UTC", "pendulum.tz.timezone.FixedTimezone", 1, True],
+ },
+ },
+ },
+ pendulum.datetime(2023, 4, 1, tz=Timezone("UTC")),
+ id="in-utc-timezone",
+ ),
+ pytest.param(
+ {
+ "__classname__": "pendulum.datetime.DateTime",
+ "__version__": 2,
+ "__data__": {
+ "timestamp": 1680292800.0,
+ "tz": {
+ "__classname__": "builtins.tuple",
+ "__version__": 1,
+ "__data__": ["Asia/Tbilisi", "pendulum.tz.timezone.Timezone", 1, True],
+ },
+ },
+ },
+ pendulum.datetime(2023, 4, 1, tz=Timezone("Asia/Tbilisi")),
+ id="non-dts-timezone",
+ ),
+ pytest.param(
+ {
+ "__classname__": "pendulum.datetime.DateTime",
+ "__version__": 2,
+ "__data__": {
+ "timestamp": 1680303600.0,
+ "tz": {
+ "__classname__": "builtins.tuple",
+ "__version__": 1,
+ "__data__": ["Europe/London", "pendulum.tz.timezone.Timezone", 1, True],
+ },
+ },
+ },
+ pendulum.datetime(2023, 4, 1, tz=Timezone("Europe/London")),
+ id="dts-timezone",
+ ),
+ pytest.param(
+ {
+ "__classname__": "pendulum.datetime.DateTime",
+ "__version__": 2,
+ "__data__": {
+ "timestamp": 1680310800.0,
+ "tz": {
+ "__classname__": "builtins.tuple",
+ "__version__": 1,
+ "__data__": [-3600, "pendulum.tz.timezone.FixedTimezone", 1, True],
+ },
+ },
+ },
+ pendulum.datetime(2023, 4, 1, tz=FixedTimezone(-3600)),
+ id="offset-timezone",
+ ),
+ ],
+ )
+ def test_pendulum_2_to_3(self, ser_value, expected):
+ """Test deserialize objects in pendulum 3 which serialised in pendulum 2."""
+ assert deserialize(ser_value) == expected
+
+ @pytest.mark.skipif(PENDULUM3, reason="Test case for pendulum~=2")
+ @pytest.mark.parametrize(
+ "ser_value, expected",
+ [
+ pytest.param(
+ {
+ "__classname__": "pendulum.datetime.DateTime",
+ "__version__": 2,
+ "__data__": {
+ "timestamp": 1680307200.0,
+ "tz": {
+ "__classname__": "builtins.tuple",
+ "__version__": 1,
+ "__data__": ["UTC", "pendulum.tz.timezone.Timezone", 1, True],
+ },
+ },
+ },
+ pendulum.datetime(2023, 4, 1, tz=Timezone("UTC")),
+ id="in-utc-timezone",
+ ),
+ pytest.param(
+ {
+ "__classname__": "pendulum.datetime.DateTime",
+ "__version__": 2,
+ "__data__": {
+ "timestamp": 1680292800.0,
+ "tz": {
+ "__classname__": "builtins.tuple",
+ "__version__": 1,
+ "__data__": ["Asia/Tbilisi", "pendulum.tz.timezone.Timezone", 1, True],
+ },
+ },
+ },
+ pendulum.datetime(2023, 4, 1, tz=Timezone("Asia/Tbilisi")),
+ id="non-dts-timezone",
+ ),
+ pytest.param(
+ {
+ "__classname__": "pendulum.datetime.DateTime",
+ "__version__": 2,
+ "__data__": {
+ "timestamp": 1680303600.0,
+ "tz": {
+ "__classname__": "builtins.tuple",
+ "__version__": 1,
+ "__data__": ["Europe/London", "pendulum.tz.timezone.Timezone", 1, True],
+ },
+ },
+ },
+ pendulum.datetime(2023, 4, 1, tz=Timezone("Europe/London")),
+ id="dts-timezone",
+ ),
+ pytest.param(
+ {
+ "__classname__": "pendulum.datetime.DateTime",
+ "__version__": 2,
+ "__data__": {
+ "timestamp": 1680310800.0,
+ "tz": {
+ "__classname__": "builtins.tuple",
+ "__version__": 1,
+ "__data__": [-3600, "pendulum.tz.timezone.FixedTimezone", 1, True],
+ },
+ },
+ },
+ pendulum.datetime(2023, 4, 1, tz=FixedTimezone(-3600)),
+ id="offset-timezone",
+ ),
+ ],
+ )
+ def test_pendulum_3_to_2(self, ser_value, expected):
+ """Test deserialize objects in pendulum 2 which serialised in pendulum 3."""
+ assert deserialize(ser_value) == expected
diff --git a/tests/serialization/test_serialized_objects.py b/tests/serialization/test_serialized_objects.py
index c059a8d236..a40e0d01ea 100644
--- a/tests/serialization/test_serialized_objects.py
+++ b/tests/serialization/test_serialized_objects.py
@@ -20,10 +20,10 @@ from __future__ import annotations
import json
from datetime import datetime, timedelta
-import pendulum
import pytest
from dateutil import relativedelta
from kubernetes.client import models as k8s
+from pendulum.tz.timezone import Timezone
from airflow.datasets import Dataset
from airflow.exceptions import SerializationError
@@ -142,7 +142,7 @@ def equal_time(a: datetime, b: datetime) -> bool:
(1, None, equals),
(datetime.utcnow(), DAT.DATETIME, equal_time),
(timedelta(minutes=2), DAT.TIMEDELTA, equals),
- (pendulum.tz.timezone("UTC"), DAT.TIMEZONE, lambda a, b: a.name == b.name),
+ (Timezone("UTC"), DAT.TIMEZONE, lambda a, b: a.name == b.name),
(relativedelta.relativedelta(hours=+1), DAT.RELATIVEDELTA, lambda a, b: a.hours == b.hours),
({"test": "dict", "test-1": 1}, None, equals),
(["array_item", 2], None, equals),
diff --git a/tests/triggers/test_temporal.py b/tests/triggers/test_temporal.py
index 655910394f..52cc2c64f6 100644
--- a/tests/triggers/test_temporal.py
+++ b/tests/triggers/test_temporal.py
@@ -64,9 +64,9 @@ def test_timedelta_trigger_serialization():
@pytest.mark.parametrize(
"tz",
[
- pendulum.tz.timezone("UTC"),
- pendulum.tz.timezone("Europe/Paris"),
- pendulum.tz.timezone("America/Toronto"),
+ timezone.parse_timezone("UTC"),
+ timezone.parse_timezone("Europe/Paris"),
+ timezone.parse_timezone("America/Toronto"),
],
)
@pytest.mark.asyncio
diff --git a/tests/utils/test_timezone.py b/tests/utils/test_timezone.py
index ff5ad26f5a..df8af04604 100644
--- a/tests/utils/test_timezone.py
+++ b/tests/utils/test_timezone.py
@@ -21,13 +21,14 @@ import datetime
import pendulum
import pytest
+from pendulum.tz.timezone import Timezone
from airflow.utils import timezone
-from airflow.utils.timezone import coerce_datetime
+from airflow.utils.timezone import coerce_datetime, parse_timezone
-CET = pendulum.tz.timezone("Europe/Paris")
-EAT = pendulum.tz.timezone("Africa/Nairobi") # Africa/Nairobi
-ICT = pendulum.tz.timezone("Asia/Bangkok") # Asia/Bangkok
+CET = Timezone("Europe/Paris")
+EAT = Timezone("Africa/Nairobi") # Africa/Nairobi
+ICT = Timezone("Asia/Bangkok") # Asia/Bangkok
UTC = timezone.utc
@@ -117,3 +118,41 @@ class TestTimezone:
)
def test_coerce_datetime(input_datetime, output_datetime):
assert output_datetime == coerce_datetime(input_datetime)
+
+
+@pytest.mark.parametrize(
+ "tz_name",
+ [
+ pytest.param("Europe/Paris", id="CET"),
+ pytest.param("Africa/Nairobi", id="EAT"),
+ pytest.param("Asia/Bangkok", id="ICT"),
+ ],
+)
+def test_parse_timezone_iana(tz_name: str):
+ tz = parse_timezone(tz_name)
+ assert tz.name == tz_name
+ assert parse_timezone(tz_name) is tz
+
+
+@pytest.mark.parametrize("tz_name", ["utc", "UTC", "uTc"])
+def test_parse_timezone_utc(tz_name):
+ tz = parse_timezone(tz_name)
+ assert tz.name == "UTC"
+ assert parse_timezone(tz_name) is tz
+ assert tz is timezone.utc, "Expected that UTC timezone is same object as `airflow.utils.timezone.utc`"
+
+
+@pytest.mark.parametrize(
+ "tz_offset, expected_offset, expected_name",
+ [
+ pytest.param(0, 0, "+00:00", id="zero-offset"),
+ pytest.param(-3600, -3600, "-01:00", id="1-hour-behind"),
+ pytest.param(19800, 19800, "+05:30", id="5.5-hours-ahead"),
+ ],
+)
+def test_parse_timezone_offset(tz_offset: int, expected_offset, expected_name):
+ tz = parse_timezone(tz_offset)
+ assert hasattr(tz, "offset")
+ assert tz.offset == expected_offset
+ assert tz.name == expected_name
+ assert parse_timezone(tz_offset) is tz
(airflow) 01/20: Optimise getting changelogs in airflow-github (#36548)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 8dde227637f7d22592237ee8cbd0e36eb9da54cc
Author: Amogh Desai <am...@gmail.com>
AuthorDate: Fri Jan 5 11:59:55 2024 +0530
Optimise getting changelogs in airflow-github (#36548)
(cherry picked from commit 142c7376c6d6566938ced345e97a1e2442e2fd32)
---
dev/airflow-github | 23 ++++++++++++-----------
1 file changed, 12 insertions(+), 11 deletions(-)
diff --git a/dev/airflow-github b/dev/airflow-github
index 46213b411a..7d4ad9ebb9 100755
--- a/dev/airflow-github
+++ b/dev/airflow-github
@@ -323,29 +323,30 @@ def changelog(previous_version, target_version, github_token):
# Get a list of issues/PRs that have been committed on the current branch.
log = get_commits_between(repo, previous_version, target_version)
+ print("Number of commits", len(log))
+
gh = Github(github_token)
gh_repo = gh.get_repo("apache/airflow")
sections = defaultdict(list)
+ existing_commits = set()
+ with open("../RELEASE_NOTES.rst") as file:
+ for line in file.readlines():
+ match = re.search(r"\(#(\w+)\)", line)
+ if match:
+ existing_commits.add(match.group(1))
+
for commit in log:
tickets = pr_title_re.findall(commit["subject"])
if tickets:
issue = gh_repo.get_issue(number=int(tickets[0][1:]))
issue_type = get_issue_type(issue)
files = files_touched(repo, commit["id"])
+ if commit["id"] in existing_commits:
+ continue
if is_core_commit(files):
- if issue_type in ["bug-fix", "doc-only", "misc/internal"]:
- with open("../RELEASE_NOTES.rst") as file:
- for line in file.readlines():
- if line.endswith(f"(#{commit['id']})"):
- continue
- else:
- sections[issue_type].append(commit["subject"])
- else:
- sections[issue_type].append(commit["subject"])
-
+ sections[issue_type].append(commit["subject"])
else:
sections[DEFAULT_SECTION_NAME].append(commit["subject"])
-
print_changelog(sections)
(airflow) 09/20: Bump Cohere min version to limit backtracking on upgrade (#36666)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit ac9c4cfbaca42c557cc8f8ae50265a0f43fb551c
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Mon Jan 8 20:49:42 2024 +0100
Bump Cohere min version to limit backtracking on upgrade (#36666)
While testing #36537 I noticed cohere backracking quite a bit with
older version. Bumping Cohere to more recent minimum version
(released in November) decreased it quite a bit.
Since Cohere is mostly standalone package, and likely you want to
bump people to later version, it's safe to assume we can bump the
minimum version.
(cherry picked from commit 9797f9278a3a0cbe725e84ba0c80f686cd16f1e0)
---
airflow/providers/cohere/provider.yaml | 2 +-
generated/provider_dependencies.json | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/airflow/providers/cohere/provider.yaml b/airflow/providers/cohere/provider.yaml
index 01313be518..d09006af39 100644
--- a/airflow/providers/cohere/provider.yaml
+++ b/airflow/providers/cohere/provider.yaml
@@ -40,7 +40,7 @@ integrations:
dependencies:
- apache-airflow>=2.6.0
- - cohere>=4.27
+ - cohere>=4.37
hooks:
- integration-name: Cohere
diff --git a/generated/provider_dependencies.json b/generated/provider_dependencies.json
index b7007c4fe2..c4eb6d64dc 100644
--- a/generated/provider_dependencies.json
+++ b/generated/provider_dependencies.json
@@ -299,7 +299,7 @@
"cohere": {
"deps": [
"apache-airflow>=2.6.0",
- "cohere>=4.27"
+ "cohere>=4.37"
],
"cross-providers-deps": [],
"excluded-python-versions": [],
(airflow) 18/20: Add support for reproducible build date epoch for Airflow releases (#36726)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 405f8712e52e34e911bb2c237df6a799c11d1da5
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Fri Jan 12 01:29:31 2024 +0100
Add support for reproducible build date epoch for Airflow releases (#36726)
Hatch has built-in support for reproducible builds, however it
uses a hard-coded 2020 date to generate the reproducible binaries,
which produces whl, tar.gz files that contain file dates that are
pretty old. This might be confusing for anyone who is looking at
the file contents and timestamp inside.
This PR adds support (similar to provider approach) to store current
reproducible date in the repository - so that it can be committed
and tagged together with Airflow sources. It is updated fully
automaticallly by pre-commit whenever release notes change, which
basically means that whenever release notes are update just
before release, the reproducible date is updated to current date.
For now we only check if the packages produced by hatchling
build are reproducible.
(cherry picked from commit a2d6c389f69034c526554b3291874dc4d66c4529)
---
.pre-commit-config.yaml | 12 +++++-
.rat-excludes | 1 +
STATIC_CODE_CHECKS.rst | 2 +
airflow/reproducible_build.yaml | 2 +
dev/README_RELEASE_AIRFLOW.md | 42 ++++++++++++++++++++
.../commands/release_management_commands.py | 5 ++-
dev/breeze/src/airflow_breeze/pre_commit_ids.py | 1 +
images/breeze/output_static-checks.svg | 6 +--
images/breeze/output_static-checks.txt | 2 +-
.../pre_commit_update_source_date_epoch.py | 45 ++++++++++++++++++++++
.../in_container/run_prepare_airflow_packages.py | 13 ++++++-
11 files changed, 122 insertions(+), 9 deletions(-)
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index f50076ba61..f7b4d0ae6c 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -125,7 +125,7 @@ repos:
- --fuzzy-match-generates-todo
- id: insert-license
name: Add license for all YAML files except Helm templates
- exclude: ^\.github/.*$|^.*/.*_vendor/|^chart/templates/.*
+ exclude: ^\.github/.*$|^.*/.*_vendor/|^chart/templates/.*|^airflow/reproducible_build.yaml$
types: [yaml]
files: \.ya?ml$
args:
@@ -244,7 +244,7 @@ repos:
name: Check YAML files with yamllint
entry: yamllint -c yamllint-config.yml --strict
types: [yaml]
- exclude: ^.*airflow\.template\.yaml$|^.*init_git_sync\.template\.yaml$|^.*/.*_vendor/|^chart/(?:templates|files)/.*\.yaml$|openapi/.*\.yaml$|^\.pre-commit-config\.yaml$
+ exclude: ^.*airflow\.template\.yaml$|^.*init_git_sync\.template\.yaml$|^.*/.*_vendor/|^chart/(?:templates|files)/.*\.yaml$|openapi/.*\.yaml$|^\.pre-commit-config\.yaml$|^airflow/reproducible_build.yaml$
- repo: https://github.com/ikamensh/flynt
rev: '1.0.1'
hooks:
@@ -794,6 +794,14 @@ repos:
files: ^dev/breeze/pyproject\.toml$|^dev/breeze/README\.md$
pass_filenames: false
require_serial: true
+ - id: update-reproducible-source-date-epoch
+ name: Update Source Date Epoch for reproducible builds
+ language: python
+ entry: ./scripts/ci/pre_commit/pre_commit_update_source_date_epoch.py
+ files: ^RELEASE_NOTES.rst$
+ additional_dependencies: ['rich>=12.4.4', 'pyyaml']
+ pass_filenames: false
+ require_serial: true
- id: check-breeze-top-dependencies-limited
name: Breeze should have small number of top-level dependencies
language: python
diff --git a/.rat-excludes b/.rat-excludes
index ab2296b487..9822c9ac41 100644
--- a/.rat-excludes
+++ b/.rat-excludes
@@ -86,6 +86,7 @@ PULL_REQUEST_TEMPLATE.md
PROVIDER_CHANGES*.md
manifests/*
redirects.txt
+reproducible_build.yaml
# Locally mounted files
.*egg-info/*
diff --git a/STATIC_CODE_CHECKS.rst b/STATIC_CODE_CHECKS.rst
index 68a0aef451..3f748579b0 100644
--- a/STATIC_CODE_CHECKS.rst
+++ b/STATIC_CODE_CHECKS.rst
@@ -394,6 +394,8 @@ require Breeze Docker image to be built locally.
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
| update-providers-dependencies | Update dependencies for provider packages | |
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
+| update-reproducible-source-date-epoch | Update Source Date Epoch for reproducible builds | |
++-----------------------------------------------------------+--------------------------------------------------------------+---------+
| update-spelling-wordlist-to-be-sorted | Sort alphabetically and uniquify spelling_wordlist.txt | |
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
| update-supported-versions | Updates supported versions in documentation | |
diff --git a/airflow/reproducible_build.yaml b/airflow/reproducible_build.yaml
new file mode 100644
index 0000000000..2bd6204c80
--- /dev/null
+++ b/airflow/reproducible_build.yaml
@@ -0,0 +1,2 @@
+release-notes-hash: 81a945804fc42c18f416b5aa1f4b0fde
+source-date-epoch: 1704922121
diff --git a/dev/README_RELEASE_AIRFLOW.md b/dev/README_RELEASE_AIRFLOW.md
index dabc02ea2d..50ee3b280d 100644
--- a/dev/README_RELEASE_AIRFLOW.md
+++ b/dev/README_RELEASE_AIRFLOW.md
@@ -34,6 +34,7 @@
- [Licence check](#licence-check)
- [Signature check](#signature-check)
- [SHA512 sum check](#sha512-sum-check)
+ - [Reproducible package check](#reproducible-package-check)
- [Source code check](#source-code-check)
- [Verify the release candidate by Contributors](#verify-the-release-candidate-by-contributors)
- [Installing release candidate in your local virtual environment](#installing-release-candidate-in-your-local-virtual-environment)
@@ -578,6 +579,47 @@ Checking apache_airflow-2.0.2rc4-py2.py3-none-any.whl.sha512
Checking apache-airflow-2.0.2rc4-source.tar.gz.sha512
```
+## Reproducible package check
+
+Airflow supports reproducible builds, which means that the packages prepared from the same sources should
+produce binary identical packages in reproducible way. You should check if the packages can be
+binary-reproduced when built from the sources.
+
+Checkout airflow sources and build packages in dist folder:
+
+```shell script
+git checkout X.Y.Zrc1
+export AIRFLOW_REPO_ROOT=$(pwd)
+rm -rf dist/*
+breeze release-management prepare-airflow-package --package-format both
+```
+
+That should produce `.whl` and `.tar.gz` packages in dist folder.
+
+Change to the directory where you have the packages from svn:
+
+```shell script
+# First clone the repo if you do not have it
+cd ..
+[ -d asf-dist ] || svn checkout --depth=immediates https://dist.apache.org/repos/dist asf-dist
+svn update --set-depth=infinity asf-dist/dev/airflow
+
+# Then compare the packages
+cd asf-dist/dev/airflow/X.Y.Zrc1
+for i in ${AIRFLOW_REPO_ROOT}/dist/*
+do
+ echo "Checking if $(basename $i) is the same as $i"
+ diff "$(basename $i)" "$i" && echo "OK"
+done
+```
+
+The output should be empty (files are identical).
+In case the files are different, you should see:
+
+```
+Binary files apache_airflow-2.9.0.dev0.tar.gz and .../apache_airflow-2.9.0.dev0.tar.gz differ
+```
+
## Source code check
You should check if the sources in the packages produced are the same as coming from the tag in git.
diff --git a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
index 39211c9658..ca2041bd94 100644
--- a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
@@ -209,12 +209,13 @@ GITPYTHON_VERSION = "3.1.40"
RICH_VERSION = "13.7.0"
NODE_VERSION = "21.2.0"
PRE_COMMIT_VERSION = "3.5.0"
+PYYAML_VERSION = "6.0.1"
AIRFLOW_BUILD_DOCKERFILE = f"""
FROM python:{DEFAULT_PYTHON_MAJOR_MINOR_VERSION}-slim-{ALLOWED_DEBIAN_VERSIONS[0]}
RUN apt-get update && apt-get install -y --no-install-recommends git
-RUN pip install pip=={AIRFLOW_PIP_VERSION} hatch==1.9.1 \
- gitpython=={GITPYTHON_VERSION} rich=={RICH_VERSION} pre-commit=={PRE_COMMIT_VERSION}
+RUN pip install pip=={AIRFLOW_PIP_VERSION} hatch==1.9.1 pyyaml=={PYYAML_VERSION}\
+ gitpython=={GITPYTHON_VERSION} rich=={RICH_VERSION} pre-commit=={PRE_COMMIT_VERSION}
COPY . /opt/airflow
"""
diff --git a/dev/breeze/src/airflow_breeze/pre_commit_ids.py b/dev/breeze/src/airflow_breeze/pre_commit_ids.py
index e5ad08afa1..8e64e536c7 100644
--- a/dev/breeze/src/airflow_breeze/pre_commit_ids.py
+++ b/dev/breeze/src/airflow_breeze/pre_commit_ids.py
@@ -129,6 +129,7 @@ PRE_COMMIT_LIST = [
"update-local-yml-file",
"update-migration-references",
"update-providers-dependencies",
+ "update-reproducible-source-date-epoch",
"update-spelling-wordlist-to-be-sorted",
"update-supported-versions",
"update-vendored-in-k8s-json-schema",
diff --git a/images/breeze/output_static-checks.svg b/images/breeze/output_static-checks.svg
index 4212e283ea..acbc21e6b4 100644
--- a/images/breeze/output_static-checks.svg
+++ b/images/breeze/output_static-checks.svg
@@ -348,9 +348,9 @@
</text><text class="breeze-static-checks-r5" x="0" y="1118" textLength="12.2" clip-path="url(#breeze-static-checks-line-45)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1118" textLength="988.2" clip-path="url(#breeze-static-checks-line-45)">update-er-diagram | update-extras | update-in-the-wild-to-be-sorted |            </text><text class="breeze-static-checks-r5" x="1451.8" y="1118" textL [...]
</text><text class="breeze-static-checks-r5" x="0" y="1142.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-46)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1142.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-46)">update-inlined-dockerfile-scripts | update-installed-providers-to-be-sorted |    </text><text class="breeze-static-checks-r5" x="1451.8" y="1142.4" textLength="12.2" clip-path="url(#breeze-static-c [...]
</text><text class="breeze-static-checks-r5" x="0" y="1166.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-47)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1166.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-47)">update-local-yml-file | update-migration-references |                           &# [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1191.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-48)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1191.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-48)">update-providers-dependencies | update-spelling-wordlist-to-be-sorted |          </text><text class="breeze-static-checks-r5" x="1451.8" y="1191.2" textLength="12.2" c [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1215.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-49)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1215.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-49)">update-supported-versions | update-vendored-in-k8s-json-schema | update-version |</text><text class="breeze-static-checks-r5" x="1451.8" y="1215.6" textLength="12.2" clip-path="url(#breeze-static-checks-line [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1240" textLength="12.2" clip-path="url(#breeze-static-checks-line-50)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1240" textLength="988.2" clip-path="url(#breeze-static-checks-line-50)">yamllint)                                       [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1191.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-48)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1191.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-48)">update-providers-dependencies | update-reproducible-source-date-epoch |          </text><text class="breeze-static-checks-r5" x="1451.8" y="1191.2" textLength="12.2" c [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1215.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-49)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1215.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-49)">update-spelling-wordlist-to-be-sorted | update-supported-versions |              </text><text class="breeze-static-checks-r5" x="1451.8" y="1215.6" [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1240" textLength="12.2" clip-path="url(#breeze-static-checks-line-50)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1240" textLength="988.2" clip-path="url(#breeze-static-checks-line-50)">update-vendored-in-k8s-json-schema | update-version | yamllint)                  </text><text class="breeze-static-checks-r5" [...]
</text><text class="breeze-static-checks-r5" x="0" y="1264.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-51)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1264.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-51)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1264.4" textLength="61" clip-path="url(#breeze-static-checks-line-51)">-show</text><text class="breeze-static-checks-r4" x="97.6" y="1264.4" textLength="195.2" clip-path="url(# [...]
</text><text class="breeze-static-checks-r5" x="0" y="1288.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-52)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1288.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-52)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1288.8" textLength="134.2" clip-path="url(#breeze-static-checks-line-52)">-initialize</text><text class="breeze-static-checks-r4" x="170.8" y="1288.8" textLength="146.4" clip-p [...]
</text><text class="breeze-static-checks-r5" x="0" y="1313.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-53)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1313.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-53)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1313.2" textLength="48.8" clip-path="url(#breeze-static-checks-line-53)">-max</text><text class="breeze-static-checks-r4" x="85.4" y="1313.2" textLength="292.8" clip-path="url( [...]
diff --git a/images/breeze/output_static-checks.txt b/images/breeze/output_static-checks.txt
index 8f9982f8cf..b1b9c6d6be 100644
--- a/images/breeze/output_static-checks.txt
+++ b/images/breeze/output_static-checks.txt
@@ -1 +1 @@
-01342faaf5558fb2af9ce20c7ca0be8c
+d466e07e997920503e147e798ed2b353
diff --git a/scripts/ci/pre_commit/pre_commit_update_source_date_epoch.py b/scripts/ci/pre_commit/pre_commit_update_source_date_epoch.py
new file mode 100755
index 0000000000..6bd15d8bd3
--- /dev/null
+++ b/scripts/ci/pre_commit/pre_commit_update_source_date_epoch.py
@@ -0,0 +1,45 @@
+#!/usr/bin/env python
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import sys
+from hashlib import md5
+from pathlib import Path
+from time import time
+
+import yaml
+
+sys.path.insert(0, str(Path(__file__).parent.resolve())) # make sure common_precommit_utils is importable
+
+from common_precommit_utils import AIRFLOW_SOURCES_ROOT_PATH
+
+RELEASE_NOTES_FILE_PATH = AIRFLOW_SOURCES_ROOT_PATH / "RELEASE_NOTES.rst"
+REPRODUCIBLE_BUILD_FILE = AIRFLOW_SOURCES_ROOT_PATH / "airflow" / "reproducible_build.yaml"
+
+if __name__ == "__main__":
+ hash_md5 = md5()
+ hash_md5.update(RELEASE_NOTES_FILE_PATH.read_bytes())
+ release_notes_hash = hash_md5.hexdigest()
+ reproducible_build_text = REPRODUCIBLE_BUILD_FILE.read_text()
+ reproducible_build = yaml.safe_load(reproducible_build_text)
+ old_hash = reproducible_build["release-notes-hash"]
+ if release_notes_hash != old_hash:
+ # Replace the hash in the file
+ reproducible_build["release-notes-hash"] = release_notes_hash
+ reproducible_build["source-date-epoch"] = int(time())
+ REPRODUCIBLE_BUILD_FILE.write_text(yaml.dump(reproducible_build))
diff --git a/scripts/in_container/run_prepare_airflow_packages.py b/scripts/in_container/run_prepare_airflow_packages.py
index 2f6f1912c2..66aae93ca4 100755
--- a/scripts/in_container/run_prepare_airflow_packages.py
+++ b/scripts/in_container/run_prepare_airflow_packages.py
@@ -26,11 +26,13 @@ from contextlib import contextmanager
from pathlib import Path
from shutil import rmtree
+import yaml
from rich.console import Console
console = Console(color_system="standard", width=200)
AIRFLOW_SOURCES_ROOT = Path(__file__).parents[2].resolve()
+REPRODUCIBLE_BUILD_FILE = AIRFLOW_SOURCES_ROOT / "airflow" / "reproducible_build.yaml"
AIRFLOW_INIT_FILE = AIRFLOW_SOURCES_ROOT / "airflow" / "__init__.py"
WWW_DIRECTORY = AIRFLOW_SOURCES_ROOT / "airflow" / "www"
VERSION_SUFFIX = os.environ.get("VERSION_SUFFIX_FOR_PYPI", "")
@@ -81,8 +83,17 @@ def build_airflow_packages(package_format: str):
if package_format in ["both", "sdist"]:
build_command.extend(["-t", "sdist"])
+ reproducible_date = yaml.safe_load(REPRODUCIBLE_BUILD_FILE.read_text())["source-date-epoch"]
+
+ envcopy = os.environ.copy()
+ envcopy["SOURCE_DATE_EPOCH"] = str(reproducible_date)
console.print(f"[bright_blue]Building packages: {package_format}\n")
- build_process = subprocess.run(build_command, capture_output=False, cwd=AIRFLOW_SOURCES_ROOT)
+ build_process = subprocess.run(
+ build_command,
+ capture_output=False,
+ cwd=AIRFLOW_SOURCES_ROOT,
+ env=envcopy,
+ )
if build_process.returncode != 0:
console.print("[red]Error building Airflow packages")
(airflow) 03/20: Upgrade to latest versions of `pip` and `pipx` in CI runners (#36646)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit a608981227805929c1a3f4f408c4ce44ca8283c3
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sun Jan 7 13:53:02 2024 +0100
Upgrade to latest versions of `pip` and `pipx` in CI runners (#36646)
The CI runners did not have latest version of `pip` and `pipx`. This
change updates the installation scripts to fix `pip` to the same
version as in the CI image and down-binds pipx to 1.4.1 which is
recently released bugfix version with better logging and installation
instructions.
(cherry picked from commit 75bc05ce1f53de112f7eee7be524a26f2a3f6845)
---
BREEZE.rst | 5 ++---
dev/breeze/README.md | 2 +-
dev/breeze/pyproject.toml | 1 +
scripts/ci/install_breeze.sh | 3 ++-
4 files changed, 6 insertions(+), 5 deletions(-)
diff --git a/BREEZE.rst b/BREEZE.rst
index 7bcda45029..85164fb456 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -176,14 +176,13 @@ The pipx tool
We are using ``pipx`` tool to install and manage Breeze. The ``pipx`` tool is created by the creators
of ``pip`` from `Python Packaging Authority <https://www.pypa.io/en/latest/>`_
-Note that ``pipx`` >= 1.2.1 is needed in order to deal with breaking ``packaging`` release in September
-2023 that broke earlier versions of ``pipx``.
+Note that ``pipx`` >= 1.4.1 is used.
Install pipx
.. code-block:: bash
- pip install --user "pipx>=1.2.1"
+ pip install --user "pipx>=1.4.1"
Breeze, is not globally accessible until your PATH is updated. Add <USER FOLDER>\.local\bin as a variable
environments. This can be done automatically by the following command (follow instructions printed).
diff --git a/dev/breeze/README.md b/dev/breeze/README.md
index 9db81c1fd4..7452be51ae 100644
--- a/dev/breeze/README.md
+++ b/dev/breeze/README.md
@@ -66,6 +66,6 @@ PLEASE DO NOT MODIFY THE HASH BELOW! IT IS AUTOMATICALLY UPDATED BY PRE-COMMIT.
---------------------------------------------------------------------------------------------------------
-Package config hash: c7d80ab49c6dc4bf2b54957663b0126ab9c8f48df28a34c0eb56340540cb1f52d063ef99ee5f9cacbd375b1a711278884f9ef9aab41e620fa70fffd81f7ece3c
+Package config hash: ac1e65234e1a780d1f21ac28e27451c763e308cc901ab8f72acc6414db8da14c8f841e4757b134a4cc6f3c76186af340db0610cd3fa155ed5fcf1abca45381e8
---------------------------------------------------------------------------------------------------------
diff --git a/dev/breeze/pyproject.toml b/dev/breeze/pyproject.toml
index aca59140ac..9cae9052e2 100644
--- a/dev/breeze/pyproject.toml
+++ b/dev/breeze/pyproject.toml
@@ -54,6 +54,7 @@ dependencies = [
"jinja2>=3.1.0",
"jsonschema>=4.19.1",
"packaging>=23.2",
+ "pipx>=1.4.1",
"pre-commit>=3.5.0",
"psutil>=5.9.6",
"pygithub>=2.1.1",
diff --git a/scripts/ci/install_breeze.sh b/scripts/ci/install_breeze.sh
index d07ef765c6..e8e3a63f80 100755
--- a/scripts/ci/install_breeze.sh
+++ b/scripts/ci/install_breeze.sh
@@ -19,6 +19,7 @@ set -euxo pipefail
cd "$( dirname "${BASH_SOURCE[0]}" )/../../"
-python -m pip install "pipx>=1.2.1"
+python -m pip install --upgrade pip==23.3.2
+python -m pip install "pipx>=1.4.1"
python -m pipx install --editable ./dev/breeze/ --force
echo '/home/runner/.local/bin' >> "${GITHUB_PATH}"
(airflow) 14/20: Get rid of pyarrow-hotfix for CVE-2023-47248 (#36697)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 5690c0d17e0fd23635ab19582bc35a261e1e3889
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Tue Jan 9 21:41:45 2024 +0100
Get rid of pyarrow-hotfix for CVE-2023-47248 (#36697)
The #35650 introduced a hotfix for Pyarrow CVE-2023-47248. So far
we have been blocked from removing it by Apache Beam that limited
Airflow from bumping pyarrow to a version that was not vulnerable.
This is now possible since Apache Beam relesed 2.53.0 version on
4th of January 2023 that allows to use non-vulnerable pyarrow.
We are now bumping both Pyarrow and Beam minimum versions to
reflect that and remove pyarrow hotfix.
(cherry picked from commit d105c7115f56f88d48a2888484a0ed7d1c01576f)
---
setup.py | 2 ++
1 file changed, 2 insertions(+)
diff --git a/setup.py b/setup.py
index 0628f3aacf..1c831fb98d 100644
--- a/setup.py
+++ b/setup.py
@@ -903,6 +903,8 @@ class AirflowDistribution(Distribution):
self.package_data["airflow"].append(provider_relative_path)
# Add python_kubernetes_script.jinja2 to package data
self.package_data["airflow"].append("providers/cncf/kubernetes/python_kubernetes_script.jinja2")
+ # Add default email template to package data
+ self.package_data["airflow"].append("providers/smtp/notifications/templates/email.html")
else:
self.install_requires.extend(
[
(airflow) 12/20: Bump min telegram-bot version (#36670)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 2d04f8a83d85f9923ad688b7aa4ee95887dbd466
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Mon Jan 8 22:58:27 2024 +0100
Bump min telegram-bot version (#36670)
This is the last one from the long-backtracking series.
Telegram 20.2 has been released in March 2023 and for all practical
purposes using recent version is a good idea to interact with such
services. Bumping it cuts down on a number of backtracking loops
pip has to do when backtracking.
(cherry picked from commit 1f6d764334925e0efb417eebc872e129d03fbf1a)
---
airflow/providers/telegram/provider.yaml | 2 +-
generated/provider_dependencies.json | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/airflow/providers/telegram/provider.yaml b/airflow/providers/telegram/provider.yaml
index d5f4c00d1a..14bcdf5ac2 100644
--- a/airflow/providers/telegram/provider.yaml
+++ b/airflow/providers/telegram/provider.yaml
@@ -43,7 +43,7 @@ versions:
dependencies:
- apache-airflow>=2.6.0
- - python-telegram-bot>=20.0.0
+ - python-telegram-bot>=20.2
integrations:
- integration-name: Telegram
diff --git a/generated/provider_dependencies.json b/generated/provider_dependencies.json
index d1d3c0fe20..e9e048141c 100644
--- a/generated/provider_dependencies.json
+++ b/generated/provider_dependencies.json
@@ -1011,7 +1011,7 @@
"telegram": {
"deps": [
"apache-airflow>=2.6.0",
- "python-telegram-bot>=20.0.0"
+ "python-telegram-bot>=20.2"
],
"cross-providers-deps": [],
"excluded-python-versions": [],
(airflow) 06/20: Make `graphviz` dependency optional (#36647)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit e6de1ac90c00912420b61b1cc531c9602bfe3c19
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sun Jan 7 17:02:32 2024 +0100
Make `graphviz` dependency optional (#36647)
The `graphviz` dependency has been problematic as Airflow required
dependency - especially for ARM-based installations. Graphviz
packages require binary graphviz libraries - which is already a
limitation, but they also require to install graphviz Python
bindings to be build and installed. This does not work for older
Linux installation but - more importantly - when you try
to install Graphviz libraries for Python 3.8, 3.9 for ARM M1
MacBooks, the packages fail to install because Python bindings
compilation for M1 can only work for Python 3.10+.
There is not an easy solution for that except commenting out
graphviz dependency from setup.py, when you want to install Airflow
for Python 3.8, 3.9 for MacBook M1.
However Graphviz is really used in two places:
* when you want to render DAGs wia airflow CLI - either to an image
or directly to terminal (for terminals/systems supporting imgcat)
* when you want to render ER diagram after you modified Airflow
models
The latter is a development-only feature, the former is production
feature, however it is a very niche one.
This PR turns rendering of the images in Airflow in optional feature
(only working when graphviz python bindings are installed) and
effectively turns graphviz into an optional extra (and removes it
from requirements).
This is not a breaking change technically - the CLIs to render the
DAGs is still there and IF you already have graphviz installed, it
will continue working as it did before. The only problem when it
does not work is where you do not have graphviz installed for
fresh installation and it will raise an error and inform that you need it.
Graphviz will remain to be installed for most users:
* the Airflow Image will still contain graphviz library, because
it is added there as extra
* when previous version of Airflow has been installed already, then
graphviz library is already installed there and Airflow will
continue working as it did
The only change will be a new installation of new version of Airflow
from the scratch, where graphviz will need to be specified as extra
or installed separately in order to enable DAG rendering option.
Taking into account this behaviour (which only requires to install
a graphviz package), this should not be considered as a breaking
change.
Extracted from: #36537
(cherry picked from commit 89f1737afb27f6e708c2e83e3d8e751d9a36f91e)
---
CONTRIBUTING.rst | 14 +++++++-------
Dockerfile | 2 +-
INSTALL | 14 +++++++-------
airflow/utils/dot_renderer.py | 15 ++++++++++++++-
dev/breeze/src/airflow_breeze/global_constants.py | 1 +
docs/apache-airflow/extra-packages-ref.rst | 2 ++
docs/docker-stack/build-arg-ref.rst | 1 +
docs/spelling_wordlist.txt | 1 +
images/breeze/output_prod-image_build.txt | 2 +-
newsfragments/36647.significant.rst | 23 +++++++++++++++++++++++
setup.cfg | 1 -
setup.py | 3 +++
12 files changed, 61 insertions(+), 18 deletions(-)
diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
index abb9ed59da..ba3ff9ec98 100644
--- a/CONTRIBUTING.rst
+++ b/CONTRIBUTING.rst
@@ -675,13 +675,13 @@ arangodb, asana, async, atlas, atlassian.jira, aws, azure, cassandra, celery, cg
cncf.kubernetes, cohere, common.io, common.sql, crypto, databricks, datadog, dbt.cloud,
deprecated_api, devel, devel_all, devel_ci, devel_hadoop, dingding, discord, doc, doc_gen, docker,
druid, elasticsearch, exasol, facebook, ftp, gcp, gcp_api, github, github_enterprise, google,
-google_auth, grpc, hashicorp, hdfs, hive, http, imap, influxdb, jdbc, jenkins, kerberos, kubernetes,
-ldap, leveldb, microsoft.azure, microsoft.mssql, microsoft.psrp, microsoft.winrm, mongo, mssql,
-mysql, neo4j, odbc, openai, openfaas, openlineage, opensearch, opsgenie, oracle, otel, pagerduty,
-pandas, papermill, password, pgvector, pinecone, pinot, postgres, presto, rabbitmq, redis, s3, s3fs,
-salesforce, samba, saml, segment, sendgrid, sentry, sftp, singularity, slack, smtp, snowflake,
-spark, sqlite, ssh, statsd, tableau, tabular, telegram, trino, vertica, virtualenv, weaviate,
-webhdfs, winrm, yandex, zendesk
+google_auth, graphviz, grpc, hashicorp, hdfs, hive, http, imap, influxdb, jdbc, jenkins, kerberos,
+kubernetes, ldap, leveldb, microsoft.azure, microsoft.mssql, microsoft.psrp, microsoft.winrm, mongo,
+mssql, mysql, neo4j, odbc, openai, openfaas, openlineage, opensearch, opsgenie, oracle, otel,
+pagerduty, pandas, papermill, password, pgvector, pinecone, pinot, postgres, presto, rabbitmq,
+redis, s3, s3fs, salesforce, samba, saml, segment, sendgrid, sentry, sftp, singularity, slack, smtp,
+snowflake, spark, sqlite, ssh, statsd, tableau, tabular, telegram, trino, vertica, virtualenv,
+weaviate, webhdfs, winrm, yandex, zendesk
.. END EXTRAS HERE
Provider packages
diff --git a/Dockerfile b/Dockerfile
index 55ba492e51..e493d99ec0 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -35,7 +35,7 @@
# much smaller.
#
# Use the same builder frontend version for everyone
-ARG AIRFLOW_EXTRAS="aiobotocore,amazon,async,celery,cncf.kubernetes,common.io,docker,elasticsearch,ftp,google,google_auth,grpc,hashicorp,http,ldap,microsoft.azure,mysql,odbc,openlineage,pandas,postgres,redis,sendgrid,sftp,slack,snowflake,ssh,statsd,virtualenv"
+ARG AIRFLOW_EXTRAS="aiobotocore,amazon,async,celery,cncf.kubernetes,common.io,docker,elasticsearch,ftp,google,google_auth,graphviz,grpc,hashicorp,http,ldap,microsoft.azure,mysql,odbc,openlineage,pandas,postgres,redis,sendgrid,sftp,slack,snowflake,ssh,statsd,virtualenv"
ARG ADDITIONAL_AIRFLOW_EXTRAS=""
ARG ADDITIONAL_PYTHON_DEPS=""
diff --git a/INSTALL b/INSTALL
index 9630c50985..832fd8603d 100644
--- a/INSTALL
+++ b/INSTALL
@@ -101,13 +101,13 @@ arangodb, asana, async, atlas, atlassian.jira, aws, azure, cassandra, celery, cg
cncf.kubernetes, cohere, common.io, common.sql, crypto, databricks, datadog, dbt.cloud,
deprecated_api, devel, devel_all, devel_ci, devel_hadoop, dingding, discord, doc, doc_gen, docker,
druid, elasticsearch, exasol, facebook, ftp, gcp, gcp_api, github, github_enterprise, google,
-google_auth, grpc, hashicorp, hdfs, hive, http, imap, influxdb, jdbc, jenkins, kerberos, kubernetes,
-ldap, leveldb, microsoft.azure, microsoft.mssql, microsoft.psrp, microsoft.winrm, mongo, mssql,
-mysql, neo4j, odbc, openai, openfaas, openlineage, opensearch, opsgenie, oracle, otel, pagerduty,
-pandas, papermill, password, pgvector, pinecone, pinot, postgres, presto, rabbitmq, redis, s3, s3fs,
-salesforce, samba, saml, segment, sendgrid, sentry, sftp, singularity, slack, smtp, snowflake,
-spark, sqlite, ssh, statsd, tableau, tabular, telegram, trino, vertica, virtualenv, weaviate,
-webhdfs, winrm, yandex, zendesk
+google_auth, graphviz, grpc, hashicorp, hdfs, hive, http, imap, influxdb, jdbc, jenkins, kerberos,
+kubernetes, ldap, leveldb, microsoft.azure, microsoft.mssql, microsoft.psrp, microsoft.winrm, mongo,
+mssql, mysql, neo4j, odbc, openai, openfaas, openlineage, opensearch, opsgenie, oracle, otel,
+pagerduty, pandas, papermill, password, pgvector, pinecone, pinot, postgres, presto, rabbitmq,
+redis, s3, s3fs, salesforce, samba, saml, segment, sendgrid, sentry, sftp, singularity, slack, smtp,
+snowflake, spark, sqlite, ssh, statsd, tableau, tabular, telegram, trino, vertica, virtualenv,
+weaviate, webhdfs, winrm, yandex, zendesk
# END EXTRAS HERE
# For installing Airflow in development environments - see CONTRIBUTING.rst
diff --git a/airflow/utils/dot_renderer.py b/airflow/utils/dot_renderer.py
index 41281fbbb1..4d44d1e2ec 100644
--- a/airflow/utils/dot_renderer.py
+++ b/airflow/utils/dot_renderer.py
@@ -19,9 +19,14 @@
"""Renderer DAG (tasks and dependencies) to the graphviz object."""
from __future__ import annotations
+import warnings
from typing import TYPE_CHECKING, Any
-import graphviz
+try:
+ import graphviz
+except ImportError:
+ warnings.warn("Could not import graphviz. Rendering graph to the graphical format will not be possible.")
+ graphviz = None
from airflow.exceptions import AirflowException
from airflow.models.baseoperator import BaseOperator
@@ -151,6 +156,10 @@ def render_dag_dependencies(deps: dict[str, list[DagDependency]]) -> graphviz.Di
:param deps: List of DAG dependencies
:return: Graphviz object
"""
+ if not graphviz:
+ raise AirflowException(
+ "Could not import graphviz. Install the graphviz python package to fix this error."
+ )
dot = graphviz.Digraph(graph_attr={"rankdir": "LR"})
for dag, dependencies in deps.items():
@@ -179,6 +188,10 @@ def render_dag(dag: DAG, tis: list[TaskInstance] | None = None) -> graphviz.Digr
:param tis: List of task instances
:return: Graphviz object
"""
+ if not graphviz:
+ raise AirflowException(
+ "Could not import graphviz. Install the graphviz python package to fix this error."
+ )
dot = graphviz.Digraph(
dag.dag_id,
graph_attr={
diff --git a/dev/breeze/src/airflow_breeze/global_constants.py b/dev/breeze/src/airflow_breeze/global_constants.py
index 3df224b0b1..15e6bbec94 100644
--- a/dev/breeze/src/airflow_breeze/global_constants.py
+++ b/dev/breeze/src/airflow_breeze/global_constants.py
@@ -437,6 +437,7 @@ DEFAULT_EXTRAS = [
"ftp",
"google",
"google_auth",
+ "graphviz",
"grpc",
"hashicorp",
"http",
diff --git a/docs/apache-airflow/extra-packages-ref.rst b/docs/apache-airflow/extra-packages-ref.rst
index 324b5e3023..0b6a51fd32 100644
--- a/docs/apache-airflow/extra-packages-ref.rst
+++ b/docs/apache-airflow/extra-packages-ref.rst
@@ -52,6 +52,8 @@ python dependencies for the provided package.
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| google_auth | ``pip install 'apache-airflow[google_auth]'`` | Google auth backend |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
+| graphviz | ``pip install 'apache-airflow[graphviz]'`` | Graphviz renderer for converting DAG to graphical output |
++---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| kerberos | ``pip install 'apache-airflow[kerberos]'`` | Kerberos integration for Kerberized services (Hadoop, Presto, Trino) |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| ldap | ``pip install 'apache-airflow[ldap]'`` | LDAP authentication for users |
diff --git a/docs/docker-stack/build-arg-ref.rst b/docs/docker-stack/build-arg-ref.rst
index a07760558e..73c30a3892 100644
--- a/docs/docker-stack/build-arg-ref.rst
+++ b/docs/docker-stack/build-arg-ref.rst
@@ -91,6 +91,7 @@ List of default extras in the production Dockerfile:
* ftp
* google
* google_auth
+* graphviz
* grpc
* hashicorp
* http
diff --git a/docs/spelling_wordlist.txt b/docs/spelling_wordlist.txt
index 56198798b4..e7b4b509b9 100644
--- a/docs/spelling_wordlist.txt
+++ b/docs/spelling_wordlist.txt
@@ -685,6 +685,7 @@ googleapiclient
GoogleDisplayVideo
gpu
gpus
+graphviz
greenlet
Groupalia
groupId
diff --git a/images/breeze/output_prod-image_build.txt b/images/breeze/output_prod-image_build.txt
index 9e5a2533b2..084f184b36 100644
--- a/images/breeze/output_prod-image_build.txt
+++ b/images/breeze/output_prod-image_build.txt
@@ -1 +1 @@
-0a11dcb596c1bf9fd08489bbba695aad
+ed924a9f4691700717cb77864ba9603a
diff --git a/newsfragments/36647.significant.rst b/newsfragments/36647.significant.rst
new file mode 100644
index 0000000000..dc3f0faad8
--- /dev/null
+++ b/newsfragments/36647.significant.rst
@@ -0,0 +1,23 @@
+Graphviz dependency is now an optional one, not required one.
+
+The ``graphviz`` dependency has been problematic as Airflow required dependency - especially for
+ARM-based installations. Graphviz packages require binary graphviz libraries - which is already a
+limitation, but they also require to install graphviz Python bindings to be build and installed.
+This does not work for older Linux installation but - more importantly - when you try to install
+Graphviz libraries for Python 3.8, 3.9 for ARM M1 MacBooks, the packages fail to install because
+Python bindings compilation for M1 can only work for Python 3.10+.
+
+This is not a breaking change technically - the CLIs to render the DAGs is still there and IF you
+already have graphviz installed, it will continue working as it did before. The only problem when it
+does not work is where you do not have graphviz installed it will raise an error and inform that you need it.
+
+Graphviz will remain to be installed for most users:
+
+* the Airflow Image will still contain graphviz library, because
+ it is added there as extra
+* when previous version of Airflow has been installed already, then
+ graphviz library is already installed there and Airflow will
+ continue working as it did
+
+The only change will be a new installation of new version of Airflow from the scratch, where graphviz will
+need to be specified as extra or installed separately in order to enable DAG rendering option.
diff --git a/setup.cfg b/setup.cfg
index 0a2478b618..ab7b34511c 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -108,7 +108,6 @@ install_requires =
flask-wtf>=0.15
fsspec>=2023.10.0
google-re2>=1.0
- graphviz>=0.12
gunicorn>=20.1.0
httpx
importlib_metadata>=1.7;python_version<"3.9"
diff --git a/setup.py b/setup.py
index 9d118bcfdb..2b01c28a6b 100644
--- a/setup.py
+++ b/setup.py
@@ -318,12 +318,14 @@ doc = [
]
doc_gen = [
"eralchemy2",
+ "graphviz>=0.12",
]
flask_appbuilder_oauth = [
"authlib>=1.0.0",
# The version here should be upgraded at the same time as flask-appbuilder in setup.cfg
"flask-appbuilder[oauth]==4.3.10",
]
+graphviz = ["graphviz>=0.12"]
kerberos = [
"pykerberos>=1.1.13",
"requests_kerberos>=0.10.0",
@@ -589,6 +591,7 @@ CORE_EXTRAS_DEPENDENCIES: dict[str, list[str]] = {
"deprecated_api": deprecated_api,
"github_enterprise": flask_appbuilder_oauth,
"google_auth": flask_appbuilder_oauth,
+ "graphviz": graphviz,
"kerberos": kerberos,
"ldap": ldap,
"leveldb": leveldb,
(airflow) 15/20: Explicitly passing `raise_on_deleted_version=True` to `read_secret_version` in Hashicorp operator (#36532)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 2ca6d8edebe16c8b126ac2e08749549fa3dc6df6
Author: rom sharon <33...@users.noreply.github.com>
AuthorDate: Wed Jan 10 21:04:50 2024 +0200
Explicitly passing `raise_on_deleted_version=True` to `read_secret_version` in Hashicorp operator (#36532)
* explicitly passing raise_on_deleted_version=True to read_secret_version
* fix tests
* update hvac version
(cherry picked from commit cd5ab08d95aaf4c65e56a91f1843d04c09f27cb1)
---
airflow/providers/hashicorp/_internal_client/vault_client.py | 10 ++++++++--
airflow/providers/hashicorp/provider.yaml | 2 +-
generated/provider_dependencies.json | 2 +-
.../hashicorp/_internal_client/test_vault_client.py | 12 ++++++------
tests/providers/hashicorp/hooks/test_vault.py | 6 +++---
tests/providers/hashicorp/secrets/test_vault.py | 4 ++--
6 files changed, 21 insertions(+), 15 deletions(-)
diff --git a/airflow/providers/hashicorp/_internal_client/vault_client.py b/airflow/providers/hashicorp/_internal_client/vault_client.py
index f8e5c254d4..0012d95802 100644
--- a/airflow/providers/hashicorp/_internal_client/vault_client.py
+++ b/airflow/providers/hashicorp/_internal_client/vault_client.py
@@ -373,7 +373,10 @@ class _VaultClient(LoggingMixin):
response = self.client.secrets.kv.v1.read_secret(path=secret_path, mount_point=mount_point)
else:
response = self.client.secrets.kv.v2.read_secret_version(
- path=secret_path, mount_point=mount_point, version=secret_version
+ path=secret_path,
+ mount_point=mount_point,
+ version=secret_version,
+ raise_on_deleted_version=True,
)
except InvalidPath:
self.log.debug("Secret not found %s with mount point %s", secret_path, mount_point)
@@ -422,7 +425,10 @@ class _VaultClient(LoggingMixin):
try:
mount_point, secret_path = self._parse_secret_path(secret_path)
return self.client.secrets.kv.v2.read_secret_version(
- path=secret_path, mount_point=mount_point, version=secret_version
+ path=secret_path,
+ mount_point=mount_point,
+ version=secret_version,
+ raise_on_deleted_version=True,
)
except InvalidPath:
self.log.debug(
diff --git a/airflow/providers/hashicorp/provider.yaml b/airflow/providers/hashicorp/provider.yaml
index ce2b3846b4..e42cad0ff9 100644
--- a/airflow/providers/hashicorp/provider.yaml
+++ b/airflow/providers/hashicorp/provider.yaml
@@ -50,7 +50,7 @@ versions:
dependencies:
- apache-airflow>=2.6.0
- - hvac>=0.10
+ - hvac>=1.1.0
integrations:
- integration-name: Hashicorp Vault
diff --git a/generated/provider_dependencies.json b/generated/provider_dependencies.json
index e9e048141c..d42ae80ee4 100644
--- a/generated/provider_dependencies.json
+++ b/generated/provider_dependencies.json
@@ -555,7 +555,7 @@
"hashicorp": {
"deps": [
"apache-airflow>=2.6.0",
- "hvac>=0.10"
+ "hvac>=1.1.0"
],
"cross-providers-deps": [
"google"
diff --git a/tests/providers/hashicorp/_internal_client/test_vault_client.py b/tests/providers/hashicorp/_internal_client/test_vault_client.py
index bb9a53ceb5..28c6944fa6 100644
--- a/tests/providers/hashicorp/_internal_client/test_vault_client.py
+++ b/tests/providers/hashicorp/_internal_client/test_vault_client.py
@@ -641,7 +641,7 @@ class TestVaultClient:
secret = vault_client.get_secret(secret_path="missing")
assert secret is None
mock_client.secrets.kv.v2.read_secret_version.assert_called_once_with(
- mount_point="secret", path="missing", version=None
+ mount_point="secret", path="missing", version=None, raise_on_deleted_version=True
)
@mock.patch("airflow.providers.hashicorp._internal_client.vault_client.hvac")
@@ -661,7 +661,7 @@ class TestVaultClient:
assert secret is None
assert "secret" == vault_client.mount_point
mock_client.secrets.kv.v2.read_secret_version.assert_called_once_with(
- mount_point="secret", path="missing", version=None
+ mount_point="secret", path="missing", version=None, raise_on_deleted_version=True
)
@mock.patch("airflow.providers.hashicorp._internal_client.vault_client.hvac")
@@ -716,7 +716,7 @@ class TestVaultClient:
secret = vault_client.get_secret(secret_path="path/to/secret")
assert {"secret_key": "secret_value"} == secret
mock_client.secrets.kv.v2.read_secret_version.assert_called_once_with(
- mount_point="secret", path="path/to/secret", version=None
+ mount_point="secret", path="path/to/secret", version=None, raise_on_deleted_version=True
)
@mock.patch("airflow.providers.hashicorp._internal_client.vault_client.hvac")
@@ -754,7 +754,7 @@ class TestVaultClient:
secret = vault_client.get_secret(secret_path="mount_point/path/to/secret")
assert {"secret_key": "secret_value"} == secret
mock_client.secrets.kv.v2.read_secret_version.assert_called_once_with(
- mount_point="mount_point", path="path/to/secret", version=None
+ mount_point="mount_point", path="path/to/secret", version=None, raise_on_deleted_version=True
)
@mock.patch("airflow.providers.hashicorp._internal_client.vault_client.hvac")
@@ -791,7 +791,7 @@ class TestVaultClient:
secret = vault_client.get_secret(secret_path="missing", secret_version=1)
assert {"secret_key": "secret_value"} == secret
mock_client.secrets.kv.v2.read_secret_version.assert_called_once_with(
- mount_point="secret", path="missing", version=1
+ mount_point="secret", path="missing", version=1, raise_on_deleted_version=True
)
@mock.patch("airflow.providers.hashicorp._internal_client.vault_client.hvac")
@@ -1015,7 +1015,7 @@ class TestVaultClient:
"auth": None,
} == metadata
mock_client.secrets.kv.v2.read_secret_version.assert_called_once_with(
- mount_point="secret", path="missing", version=None
+ mount_point="secret", path="missing", version=None, raise_on_deleted_version=True
)
@mock.patch("airflow.providers.hashicorp._internal_client.vault_client.hvac")
diff --git a/tests/providers/hashicorp/hooks/test_vault.py b/tests/providers/hashicorp/hooks/test_vault.py
index 29bace0642..b9db1e7c1f 100644
--- a/tests/providers/hashicorp/hooks/test_vault.py
+++ b/tests/providers/hashicorp/hooks/test_vault.py
@@ -1005,7 +1005,7 @@ class TestVaultHook:
secret = test_hook.get_secret(secret_path="missing")
assert {"secret_key": "secret_value"} == secret
mock_client.secrets.kv.v2.read_secret_version.assert_called_once_with(
- mount_point="secret", path="missing", version=None
+ mount_point="secret", path="missing", version=None, raise_on_deleted_version=True
)
@mock.patch("airflow.providers.hashicorp.hooks.vault.VaultHook.get_connection")
@@ -1044,7 +1044,7 @@ class TestVaultHook:
secret = test_hook.get_secret(secret_path="missing", secret_version=1)
assert {"secret_key": "secret_value"} == secret
mock_client.secrets.kv.v2.read_secret_version.assert_called_once_with(
- mount_point="secret", path="missing", version=1
+ mount_point="secret", path="missing", version=1, raise_on_deleted_version=True
)
@mock.patch("airflow.providers.hashicorp.hooks.vault.VaultHook.get_connection")
@@ -1189,7 +1189,7 @@ class TestVaultHook:
"auth": None,
} == metadata
mock_client.secrets.kv.v2.read_secret_version.assert_called_once_with(
- mount_point="secret", path="missing", version=None
+ mount_point="secret", path="missing", version=None, raise_on_deleted_version=True
)
@mock.patch("airflow.providers.hashicorp.hooks.vault.VaultHook.get_connection")
diff --git a/tests/providers/hashicorp/secrets/test_vault.py b/tests/providers/hashicorp/secrets/test_vault.py
index 4897a73c22..fc30da9add 100644
--- a/tests/providers/hashicorp/secrets/test_vault.py
+++ b/tests/providers/hashicorp/secrets/test_vault.py
@@ -302,7 +302,7 @@ class TestVaultSecrets:
test_client = VaultBackend(**kwargs)
assert test_client.get_conn_uri(conn_id="test_mysql") is None
mock_client.secrets.kv.v2.read_secret_version.assert_called_once_with(
- mount_point="airflow", path="connections/test_mysql", version=None
+ mount_point="airflow", path="connections/test_mysql", version=None, raise_on_deleted_version=True
)
assert test_client.get_connection(conn_id="test_mysql") is None
@@ -454,7 +454,7 @@ class TestVaultSecrets:
test_client = VaultBackend(**kwargs)
assert test_client.get_variable("hello") is None
mock_client.secrets.kv.v2.read_secret_version.assert_called_once_with(
- mount_point="airflow", path="variables/hello", version=None
+ mount_point="airflow", path="variables/hello", version=None, raise_on_deleted_version=True
)
assert test_client.get_variable("hello") is None
(airflow) 02/20: Tracking airflow-github changelog activities using progress bar (#36610)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit eee1cd7fcc1e6d66509041f8d354642d6145e791
Author: Amogh Desai <am...@gmail.com>
AuthorDate: Fri Jan 5 18:30:39 2024 +0530
Tracking airflow-github changelog activities using progress bar (#36610)
* Tracking airflow-github changelog activities using progress bar
* review comments from ephraimbuddy
(cherry picked from commit b3b8ae99ca1ef0524854148ca8a549232e2f3ae0)
---
dev/airflow-github | 41 ++++++++++++++++++++++++-----------------
1 file changed, 24 insertions(+), 17 deletions(-)
diff --git a/dev/airflow-github b/dev/airflow-github
index 7d4ad9ebb9..4b33802aef 100755
--- a/dev/airflow-github
+++ b/dev/airflow-github
@@ -31,6 +31,8 @@ import git
import rich_click as click
from github import Github
from packaging import version
+from rich.console import Console
+from rich.progress import Progress
if TYPE_CHECKING:
from github.Issue import Issue
@@ -40,7 +42,6 @@ GIT_COMMIT_FIELDS = ["id", "author_name", "author_email", "date", "subject", "bo
GIT_LOG_FORMAT = "%x1f".join(["%h", "%an", "%ae", "%ad", "%s", "%b"]) + "%x1e"
pr_title_re = re.compile(r".*\((#[0-9]{1,6})\)$")
-
STATUS_COLOR_MAP = {
"Closed": "green",
"Open": "red",
@@ -318,13 +319,11 @@ def compare(target_version, github_token, previous_version=None, show_uncherrypi
@click.argument("previous_version")
@click.argument("target_version")
@click.argument("github-token", envvar="GITHUB_TOKEN")
-def changelog(previous_version, target_version, github_token):
+@click.option("--disable-progress", is_flag=True, help="Disable the progress bar")
+def changelog(previous_version, target_version, github_token, disable_progress):
repo = git.Repo(".", search_parent_directories=True)
# Get a list of issues/PRs that have been committed on the current branch.
log = get_commits_between(repo, previous_version, target_version)
-
- print("Number of commits", len(log))
-
gh = Github(github_token)
gh_repo = gh.get_repo("apache/airflow")
sections = defaultdict(list)
@@ -335,18 +334,26 @@ def changelog(previous_version, target_version, github_token):
if match:
existing_commits.add(match.group(1))
- for commit in log:
- tickets = pr_title_re.findall(commit["subject"])
- if tickets:
- issue = gh_repo.get_issue(number=int(tickets[0][1:]))
- issue_type = get_issue_type(issue)
- files = files_touched(repo, commit["id"])
- if commit["id"] in existing_commits:
- continue
- if is_core_commit(files):
- sections[issue_type].append(commit["subject"])
- else:
- sections[DEFAULT_SECTION_NAME].append(commit["subject"])
+ console = Console(width=180)
+
+ with Progress(console=console, disable=disable_progress) as progress:
+ task = progress.add_task("Processing commits from changelog", total=len(log))
+ for commit in progress.track(log, description="Processing commits from changelog"):
+ tickets = pr_title_re.findall(commit["subject"])
+ if tickets:
+ issue = gh_repo.get_issue(number=int(tickets[0][1:]))
+ issue_type = get_issue_type(issue)
+ files = files_touched(repo, commit["id"])
+ if commit["id"] in existing_commits:
+ continue
+ if is_core_commit(files):
+ sections[issue_type].append(commit["subject"])
+ progress.update(task, advance=1)
+ else:
+ sections[DEFAULT_SECTION_NAME].append(commit["subject"])
+ progress.update(task, advance=1)
+
+ console.print("\n")
print_changelog(sections)
(airflow) 05/20: Add Airflow logo to the README (#36645)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 5fc90253f50dddfa25b2ba3cfaa40676ec69f932
Author: Kaxil Naik <ka...@apache.org>
AuthorDate: Sun Jan 7 18:47:45 2024 +0530
Add Airflow logo to the README (#36645)
Similar to Apache Superset, adding this repo to README so that Airflow repo shows up nicely with thumbnails on various Topics pages like https://github.com/topics/airflow
(cherry picked from commit 9a20aa8f0fe55f6dc2b6f94b8b41d36108bdd0d7)
---
README.md | 7 +++++++
generated/PYPI_README.md | 7 +++++++
2 files changed, 14 insertions(+)
diff --git a/README.md b/README.md
index 0b8ceef072..d55d4e8814 100644
--- a/README.md
+++ b/README.md
@@ -35,6 +35,13 @@
[](https://github.com/apache/airflow/graphs/contributors)
[](https://ossrank.com/p/6)
+<picture width="500">
+ <img
+ src="https://github.com/apache/airflow/blob/19ebcac2395ef9a6b6ded3a2faa29dc960c1e635/docs/apache-airflow/img/logos/wordmark_1.png?raw=true"
+ alt="Apache Airflow logo"
+ />
+</picture>
+
[Apache Airflow](https://airflow.apache.org/docs/apache-airflow/stable/) (or simply Airflow) is a platform to programmatically author, schedule, and monitor workflows.
When workflows are defined as code, they become more maintainable, versionable, testable, and collaborative.
diff --git a/generated/PYPI_README.md b/generated/PYPI_README.md
index 105bf11f67..d11eb11cc5 100644
--- a/generated/PYPI_README.md
+++ b/generated/PYPI_README.md
@@ -37,6 +37,13 @@ PROJECT BY THE `generate-pypi-readme` PRE-COMMIT. YOUR CHANGES HERE WILL BE AUTO
[](https://github.com/apache/airflow/graphs/contributors)
[](https://ossrank.com/p/6)
+<picture width="500">
+ <img
+ src="https://github.com/apache/airflow/blob/19ebcac2395ef9a6b6ded3a2faa29dc960c1e635/docs/apache-airflow/img/logos/wordmark_1.png?raw=true"
+ alt="Apache Airflow logo"
+ />
+</picture>
+
[Apache Airflow](https://airflow.apache.org/docs/apache-airflow/stable/) (or simply Airflow) is a platform to programmatically author, schedule, and monitor workflows.
When workflows are defined as code, they become more maintainable, versionable, testable, and collaborative.