You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ep...@apache.org on 2023/11/23 07:28:40 UTC
(airflow) branch v2-8-test updated (d6c7d33d43 -> 5bbc46005f)
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a change to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
omit d6c7d33d43 Update RELEASE_NOTES.rst
omit 729c1df9eb Update version to 2.8.0
new 1fc0633f89 Add v2-8 branches to codecov.yml and .asf.yaml (#35750)
new 70e2f419e6 Update minor release command (#35751)
new 6198558843 Remove backcompat with Airflow 2.3/2.4 in providers (#35727)
new b88d0d7ce2 OpenLineage integration tried to use non-existed method in SnowflakeHook (#35752)
new 25990d159c Remove usage of deprecated method from BigQueryToBigQueryOperator (#35605)
new bd644fab3e feature(providers): added `OpsgenieNotifier` (#35530)
new 38afe9ffb1 Rename --aditional-extras flag to --aditional-airflow-extras (#35760)
new 9b832f6acd More detail on mandatory task arguments (#35740)
new 7642b29f5c Update README.md to reflect changes we agreed to the versioning (#35764)
new 5d69fc142d Add basic metrics to stats collector. (#35368)
new d9cfdd8131 Extend task context logging support for remote logging using Elasticsearch (#32977)
new 7310693277 Remove backcompat inheritance for DbApiHook (#35754)
new 26d5e3f4e7 Improve ownership fixing for Breeze (#35759)
new 1df306a337 Fix broken link to Weaviate docs (#35776)
new f40e1c17ee Set mark_end_on_close after set_context (#35761)
new c0a1dfe9ff Make passing build args explicit in ci/prod builds (#35768)
new 4454fc870c Move `BaseOperatorLink` into the separate module (#35032)
new b51aaf59d2 Add OpenLineage support to GCSToBigQueryOperator (#35778)
new e2e89668d4 Implement login and logout in AWS auth manager (#35488)
new feaeb8c5fb Check attr on parent not self re TaskContextLogger set_context (#35780)
new 528d2bc51b Remove --force-build command in cache steps in CI (#35784)
new 599189e41e Fix DataFusion example type annotations (#35753)
new b904523c5a Fix permission check on menus (#35781)
new 2a240e65d5 Remove pendulum as dependency of breeze (#35786)
new 58743f28d5 Reflect drop/add support of DB Backends versions in documentation (#35785)
new e0736092a8 Update emr.py (#35787)
new 7937cba0dc Add missing docker test_exceptions.py (#35674)
new 08785f50f1 added Topic params for schema_settings and message_retention_duration. (#35767)
new 0e6b5d8f8e Added retry strategy parameter to Amazon AWS provider Batch Operator to allow dynamic Batch retry strategies (#35789)
new 673f7f837f Fix for infinite recursion due to secrets_masker (#35048)
new d568543d87 feat: K8S resource operator - CRD (#35600)
new b331cb2e0e Upgrade to Pydantic v2 (#35551)
new e022710039 Fix HttpOperator pagination with `str` data (#35782)
new 840707cedf Updated docstring: `check_key_async` is now in line with description of `_check_key_async` (#35799)
new 55914e14cb Add borderWidthRight to grid for Firefox scrollbar (#35346)
new 20cb92e52a Update version to 2.8.0
new 5bbc46005f Update RELEASE_NOTES.rst
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (d6c7d33d43)
\
N -- N -- N refs/heads/v2-8-test (5bbc46005f)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 37 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.asf.yaml | 3 +
.github/actions/build-ci-images/action.yml | 4 -
.github/actions/build-prod-images/action.yml | 4 -
.github/actions/post_tests_failure/action.yml | 3 -
.github/actions/post_tests_success/action.yml | 3 -
.github/workflows/build-images.yml | 3 -
.github/workflows/ci.yml | 55 ---
.github/workflows/release_dockerhub_image.yml | 3 -
.pre-commit-config.yaml | 23 +-
BREEZE.rst | 2 +-
CONTRIBUTING.rst | 6 +-
Dockerfile | 4 +-
Dockerfile.ci | 6 +-
IMAGES.rst | 10 +-
INSTALL | 6 +-
README.md | 17 +-
RELEASE_NOTES.rst | 7 +
STATIC_CODE_CHECKS.rst | 3 +-
airflow/auth/managers/base_auth_manager.py | 32 +-
airflow/auth/managers/fab/fab_auth_manager.py | 16 +-
airflow/auth/managers/fab/models/__init__.py | 3 +
airflow/auth/managers/models/base_user.py | 7 +-
airflow/configuration.py | 4 -
airflow/models/__init__.py | 5 +-
airflow/models/abstractoperator.py | 3 +-
airflow/models/baseoperator.py | 49 ++-
airflow/models/baseoperatorlink.py | 57 ++++
airflow/models/mappedoperator.py | 3 +-
airflow/operators/trigger_dagrun.py | 3 +-
.../amazon/aws/auth_manager}/__init__.py | 0
.../amazon/aws/auth_manager/aws_auth_manager.py | 143 ++++++++
.../providers/amazon/aws/auth_manager/constants.py | 10 +-
.../aws/auth_manager/security_manager}/__init__.py | 0
.../aws_security_manager_override.py} | 25 +-
airflow/providers/amazon/aws/auth_manager/user.py | 51 +++
.../amazon/aws/auth_manager/views}/__init__.py | 0
.../amazon/aws/auth_manager/views/auth.py | 149 ++++++++
airflow/providers/amazon/aws/hooks/s3.py | 6 +-
.../providers/amazon/aws/log/s3_task_handler.py | 4 +-
airflow/providers/amazon/aws/operators/batch.py | 5 +
airflow/providers/amazon/aws/operators/emr.py | 18 +-
airflow/providers/amazon/provider.yaml | 25 ++
airflow/providers/apache/impala/hooks/impala.py | 3 +-
.../cncf/kubernetes/operators/resource.py | 62 +++-
.../cncf/kubernetes/utils/k8s_resource_iterator.py | 46 +++
airflow/providers/common/sql/hooks/sql.py | 18 +-
airflow/providers/common/sql/hooks/sql.pyi | 4 +-
.../providers/elasticsearch/hooks/elasticsearch.py | 4 +-
.../providers/elasticsearch/log/es_task_handler.py | 48 ++-
airflow/providers/google/cloud/hooks/gcs.py | 9 +-
airflow/providers/google/cloud/hooks/pubsub.py | 10 +
.../providers/google/cloud/log/gcs_task_handler.py | 4 +-
airflow/providers/google/cloud/operators/pubsub.py | 7 +
.../google/cloud/secrets/secret_manager.py | 14 +-
.../google/cloud/transfers/bigquery_to_bigquery.py | 88 +++--
.../google/cloud/transfers/gcs_to_bigquery.py | 77 +++++
airflow/providers/http/operators/http.py | 50 ++-
airflow/providers/http/triggers/http.py | 2 +-
.../microsoft/azure/log/wasb_task_handler.py | 4 +-
.../providers/microsoft/azure/secrets/key_vault.py | 14 +-
airflow/providers/openlineage/plugins/adapter.py | 5 +-
.../opsgenie/notifications}/__init__.py | 0
.../providers/opsgenie/notifications/opsgenie.py | 81 +++++
airflow/providers/opsgenie/provider.yaml | 3 +
.../opsgenie/typing}/__init__.py | 0
airflow/providers/opsgenie/typing/opsgenie.py | 59 ++++
airflow/providers/snowflake/hooks/snowflake.py | 2 +-
airflow/providers/trino/hooks/trino.py | 19 +-
airflow/sensors/external_task.py | 2 +-
airflow/serialization/pydantic/dag.py | 29 +-
airflow/serialization/pydantic/dag_run.py | 9 +-
airflow/serialization/pydantic/dataset.py | 27 +-
airflow/serialization/pydantic/job.py | 8 +-
airflow/serialization/pydantic/taskinstance.py | 9 +-
airflow/serialization/serde.py | 9 +-
airflow/serialization/serialized_objects.py | 7 +-
airflow/utils/log/secrets_masker.py | 9 +-
airflow/utils/log/task_context_logger.py | 2 +-
airflow/www/extensions/init_appbuilder.py | 2 +-
airflow/www/extensions/init_auth_manager.py | 6 +-
airflow/www/extensions/init_security.py | 4 +-
airflow/www/security_manager.py | 56 +--
airflow/www/static/js/dag/grid/index.tsx | 2 +-
airflow/www/utils.py | 5 +-
codecov.yml | 2 +
dev/breeze/README.md | 2 +-
...-root-ownership-after-exiting-docker-command.md | 60 ++++
dev/breeze/pyproject.toml | 1 -
.../airflow_breeze/commands/ci_image_commands.py | 119 +++++--
.../commands/ci_image_commands_config.py | 29 +-
.../airflow_breeze/commands/developer_commands.py | 7 +
.../commands/minor_release_command.py | 1 +
.../commands/production_image_commands.py | 160 +++++++--
.../commands/production_image_commands_config.py | 46 ++-
.../commands/release_management_commands.py | 21 ++
.../airflow_breeze/commands/testing_commands.py | 5 +
.../src/airflow_breeze/params/build_ci_params.py | 113 +++---
.../src/airflow_breeze/params/build_prod_params.py | 115 +++----
.../airflow_breeze/params/common_build_params.py | 84 +++--
.../src/airflow_breeze/utils/common_options.py | 14 +-
.../airflow_breeze/utils/docker_command_utils.py | 59 +---
dev/breeze/src/airflow_breeze/utils/parallel.py | 6 +
dev/breeze/src/airflow_breeze/utils/run_utils.py | 2 +-
docs/apache-airflow-providers-opsgenie/index.rst | 1 +
.../notifications/index.rst | 30 ++
.../notifications/opsgenie_notifier.rst | 33 ++
.../connections.rst | 2 +-
.../administration-and-deployment/scheduler.rst | 14 +-
docs/apache-airflow/extra-packages-ref.rst | 2 +
docs/apache-airflow/howto/define-extra-link.rst | 9 +-
docs/apache-airflow/howto/set-up-database.rst | 10 +-
docs/apache-airflow/installation/prerequisites.rst | 11 +-
docs/apache-airflow/public-airflow-interface.rst | 7 +
docs/apache-airflow/tutorial/fundamentals.rst | 7 +-
docs/conf.py | 1 +
docs/docker-stack/changelog.rst | 3 +
docs/spelling_wordlist.txt | 1 +
generated/PYPI_README.md | 2 +-
images/breeze/output_ci-image_build.svg | 160 +++++----
images/breeze/output_ci-image_build.txt | 2 +-
images/breeze/output_prod-image_build.svg | 190 ++++++-----
images/breeze/output_prod-image_build.txt | 2 +-
..._release-management_prepare-airflow-package.txt | 2 +-
...elease-management_prepare-provider-packages.txt | 2 +-
pyproject.toml | 1 +
scripts/docker/entrypoint_ci.sh | 2 -
scripts/docker/install_os_dependencies.sh | 4 +-
scripts/in_container/_in_container_utils.sh | 40 ---
scripts/in_container/run_fix_ownership.py | 84 +++++
.../in_container/run_prepare_airflow_packages.py | 12 +-
setup.cfg | 6 +-
setup.py | 7 +
tests/always/test_project_structure.py | 1 -
.../endpoints/test_extra_link_endpoint.py | 2 +-
.../endpoints/test_forward_to_fab_endpoint.py | 2 +-
.../endpoints/test_plugin_endpoint.py | 2 +-
tests/api_connexion/schemas/test_plugin_schema.py | 2 +-
tests/auth/managers/fab/test_fab_auth_manager.py | 38 +--
tests/auth/managers/test_base_auth_manager.py | 48 ++-
.../providers/amazon/aws/auth_manager}/__init__.py | 0
.../aws/auth_manager/security_manager}/__init__.py | 0
.../test_aws_security_manager_override.py | 56 +++
.../aws/auth_manager/test_aws_auth_manager.py | 114 +++++++
.../amazon/aws/auth_manager/test_constants.py | 23 +-
.../providers/amazon/aws/auth_manager/test_user.py | 32 +-
.../amazon/aws/auth_manager/views}/__init__.py | 0
.../amazon/aws/auth_manager/views/test_auth.py | 156 +++++++++
tests/providers/amazon/aws/operators/test_batch.py | 6 +
.../cncf/kubernetes/operators/test_resource.py | 63 ++++
.../kubernetes/utils/test_k8s_resource_iterator.py | 65 ++++
tests/providers/docker/test_exceptions.py | 80 +++++
tests/providers/google/cloud/hooks/test_pubsub.py | 9 +-
.../google/cloud/operators/test_pubsub.py | 4 +
.../cloud/transfers/test_bigquery_to_bigquery.py | 94 +++--
.../google/cloud/transfers/test_gcs_to_bigquery.py | 380 ++++++++++++++++++++-
tests/providers/http/operators/test_http.py | 63 +++-
tests/providers/odbc/hooks/test_odbc.py | 3 +-
.../plugins/test_openlineage_adapter.py | 75 +++-
.../providers/opsgenie/notifications}/__init__.py | 0
.../opsgenie/notifications/test_opsgenie.py | 119 +++++++
.../providers/opsgenie/typing}/__init__.py | 0
.../providers/opsgenie/typing/test_opsgenie.py | 16 +-
tests/providers/trino/hooks/test_trino.py | 38 ++-
tests/serialization/test_dag_serialization.py | 3 +-
.../google/cloud/datafusion/example_datafusion.py | 4 +-
.../opsgenie/example_opsgenie_notifier.py | 31 +-
tests/test_utils/mock_operators.py | 3 +-
tests/utils/log/test_secrets_masker.py | 18 +
tests/www/test_security_manager.py | 115 +++++++
tests/www/views/test_views_extra_links.py | 3 +-
170 files changed, 3609 insertions(+), 1151 deletions(-)
create mode 100644 airflow/models/baseoperatorlink.py
copy airflow/{api_connexion => providers/amazon/aws/auth_manager}/__init__.py (100%)
create mode 100644 airflow/providers/amazon/aws/auth_manager/aws_auth_manager.py
rename scripts/in_container/run_fix_ownership.sh => airflow/providers/amazon/aws/auth_manager/constants.py (82%)
mode change 100755 => 100644
copy airflow/{api_connexion => providers/amazon/aws/auth_manager/security_manager}/__init__.py (100%)
copy airflow/{auth/managers/models/base_user.py => providers/amazon/aws/auth_manager/security_manager/aws_security_manager_override.py} (52%)
create mode 100644 airflow/providers/amazon/aws/auth_manager/user.py
copy airflow/{api_connexion => providers/amazon/aws/auth_manager/views}/__init__.py (100%)
create mode 100644 airflow/providers/amazon/aws/auth_manager/views/auth.py
create mode 100644 airflow/providers/cncf/kubernetes/utils/k8s_resource_iterator.py
copy airflow/{api_connexion => providers/opsgenie/notifications}/__init__.py (100%)
create mode 100644 airflow/providers/opsgenie/notifications/opsgenie.py
copy airflow/{api_connexion => providers/opsgenie/typing}/__init__.py (100%)
create mode 100644 airflow/providers/opsgenie/typing/opsgenie.py
create mode 100644 dev/breeze/doc/adr/0014-fix-root-ownership-after-exiting-docker-command.md
create mode 100644 docs/apache-airflow-providers-opsgenie/notifications/index.rst
create mode 100644 docs/apache-airflow-providers-opsgenie/notifications/opsgenie_notifier.rst
create mode 100755 scripts/in_container/run_fix_ownership.py
copy {airflow/api_connexion => tests/providers/amazon/aws/auth_manager}/__init__.py (100%)
copy {airflow/api_connexion => tests/providers/amazon/aws/auth_manager/security_manager}/__init__.py (100%)
create mode 100644 tests/providers/amazon/aws/auth_manager/security_manager/test_aws_security_manager_override.py
create mode 100644 tests/providers/amazon/aws/auth_manager/test_aws_auth_manager.py
copy airflow/auth/managers/models/base_user.py => tests/providers/amazon/aws/auth_manager/test_constants.py (64%)
copy airflow/auth/managers/models/base_user.py => tests/providers/amazon/aws/auth_manager/test_user.py (55%)
copy {airflow/api_connexion => tests/providers/amazon/aws/auth_manager/views}/__init__.py (100%)
create mode 100644 tests/providers/amazon/aws/auth_manager/views/test_auth.py
create mode 100644 tests/providers/cncf/kubernetes/utils/test_k8s_resource_iterator.py
create mode 100644 tests/providers/docker/test_exceptions.py
copy {airflow/api_connexion => tests/providers/opsgenie/notifications}/__init__.py (100%)
create mode 100644 tests/providers/opsgenie/notifications/test_opsgenie.py
copy {airflow/api_connexion => tests/providers/opsgenie/typing}/__init__.py (100%)
copy airflow/auth/managers/models/base_user.py => tests/providers/opsgenie/typing/test_opsgenie.py (74%)
copy airflow/auth/managers/models/base_user.py => tests/system/providers/opsgenie/example_opsgenie_notifier.py (51%)
create mode 100644 tests/www/test_security_manager.py
(airflow) 33/37: Fix HttpOperator pagination with `str` data (#35782)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit e02271003931c2d9df58a2b769c89cac8d4eb0a2
Author: Joffrey Bienvenu <jo...@infrabel.be>
AuthorDate: Wed Nov 22 15:34:34 2023 +0100
Fix HttpOperator pagination with `str` data (#35782)
* feat: Restrict `data` parameter typing
Follows the hook's typing
* feat: Implement `data` override when string
* feat: Improve docstring about merging and overriding behavior
* fix: Add correct typing for mypy
* feat: add test
* fix: remove unused imports
* fix: Update SimpleHttpOperator docstring
* feat: Correctly test parameters overriding
---
airflow/providers/http/operators/http.py | 50 +++++++++++++++--------
airflow/providers/http/triggers/http.py | 2 +-
tests/providers/http/operators/test_http.py | 63 ++++++++++++++++++++++-------
3 files changed, 84 insertions(+), 31 deletions(-)
diff --git a/airflow/providers/http/operators/http.py b/airflow/providers/http/operators/http.py
index 96415ed977..524de8c585 100644
--- a/airflow/providers/http/operators/http.py
+++ b/airflow/providers/http/operators/http.py
@@ -56,12 +56,17 @@ class HttpOperator(BaseOperator):
:param pagination_function: A callable that generates the parameters used to call the API again,
based on the previous response. Typically used when the API is paginated and returns for e.g a
cursor, a 'next page id', or a 'next page URL'. When provided, the Operator will call the API
- repeatedly until this callable returns None. Also, the result of the Operator will become by
- default a list of Response.text objects (instead of a single response object). Same with the
- other injected functions (like response_check, response_filter, ...) which will also receive a
- list of Response object. This function receives a Response object form previous call, and should
- return a dict of parameters (`endpoint`, `data`, `headers`, `extra_options`), which will be merged
- and will override the one used in the initial API call.
+ repeatedly until this callable returns None. The result of the Operator will become by default a
+ list of Response.text objects (instead of a single response object). Same with the other injected
+ functions (like response_check, response_filter, ...) which will also receive a list of Response
+ objects. This function receives a Response object form previous call, and should return a nested
+ dictionary with the following optional keys: `endpoint`, `data`, `headers` and `extra_options.
+ Those keys will be merged and/or override the parameters provided into the HttpOperator declaration.
+ Parameters are merged when they are both a dictionary (e.g.: HttpOperator.headers will be merged
+ with the `headers` dict provided by this function). When merging, dict items returned by this
+ function will override initial ones (e.g: if both HttpOperator.headers and `headers` have a 'cookie'
+ item, the one provided by `headers` is kept). Parameters are simply overridden when any of them are
+ string (e.g.: HttpOperator.endpoint is overridden by `endpoint`).
:param response_check: A check against the 'requests' response object.
The callable takes the response object as the first positional argument
and optionally any number of keyword arguments available in the context dictionary.
@@ -101,7 +106,7 @@ class HttpOperator(BaseOperator):
*,
endpoint: str | None = None,
method: str = "POST",
- data: Any = None,
+ data: dict[str, Any] | str | None = None,
headers: dict[str, str] | None = None,
pagination_function: Callable[..., Any] | None = None,
response_check: Callable[..., bool] | None = None,
@@ -271,9 +276,16 @@ class HttpOperator(BaseOperator):
:param next_page_params: A dictionary containing the parameters for the next page.
:return: A dictionary containing the merged parameters.
"""
+ data: str | dict | None = None # makes mypy happy
+ next_page_data_param = next_page_params.get("data")
+ if isinstance(self.data, dict) and isinstance(next_page_data_param, dict):
+ data = merge_dicts(self.data, next_page_data_param)
+ else:
+ data = next_page_data_param or self.data
+
return dict(
endpoint=next_page_params.get("endpoint") or self.endpoint,
- data=merge_dicts(self.data, next_page_params.get("data", {})),
+ data=data,
headers=merge_dicts(self.headers, next_page_params.get("headers", {})),
extra_options=merge_dicts(self.extra_options, next_page_params.get("extra_options", {})),
)
@@ -294,14 +306,20 @@ class SimpleHttpOperator(HttpOperator):
:param data: The data to pass. POST-data in POST/PUT and params
in the URL for a GET request. (templated)
:param headers: The HTTP headers to be added to the GET request
- :param pagination_function: A callable that generates the parameters used to call the API again.
- Typically used when the API is paginated and returns for e.g a cursor, a 'next page id', or
- a 'next page URL'. When provided, the Operator will call the API repeatedly until this callable
- returns None. Also, the result of the Operator will become by default a list of Response.text
- objects (instead of a single response object). Same with the other injected functions (like
- response_check, response_filter, ...) which will also receive a list of Response object. This
- function should return a dict of parameters (`endpoint`, `data`, `headers`, `extra_options`),
- which will be merged and override the one used in the initial API call.
+ :param pagination_function: A callable that generates the parameters used to call the API again,
+ based on the previous response. Typically used when the API is paginated and returns for e.g a
+ cursor, a 'next page id', or a 'next page URL'. When provided, the Operator will call the API
+ repeatedly until this callable returns None. The result of the Operator will become by default a
+ list of Response.text objects (instead of a single response object). Same with the other injected
+ functions (like response_check, response_filter, ...) which will also receive a list of Response
+ objects. This function receives a Response object form previous call, and should return a nested
+ dictionary with the following optional keys: `endpoint`, `data`, `headers` and `extra_options.
+ Those keys will be merged and/or override the parameters provided into the HttpOperator declaration.
+ Parameters are merged when they are both a dictionary (e.g.: HttpOperator.headers will be merged
+ with the `headers` dict provided by this function). When merging, dict items returned by this
+ function will override initial ones (e.g: if both HttpOperator.headers and `headers` have a 'cookie'
+ item, the one provided by `headers` is kept). Parameters are simply overridden when any of them are
+ string (e.g.: HttpOperator.endpoint is overridden by `endpoint`).
:param response_check: A check against the 'requests' response object.
The callable takes the response object as the first positional argument
and optionally any number of keyword arguments available in the context dictionary.
diff --git a/airflow/providers/http/triggers/http.py b/airflow/providers/http/triggers/http.py
index 89aa9ca606..b4598984f3 100644
--- a/airflow/providers/http/triggers/http.py
+++ b/airflow/providers/http/triggers/http.py
@@ -56,7 +56,7 @@ class HttpTrigger(BaseTrigger):
method: str = "POST",
endpoint: str | None = None,
headers: dict[str, str] | None = None,
- data: Any = None,
+ data: dict[str, Any] | str | None = None,
extra_options: dict[str, Any] | None = None,
):
super().__init__()
diff --git a/tests/providers/http/operators/test_http.py b/tests/providers/http/operators/test_http.py
index 451cd93d44..dfd82a17ae 100644
--- a/tests/providers/http/operators/test_http.py
+++ b/tests/providers/http/operators/test_http.py
@@ -24,6 +24,7 @@ from unittest import mock
import pytest
from requests import Response
+from requests.models import RequestEncodingMixin
from airflow.exceptions import AirflowException, TaskDeferred
from airflow.providers.http.operators.http import HttpOperator
@@ -112,41 +113,75 @@ class TestHttpOperator:
)
assert result == "content"
- def test_paginated_responses(self, requests_mock):
+ @pytest.mark.parametrize(
+ "data, headers, extra_options, pagination_data, pagination_headers, pagination_extra_options",
+ [
+ ({"data": 1}, {"x-head": "1"}, {"verify": False}, {"data": 2}, {"x-head": "0"}, {"verify": True}),
+ ("data foo", {"x-head": "1"}, {"verify": False}, {"data": 2}, {"x-head": "0"}, {"verify": True}),
+ ("data foo", {"x-head": "1"}, {"verify": False}, "data bar", {"x-head": "0"}, {"verify": True}),
+ ({"data": 1}, {"x-head": "1"}, {"verify": False}, "data foo", {"x-head": "0"}, {"verify": True}),
+ ],
+ )
+ def test_pagination(
+ self,
+ requests_mock,
+ data,
+ headers,
+ extra_options,
+ pagination_data,
+ pagination_headers,
+ pagination_extra_options,
+ ):
"""
Test that the HttpOperator calls repetitively the API when a
pagination_function is provided, and as long as this function returns
a dictionary that override previous' call parameters.
"""
- iterations: int = 0
+ is_second_call: bool = False
def pagination_function(response: Response) -> dict | None:
"""Paginated function which returns None at the second call."""
- nonlocal iterations
- if iterations < 2:
- iterations += 1
+ nonlocal is_second_call
+ if not is_second_call:
+ is_second_call = True
return dict(
endpoint=response.json()["endpoint"],
- data={},
- headers={},
- extra_options={},
+ data=pagination_data,
+ headers=pagination_headers,
+ extra_options=pagination_extra_options,
)
return None
- requests_mock.get("http://www.example.com/foo", json={"value": 5, "endpoint": "bar"})
- requests_mock.get("http://www.example.com/bar", json={"value": 10, "endpoint": "foo"})
+ first_endpoint = requests_mock.post("http://www.example.com/1", json={"value": 5, "endpoint": "2"})
+ second_endpoint = requests_mock.post("http://www.example.com/2", json={"value": 10, "endpoint": "3"})
operator = HttpOperator(
task_id="test_HTTP_op",
- method="GET",
- endpoint="/foo",
+ method="POST",
+ endpoint="/1",
+ data=data,
+ headers=headers,
+ extra_options=extra_options,
http_conn_id="HTTP_EXAMPLE",
pagination_function=pagination_function,
response_filter=lambda resp: [entry.json()["value"] for entry in resp],
)
result = operator.execute({})
- assert result == [5, 10, 5]
- def test_async_paginated_responses(self, requests_mock):
+ # Ensure the initial call is made with parameters passed to the Operator
+ first_call = first_endpoint.request_history[0]
+ assert first_call.headers.items() >= headers.items()
+ assert first_call.body == RequestEncodingMixin._encode_params(data)
+ assert first_call.verify is extra_options["verify"]
+
+ # Ensure the second - paginated - call is made with parameters merged from the pagination function
+ second_call = second_endpoint.request_history[0]
+ assert second_call.headers.items() >= pagination_headers.items()
+ assert second_call.body == RequestEncodingMixin._encode_params(pagination_data)
+ assert second_call.verify is pagination_extra_options["verify"]
+
+ assert result == [5, 10]
+
+ def test_async_pagination(self, requests_mock):
"""
Test that the HttpOperator calls asynchronously and repetitively
the API when a pagination_function is provided, and as long as this function
(airflow) 25/37: Reflect drop/add support of DB Backends versions in documentation (#35785)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 58743f28d5d362c6f4eda9cc10c51eaecd9ad6ab
Author: Andrey Anshin <An...@taragol.is>
AuthorDate: Wed Nov 22 01:43:16 2023 +0400
Reflect drop/add support of DB Backends versions in documentation (#35785)
---
.../administration-and-deployment/scheduler.rst | 14 ++++----------
docs/apache-airflow/howto/set-up-database.rst | 10 +++++++---
docs/apache-airflow/installation/prerequisites.rst | 11 +++--------
3 files changed, 14 insertions(+), 21 deletions(-)
diff --git a/docs/apache-airflow/administration-and-deployment/scheduler.rst b/docs/apache-airflow/administration-and-deployment/scheduler.rst
index 06769d1595..a01020a686 100644
--- a/docs/apache-airflow/administration-and-deployment/scheduler.rst
+++ b/docs/apache-airflow/administration-and-deployment/scheduler.rst
@@ -118,7 +118,7 @@ This does, however, place some requirements on the Database.
Database Requirements
"""""""""""""""""""""
-The short version is that users of PostgreSQL 10+ or MySQL 8+ are all ready to go -- you can start running as
+The short version is that users of PostgreSQL 12+ or MySQL 8.0+ are all ready to go -- you can start running as
many copies of the scheduler as you like -- there is no further set up or config options needed. If you are
using a different database please read on.
@@ -134,8 +134,8 @@ UPDATE NOWAIT`` but the exact query is slightly different).
The following databases are fully supported and provide an "optimal" experience:
-- PostgreSQL 10+
-- MySQL 8+
+- PostgreSQL 12+
+- MySQL 8.0+
.. warning::
@@ -144,15 +144,9 @@ The following databases are fully supported and provide an "optimal" experience:
Without these features, running multiple schedulers is not supported and deadlock errors have been reported. MariaDB
10.6.0 and following may work appropriately with multiple schedulers, but this has not been tested.
-.. warning::
-
- MySQL 5.x does not support ``SKIP LOCKED`` or ``NOWAIT``, and additionally is more prone to deciding
- queries are deadlocked, so running with more than a single scheduler on MySQL 5.x is not supported or
- recommended.
-
.. note::
- Microsoft SQLServer has not been tested with HA.
+ Microsoft SQL Server has not been tested with HA.
.. _fine-tuning-scheduler:
diff --git a/docs/apache-airflow/howto/set-up-database.rst b/docs/apache-airflow/howto/set-up-database.rst
index 3c8f7a9bb1..b164552814 100644
--- a/docs/apache-airflow/howto/set-up-database.rst
+++ b/docs/apache-airflow/howto/set-up-database.rst
@@ -32,8 +32,8 @@ By default, Airflow uses **SQLite**, which is intended for development purposes
Airflow supports the following database engine versions, so make sure which version you have. Old versions may not support all SQL statements.
-* PostgreSQL: 11, 12, 13, 14, 15
-* MySQL: 5.7, 8
+* PostgreSQL: 12, 13, 14, 15, 16
+* MySQL: 8.0, `Innovation <https://dev.mysql.com/blog-archive/introducing-mysql-innovation-and-long-term-support-lts-versions>`_
* MSSQL (Experimental, **Discontinued soon**): 2017, 2019
* SQLite: 3.15.0+
@@ -309,7 +309,11 @@ In addition, you also should pay particular attention to MySQL's encoding. Altho
.. note::
- In strict mode, MySQL doesn't allow ``0000-00-00`` as a valid date. Then you might get errors like ``"Invalid default value for 'end_date'"`` in some cases (some Airflow tables use ``0000-00-00 00:00:00`` as timestamp field default value). To avoid this error, you could disable ``NO_ZERO_DATE`` mode on you MySQL server. Read https://stackoverflow.com/questions/9192027/invalid-default-value-for-create-date-timestamp-field for how to disable it. See `SQL Mode - NO_ZERO_DATE <https://dev [...]
+ In strict mode, MySQL doesn't allow ``0000-00-00`` as a valid date. Then you might get errors like
+ ``"Invalid default value for 'end_date'"`` in some cases (some Airflow tables use ``0000-00-00 00:00:00`` as timestamp field default value).
+ To avoid this error, you could disable ``NO_ZERO_DATE`` mode on you MySQL server.
+ Read https://stackoverflow.com/questions/9192027/invalid-default-value-for-create-date-timestamp-field for how to disable it.
+ See `SQL Mode - NO_ZERO_DATE <https://dev.mysql.com/doc/refman/8.0/en/sql-mode.html#sqlmode_no_zero_date>`__ for more information.
Setting up a MsSQL Database
---------------------------
diff --git a/docs/apache-airflow/installation/prerequisites.rst b/docs/apache-airflow/installation/prerequisites.rst
index 3b8f5c84d9..f2fefbfbaa 100644
--- a/docs/apache-airflow/installation/prerequisites.rst
+++ b/docs/apache-airflow/installation/prerequisites.rst
@@ -24,10 +24,10 @@ Airflow™ is tested with:
* Databases:
- * PostgreSQL: 11, 12, 13, 14, 15
- * MySQL: 5.7, 8.0, 8.1
+ * PostgreSQL: 12, 13, 14, 15, 16
+ * MySQL: 8.0, `Innovation <https://dev.mysql.com/blog-archive/introducing-mysql-innovation-and-long-term-support-lts-versions>`_
* SQLite: 3.15.0+
- * MSSQL(Experimental): 2017, 2019
+ * MSSQL (Experimental, **Discontinued soon**): 2017, 2019
* Kubernetes: 1.25, 1.26, 1.27, 1.28
@@ -43,11 +43,6 @@ wildly on the deployment options you have
attempts to use MariaDB as a backend and users cannot expect any community support for it
because the number of users who tried to use MariaDB for Airflow is very small.
-.. warning::
-
- MySQL 5.x versions are unable to or have limitations with
- running multiple schedulers -- please see: :doc:`/administration-and-deployment/scheduler`.
-
.. warning::
SQLite is used in Airflow tests. Do not use it in production. We recommend
using the latest stable version of SQLite for local development.
(airflow) 16/37: Make passing build args explicit in ci/prod builds (#35768)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit c0a1dfe9ff1c8eea8643227503a38734eb352cc1
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Tue Nov 21 16:03:14 2023 +0100
Make passing build args explicit in ci/prod builds (#35768)
When building hte image, breeze converts some simple parameters
passed as breeze command (with autocompletion and explanation)
into much longer and more complex set of build args that are passed
to `docker build` command. The way how passing hte args worked so far
is that it was pretty implicit:
* **kwargs were used to ingest `click` flags
* parameters found as empty/None were filtered out from these
* Build*Params dataclass was created out of such kwargs dict
* argumenst from dataclass (with some customization) were
converted to --build-arg (CAPITALIZED_PROPERTY_NAME)
This had a lot of implicitness and it was not easy to understand
whether the parameters passed were correct and how they passed
through this chain.
This change makes all the build arg much more explicit - without
kwargs and dictionary. Each CI/PROD build param has now a method
where it explicitly converts arguments into build-args - including
specifying which of those are optional (where you can actually
filter out Empty and None values) and which are required (where
an actual value is expected).
This PR also cleans up the click flags sequence and their
presence as well as the output of help command (they were grouped
with more related parameters)
---
.../airflow_breeze/commands/ci_image_commands.py | 119 +++++++++----
.../commands/ci_image_commands_config.py | 29 ++--
.../commands/production_image_commands.py | 160 +++++++++++++----
.../commands/production_image_commands_config.py | 44 +++--
.../src/airflow_breeze/params/build_ci_params.py | 113 +++++-------
.../src/airflow_breeze/params/build_prod_params.py | 115 ++++++-------
.../airflow_breeze/params/common_build_params.py | 84 +++++----
.../src/airflow_breeze/utils/common_options.py | 12 +-
.../airflow_breeze/utils/docker_command_utils.py | 42 +----
images/breeze/output_ci-image_build.svg | 160 +++++++++--------
images/breeze/output_ci-image_build.txt | 2 +-
images/breeze/output_prod-image_build.svg | 190 ++++++++++++---------
images/breeze/output_prod-image_build.txt | 2 +-
..._release-management_prepare-airflow-package.txt | 2 +-
...elease-management_prepare-provider-packages.txt | 2 +-
15 files changed, 619 insertions(+), 457 deletions(-)
diff --git a/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py b/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
index 560f3852ec..f33fdad2ef 100644
--- a/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
@@ -22,6 +22,7 @@ import signal
import subprocess
import sys
import time
+from copy import deepcopy
from functools import partial
from pathlib import Path
from typing import TYPE_CHECKING, Any, Callable
@@ -32,10 +33,10 @@ from airflow_breeze.params.build_ci_params import BuildCiParams
from airflow_breeze.utils.ci_group import ci_group
from airflow_breeze.utils.click_utils import BreezeGroup
from airflow_breeze.utils.common_options import (
+ option_additional_airflow_extras,
option_additional_dev_apt_command,
option_additional_dev_apt_deps,
option_additional_dev_apt_env,
- option_additional_extras,
option_additional_pip_install_flags,
option_additional_python_deps,
option_airflow_constraints_location,
@@ -53,7 +54,6 @@ from airflow_breeze.utils.common_options import (
option_docker_cache,
option_dry_run,
option_eager_upgrade_additional_requirements,
- option_force_build,
option_github_repository,
option_github_token,
option_image_name,
@@ -77,7 +77,7 @@ from airflow_breeze.utils.common_options import (
option_upgrade_to_newer_dependencies,
option_verbose,
option_verify,
- option_version_suffix_for_pypi,
+ option_version_suffix_for_pypi_ci,
option_wait_for_image,
)
from airflow_breeze.utils.confirm import STANDARD_TIMEOUT, Answer, user_confirm
@@ -99,7 +99,6 @@ from airflow_breeze.utils.python_versions import get_python_version_list
from airflow_breeze.utils.registry import login_to_github_docker_registry
from airflow_breeze.utils.run_tests import verify_an_image
from airflow_breeze.utils.run_utils import (
- filter_out_none,
fix_group_permissions,
instruct_build_image,
is_repo_rebased,
@@ -224,12 +223,6 @@ def get_exitcode(status: int) -> int:
@ci_image.command(name="build")
@option_python
@option_debian_version
-@option_run_in_parallel
-@option_parallelism
-@option_skip_cleanup
-@option_debug_resources
-@option_include_success_outputs
-@option_python_versions
@option_upgrade_to_newer_dependencies
@option_upgrade_on_failure
@option_platform_multiple
@@ -239,18 +232,16 @@ def get_exitcode(status: int) -> int:
@option_prepare_buildx_cache
@option_push
@option_install_providers_from_sources
-@option_additional_extras
+@option_additional_airflow_extras
@option_additional_dev_apt_deps
@option_additional_python_deps
@option_additional_dev_apt_command
@option_additional_dev_apt_env
@option_builder
@option_build_progress
-@option_build_timeout_minutes
@option_commit_sha
@option_dev_apt_command
@option_dev_apt_deps
-@option_force_build
@option_python_image
@option_eager_upgrade_additional_requirements
@option_airflow_constraints_location
@@ -259,19 +250,58 @@ def get_exitcode(status: int) -> int:
@option_tag_as_latest
@option_additional_pip_install_flags
@option_github_repository
-@option_version_suffix_for_pypi
+@option_version_suffix_for_pypi_ci
+@option_build_timeout_minutes
+@option_run_in_parallel
+@option_parallelism
+@option_skip_cleanup
+@option_debug_resources
+@option_include_success_outputs
+@option_python_versions
@option_verbose
@option_dry_run
@option_answer
def build(
+ # Build options
+ python: str,
+ debian_version: str,
+ upgrade_to_newer_dependencies: bool,
+ upgrade_on_failure: bool,
+ platform: str | None,
+ github_token: str | None,
+ docker_cache: str,
+ image_tag: str,
+ prepare_buildx_cache: bool,
+ push: bool,
+ install_providers_from_sources: bool,
+ additional_airflow_extras: str | None,
+ additional_dev_apt_deps: str | None,
+ additional_python_deps: str | None,
+ additional_dev_apt_command: str | None,
+ additional_dev_apt_env: str | None,
+ builder: str,
+ build_progress: str,
+ commit_sha: str | None,
+ dev_apt_command: str | None,
+ dev_apt_deps: str | None,
+ eager_upgrade_additional_requirements: str | None,
+ airflow_constraints_location: str | None,
+ airflow_constraints_mode: str,
+ airflow_constraints_reference: str,
+ tag_as_latest: bool,
+ additional_pip_install_flags: str | None,
+ github_repository: str,
+ python_image: str | None,
+ version_suffix_for_pypi: str,
+ # Parallel building
run_in_parallel: bool,
parallelism: int,
skip_cleanup: bool,
debug_resources: bool,
include_success_outputs,
python_versions: str,
+ # Other options
build_timeout_minutes: int | None,
- **kwargs: dict[str, Any],
):
"""Build CI image. Include building multiple images for all python versions."""
@@ -306,16 +336,51 @@ def build(
perform_environment_checks()
check_remote_ghcr_io_commands()
- parameters_passed = filter_out_none(**kwargs)
- parameters_passed["force_build"] = True
fix_group_permissions()
+ base_build_params = BuildCiParams(
+ force_build=True,
+ python=python,
+ debian_version=debian_version,
+ upgrade_to_newer_dependencies=upgrade_to_newer_dependencies,
+ upgrade_on_failure=upgrade_on_failure,
+ github_token=github_token,
+ docker_cache=docker_cache,
+ image_tag=image_tag,
+ prepare_buildx_cache=prepare_buildx_cache,
+ push=push,
+ install_providers_from_sources=install_providers_from_sources,
+ additional_airflow_extras=additional_airflow_extras,
+ additional_python_deps=additional_python_deps,
+ additional_dev_apt_command=additional_dev_apt_command,
+ additional_dev_apt_env=additional_dev_apt_env,
+ builder=builder,
+ build_progress=build_progress,
+ commit_sha=commit_sha,
+ dev_apt_command=dev_apt_command,
+ dev_apt_deps=dev_apt_deps,
+ eager_upgrade_additional_requirements=eager_upgrade_additional_requirements,
+ airflow_constraints_location=airflow_constraints_location,
+ airflow_constraints_mode=airflow_constraints_mode,
+ airflow_constraints_reference=airflow_constraints_reference,
+ tag_as_latest=tag_as_latest,
+ additional_pip_install_flags=additional_pip_install_flags,
+ github_repository=github_repository,
+ python_image=python_image,
+ version_suffix_for_pypi=version_suffix_for_pypi,
+ )
+ if platform:
+ base_build_params.platform = platform
+ if additional_dev_apt_deps:
+ # For CI image we only set additional_dev_apt_deps when we explicitly pass it
+ base_build_params.additional_dev_apt_deps = additional_dev_apt_deps
+
if run_in_parallel:
python_version_list = get_python_version_list(python_versions)
params_list: list[BuildCiParams] = []
for python in python_version_list:
- params = BuildCiParams(**parameters_passed)
- params.python = python
- params_list.append(params)
+ build_params = deepcopy(base_build_params)
+ build_params.python = python
+ params_list.append(build_params)
prepare_for_building_ci_image(params=params_list[0])
run_build_in_parallel(
image_params_list=params_list,
@@ -326,9 +391,8 @@ def build(
debug_resources=debug_resources,
)
else:
- params = BuildCiParams(**parameters_passed)
- prepare_for_building_ci_image(params=params)
- run_build(ci_image_params=params)
+ prepare_for_building_ci_image(params=base_build_params)
+ run_build(ci_image_params=base_build_params)
@ci_image.command(name="pull")
@@ -550,13 +614,6 @@ def run_build_ci_image(
:param ci_image_params: CI image parameters
:param output: output redirection
"""
- if not ci_image_params.version_suffix_for_pypi:
- # We need that to handle the >= 2.7.0 limit we have for openlineage provider at least until
- # Airflow 2.7.0 release is out, in order to avoid conflicting dependencies while building the image
- # We are setting version_suffix_for_pypi to dev0 for CI builds where cache is prepared, so in
- # order to have the cache used effectively, we should also locally force the version_suffix_for_pypi
- # to dev0. We might evan leave it as default value in the future (to be decided after 2.7.0 release)
- ci_image_params.version_suffix_for_pypi = "dev0"
if (
ci_image_params.is_multi_platform()
and not ci_image_params.push
@@ -645,8 +702,6 @@ def rebuild_or_pull_ci_image_if_needed(command_params: ShellParams | BuildCiPara
Rebuilds CI image if needed and user confirms it.
:param command_params: parameters of the command to execute
-
-
"""
build_ci_image_check_cache = Path(
BUILD_CACHE_DIR, command_params.airflow_branch, f".built_{command_params.python}"
diff --git a/dev/breeze/src/airflow_breeze/commands/ci_image_commands_config.py b/dev/breeze/src/airflow_breeze/commands/ci_image_commands_config.py
index 87e6ab6e91..e128aa035b 100644
--- a/dev/breeze/src/airflow_breeze/commands/ci_image_commands_config.py
+++ b/dev/breeze/src/airflow_breeze/commands/ci_image_commands_config.py
@@ -35,7 +35,7 @@ CI_IMAGE_TOOLS_PARAMETERS: dict[str, list[dict[str, str | list[str]]]] = {
"--image-tag",
"--tag-as-latest",
"--docker-cache",
- "--force-build",
+ "--version-suffix-for-pypi",
"--build-progress",
],
},
@@ -51,24 +51,33 @@ CI_IMAGE_TOOLS_PARAMETERS: dict[str, list[dict[str, str | list[str]]]] = {
],
},
{
- "name": "Advanced options (for power users)",
+ "name": "Advanced build options (for power users)",
"options": [
"--debian-version",
+ "--python-image",
+ "--commit-sha",
+ "--additional-pip-install-flags",
"--install-providers-from-sources",
+ ],
+ },
+ {
+ "name": "Selecting constraint location (for power users)",
+ "options": [
"--airflow-constraints-location",
"--airflow-constraints-mode",
"--airflow-constraints-reference",
- "--python-image",
- "--additional-python-deps",
+ ],
+ },
+ {
+ "name": "Choosing dependencies and extras (for power users)",
+ "options": [
"--additional-airflow-extras",
- "--additional-pip-install-flags",
- "--additional-dev-apt-deps",
- "--additional-dev-apt-env",
- "--additional-dev-apt-command",
+ "--additional-python-deps",
"--dev-apt-deps",
+ "--additional-dev-apt-deps",
"--dev-apt-command",
- "--version-suffix-for-pypi",
- "--commit-sha",
+ "--additional-dev-apt-command",
+ "--additional-dev-apt-env",
],
},
{
diff --git a/dev/breeze/src/airflow_breeze/commands/production_image_commands.py b/dev/breeze/src/airflow_breeze/commands/production_image_commands.py
index 9f132aa3e7..ca1780738d 100644
--- a/dev/breeze/src/airflow_breeze/commands/production_image_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/production_image_commands.py
@@ -18,7 +18,7 @@ from __future__ import annotations
import os
import sys
-from typing import Any
+from copy import deepcopy
import click
@@ -27,10 +27,10 @@ from airflow_breeze.params.build_prod_params import BuildProdParams
from airflow_breeze.utils.ci_group import ci_group
from airflow_breeze.utils.click_utils import BreezeGroup
from airflow_breeze.utils.common_options import (
+ option_additional_airflow_extras,
option_additional_dev_apt_command,
option_additional_dev_apt_deps,
option_additional_dev_apt_env,
- option_additional_extras,
option_additional_pip_install_flags,
option_additional_python_deps,
option_additional_runtime_apt_command,
@@ -39,6 +39,7 @@ from airflow_breeze.utils.common_options import (
option_airflow_constraints_location,
option_airflow_constraints_mode_prod,
option_airflow_constraints_reference_build,
+ option_answer,
option_build_progress,
option_builder,
option_commit_sha,
@@ -90,7 +91,7 @@ from airflow_breeze.utils.path_utils import AIRFLOW_SOURCES_ROOT, DOCKER_CONTEXT
from airflow_breeze.utils.python_versions import get_python_version_list
from airflow_breeze.utils.registry import login_to_github_docker_registry
from airflow_breeze.utils.run_tests import verify_an_image
-from airflow_breeze.utils.run_utils import filter_out_none, fix_group_permissions, run_command
+from airflow_breeze.utils.run_utils import fix_group_permissions, run_command
from airflow_breeze.utils.shared_options import get_dry_run, get_verbose
@@ -151,26 +152,38 @@ def prod_image():
@prod_image.command(name="build")
@option_python
@option_debian_version
-@option_run_in_parallel
-@option_parallelism
-@option_skip_cleanup
-@option_debug_resources
-@option_include_success_outputs
-@option_python_versions
@option_platform_multiple
@option_github_token
@option_docker_cache
@option_image_tag_for_building
@option_prepare_buildx_cache
@option_push
+@option_install_providers_from_sources
+@click.option("-V", "--install-airflow-version", help="Install version of Airflow from PyPI.")
+@option_additional_airflow_extras
+@option_additional_dev_apt_deps
+@option_additional_runtime_apt_deps
+@option_additional_python_deps
+@option_additional_dev_apt_command
+@option_additional_dev_apt_env
+@option_additional_runtime_apt_env
+@option_additional_runtime_apt_command
+@option_builder
+@option_build_progress
+@option_commit_sha
+@option_dev_apt_command
+@option_dev_apt_deps
+@option_runtime_apt_command
+@option_runtime_apt_deps
@option_airflow_constraints_location
@option_airflow_constraints_mode_prod
@click.option(
"--installation-method",
help="Install Airflow from: sources or PyPI.",
type=BetterChoice(ALLOWED_INSTALLATION_METHODS),
+ default=ALLOWED_INSTALLATION_METHODS[0],
+ show_default=True,
)
-@option_install_providers_from_sources
@click.option(
"--install-packages-from-context",
help="Install wheels from local docker-context-files when building image. "
@@ -208,37 +221,72 @@ def prod_image():
help="Install Airflow using GitHub tag or branch.",
)
@option_airflow_constraints_reference_build
-@click.option("-V", "--install-airflow-version", help="Install version of Airflow from PyPI.")
-@option_additional_extras
-@option_additional_dev_apt_deps
-@option_additional_runtime_apt_deps
-@option_additional_python_deps
-@option_additional_dev_apt_command
-@option_additional_dev_apt_env
-@option_additional_runtime_apt_env
-@option_additional_runtime_apt_command
-@option_builder
-@option_build_progress
-@option_dev_apt_command
-@option_dev_apt_deps
-@option_python_image
-@option_runtime_apt_command
-@option_runtime_apt_deps
@option_tag_as_latest
@option_additional_pip_install_flags
@option_github_repository
+@option_python_image
@option_version_suffix_for_pypi
-@option_commit_sha
+@option_run_in_parallel
+@option_parallelism
+@option_skip_cleanup
+@option_debug_resources
+@option_include_success_outputs
+@option_python_versions
@option_verbose
@option_dry_run
+@option_answer
def build(
+ # build options
+ python: str,
+ debian_version: str,
+ platform: str | None,
+ github_token: str | None,
+ docker_cache: str,
+ image_tag: str,
+ prepare_buildx_cache: bool,
+ push: bool,
+ install_providers_from_sources: bool,
+ install_airflow_version: str | None,
+ additional_airflow_extras: str | None,
+ additional_dev_apt_deps: str | None,
+ additional_runtime_apt_deps: str | None,
+ additional_python_deps: str | None,
+ additional_dev_apt_command: str | None,
+ additional_dev_apt_env: str | None,
+ additional_runtime_apt_command: str | None,
+ additional_runtime_apt_env: str | None,
+ builder: str,
+ build_progress: str,
+ commit_sha: str | None,
+ dev_apt_command: str | None,
+ dev_apt_deps: str | None,
+ runtime_apt_command: str | None,
+ runtime_apt_deps: str | None,
+ airflow_constraints_location: str | None,
+ airflow_constraints_mode: str,
+ installation_method: str,
+ install_packages_from_context: bool,
+ use_constraints_for_context_packages: bool,
+ cleanup_context: bool,
+ airflow_extras: str,
+ disable_mysql_client_installation: bool,
+ disable_mssql_client_installation: bool,
+ disable_postgres_client_installation: bool,
+ disable_airflow_repo_cache: bool,
+ install_airflow_reference: str | None,
+ airflow_constraints_reference: str | None,
+ tag_as_latest: bool,
+ additional_pip_install_flags: str | None,
+ github_repository: str,
+ python_image: str | None,
+ version_suffix_for_pypi: str,
+ # Parallel building
run_in_parallel: bool,
parallelism: int,
skip_cleanup: bool,
debug_resources: bool,
- include_success_outputs: bool,
+ include_success_outputs,
python_versions: str,
- **kwargs: dict[str, Any],
):
"""
Build Production image. Include building multiple images for all or selected Python versions sequentially.
@@ -252,14 +300,59 @@ def build(
perform_environment_checks()
check_remote_ghcr_io_commands()
- parameters_passed = filter_out_none(**kwargs)
+ base_build_params = BuildProdParams(
+ python=python,
+ debian_version=debian_version,
+ github_token=github_token,
+ docker_cache=docker_cache,
+ image_tag=image_tag,
+ prepare_buildx_cache=prepare_buildx_cache,
+ push=push,
+ install_providers_from_sources=install_providers_from_sources,
+ install_airflow_version=install_airflow_version,
+ additional_airflow_extras=additional_airflow_extras,
+ additional_dev_apt_deps=additional_dev_apt_deps,
+ additional_runtime_apt_deps=additional_runtime_apt_deps,
+ additional_python_deps=additional_python_deps,
+ additional_dev_apt_command=additional_dev_apt_command,
+ additional_dev_apt_env=additional_dev_apt_env,
+ additional_runtime_apt_command=additional_runtime_apt_command,
+ additional_runtime_apt_env=additional_runtime_apt_env,
+ builder=builder,
+ build_progress=build_progress,
+ commit_sha=commit_sha,
+ dev_apt_command=dev_apt_command,
+ dev_apt_deps=dev_apt_deps,
+ runtime_apt_command=runtime_apt_command,
+ runtime_apt_deps=runtime_apt_deps,
+ airflow_constraints_location=airflow_constraints_location,
+ airflow_constraints_mode=airflow_constraints_mode,
+ installation_method=installation_method,
+ install_packages_from_context=install_packages_from_context,
+ use_constraints_for_context_packages=use_constraints_for_context_packages,
+ cleanup_context=cleanup_context,
+ airflow_extras=airflow_extras,
+ disable_mysql_client_installation=disable_mysql_client_installation,
+ disable_mssql_client_installation=disable_mssql_client_installation,
+ disable_postgres_client_installation=disable_postgres_client_installation,
+ disable_airflow_repo_cache=disable_airflow_repo_cache,
+ install_airflow_reference=install_airflow_reference,
+ airflow_constraints_reference=airflow_constraints_reference,
+ tag_as_latest=tag_as_latest,
+ additional_pip_install_flags=additional_pip_install_flags,
+ github_repository=github_repository,
+ python_image=python_image,
+ version_suffix_for_pypi=version_suffix_for_pypi,
+ )
+ if platform:
+ base_build_params.platform = platform
fix_group_permissions()
if run_in_parallel:
python_version_list = get_python_version_list(python_versions)
params_list: list[BuildProdParams] = []
for python in python_version_list:
- params = BuildProdParams(**parameters_passed)
+ params = deepcopy(base_build_params)
params.python = python
params_list.append(params)
prepare_for_building_prod_image(prod_image_params=params_list[0])
@@ -272,9 +365,8 @@ def build(
include_success_outputs=include_success_outputs,
)
else:
- params = BuildProdParams(**parameters_passed)
- prepare_for_building_prod_image(prod_image_params=params)
- run_build(prod_image_params=params)
+ prepare_for_building_prod_image(prod_image_params=base_build_params)
+ run_build(prod_image_params=base_build_params)
@prod_image.command(name="pull")
diff --git a/dev/breeze/src/airflow_breeze/commands/production_image_commands_config.py b/dev/breeze/src/airflow_breeze/commands/production_image_commands_config.py
index 713bd0bfa4..52291268f8 100644
--- a/dev/breeze/src/airflow_breeze/commands/production_image_commands_config.py
+++ b/dev/breeze/src/airflow_breeze/commands/production_image_commands_config.py
@@ -34,6 +34,7 @@ PRODUCTION_IMAGE_TOOLS_PARAMETERS: dict[str, list[dict[str, str | list[str]]]] =
"--image-tag",
"--tag-as-latest",
"--docker-cache",
+ "--version-suffix-for-pypi",
"--build-progress",
],
},
@@ -49,44 +50,53 @@ PRODUCTION_IMAGE_TOOLS_PARAMETERS: dict[str, list[dict[str, str | list[str]]]] =
],
},
{
- "name": "Options for customizing images",
+ "name": "Advanced build options (for power users)",
"options": [
+ "--debian-version",
+ "--python-image",
+ "--commit-sha",
+ "--additional-pip-install-flags",
"--install-providers-from-sources",
- "--airflow-extras",
+ ],
+ },
+ {
+ "name": "Selecting constraint location (for power users)",
+ "options": [
"--airflow-constraints-location",
"--airflow-constraints-mode",
"--airflow-constraints-reference",
- "--python-image",
+ ],
+ },
+ {
+ "name": "Choosing dependencies and extras (for power users)",
+ "options": [
+ "--airflow-extras",
"--additional-airflow-extras",
- "--additional-pip-install-flags",
"--additional-python-deps",
- "--additional-runtime-apt-deps",
- "--additional-runtime-apt-env",
- "--additional-runtime-apt-command",
+ "--dev-apt-deps",
"--additional-dev-apt-deps",
- "--additional-dev-apt-env",
+ "--dev-apt-command",
"--additional-dev-apt-command",
+ "--additional-dev-apt-env",
"--runtime-apt-deps",
+ "--additional-runtime-apt-deps",
"--runtime-apt-command",
- "--dev-apt-deps",
- "--dev-apt-command",
- "--version-suffix-for-pypi",
- "--commit-sha",
+ "--additional-runtime-apt-command",
+ "--additional-runtime-apt-env",
],
},
{
"name": "Advanced customization options (for specific customization needs)",
"options": [
- "--debian-version",
+ "--installation-method",
+ "--install-airflow-reference",
"--install-packages-from-context",
- "--use-constraints-for-context-packages",
"--cleanup-context",
+ "--use-constraints-for-context-packages",
+ "--disable-airflow-repo-cache",
"--disable-mysql-client-installation",
"--disable-mssql-client-installation",
"--disable-postgres-client-installation",
- "--disable-airflow-repo-cache",
- "--install-airflow-reference",
- "--installation-method",
],
},
{
diff --git a/dev/breeze/src/airflow_breeze/params/build_ci_params.py b/dev/breeze/src/airflow_breeze/params/build_ci_params.py
index a713c280b5..b29ebacff0 100644
--- a/dev/breeze/src/airflow_breeze/params/build_ci_params.py
+++ b/dev/breeze/src/airflow_breeze/params/build_ci_params.py
@@ -16,6 +16,7 @@
# under the License.
from __future__ import annotations
+import random
from dataclasses import dataclass
from pathlib import Path
@@ -38,7 +39,7 @@ class BuildCiParams(CommonBuildParams):
force_build: bool = False
upgrade_to_newer_dependencies: bool = False
upgrade_on_failure: bool = False
- eager_upgrade_additional_requirements: str = ""
+ eager_upgrade_additional_requirements: str | None = None
skip_provider_dependencies_check: bool = False
@property
@@ -49,72 +50,52 @@ class BuildCiParams(CommonBuildParams):
def image_type(self) -> str:
return "CI"
- @property
- def extra_docker_build_flags(self) -> list[str]:
- extra_ci_flags = []
- extra_ci_flags.extend(
- ["--build-arg", f"AIRFLOW_CONSTRAINTS_REFERENCE={self.airflow_constraints_reference}"]
- )
- if self.airflow_constraints_location:
- extra_ci_flags.extend(
- ["--build-arg", f"AIRFLOW_CONSTRAINTS_LOCATION={self.airflow_constraints_location}"]
- )
- if self.upgrade_to_newer_dependencies:
- eager_upgrade_arg = self.eager_upgrade_additional_requirements.strip().replace("\n", "")
- if eager_upgrade_arg:
- extra_ci_flags.extend(
- [
- "--build-arg",
- f"EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS={eager_upgrade_arg}",
- ]
- )
- return super().extra_docker_build_flags + extra_ci_flags
-
@property
def md5sum_cache_dir(self) -> Path:
return Path(BUILD_CACHE_DIR, self.airflow_branch, self.python, "CI")
- @property
- def required_image_args(self) -> list[str]:
- return [
- "airflow_branch",
- "airflow_constraints_mode",
- "airflow_constraints_reference",
- "airflow_extras",
- "airflow_image_date_created",
- "airflow_image_repository",
- "airflow_pre_cached_pip_packages",
- "airflow_version",
- "build_id",
- "constraints_github_repository",
- "python_base_image",
- "upgrade_to_newer_dependencies",
- ]
-
- @property
- def optional_image_args(self) -> list[str]:
- return [
- "additional_airflow_extras",
- "additional_dev_apt_command",
- "additional_dev_apt_deps",
- "additional_dev_apt_env",
- "additional_pip_install_flags",
- "additional_python_deps",
- "additional_runtime_apt_command",
- "additional_runtime_apt_deps",
- "additional_runtime_apt_env",
- "dev_apt_command",
- "dev_apt_deps",
- "additional_dev_apt_command",
- "additional_dev_apt_deps",
- "additional_dev_apt_env",
- "additional_airflow_extras",
- "additional_pip_install_flags",
- "additional_python_deps",
- "version_suffix_for_pypi",
- "commit_sha",
- "build_progress",
- ]
-
- def __post_init__(self):
- pass
+ def prepare_arguments_for_docker_build_command(self) -> list[str]:
+ self.build_arg_values: list[str] = []
+ # Required build args
+ self._req_arg("AIRFLOW_BRANCH", self.airflow_branch)
+ self._req_arg("AIRFLOW_CONSTRAINTS_MODE", self.airflow_constraints_mode)
+ self._req_arg("AIRFLOW_CONSTRAINTS_REFERENCE", self.airflow_constraints_reference)
+ self._req_arg("AIRFLOW_EXTRAS", self.airflow_extras)
+ self._req_arg("AIRFLOW_IMAGE_DATE_CREATED", self.airflow_image_date_created)
+ self._req_arg("AIRFLOW_IMAGE_REPOSITORY", self.airflow_image_repository)
+ self._req_arg("AIRFLOW_PRE_CACHED_PIP_PACKAGES", self.airflow_pre_cached_pip_packages)
+ self._req_arg("AIRFLOW_VERSION", self.airflow_version)
+ self._req_arg("BUILD_ID", self.build_id)
+ self._req_arg("CONSTRAINTS_GITHUB_REPOSITORY", self.constraints_github_repository)
+ self._req_arg("PYTHON_BASE_IMAGE", self.python_base_image)
+ if self.upgrade_to_newer_dependencies:
+ self._opt_arg("UPGRADE_TO_NEWER_DEPENDENCIES", f"{random.randrange(2**32):x}")
+ if self.eager_upgrade_additional_requirements:
+ # in case eager upgrade additional requirements have EOL, connect them together
+ self._opt_arg(
+ "EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS",
+ self.eager_upgrade_additional_requirements.replace("\n", ""),
+ )
+ # optional build args
+ self._opt_arg("AIRFLOW_CONSTRAINTS_LOCATION", self.airflow_constraints_location)
+ self._opt_arg("ADDITIONAL_AIRFLOW_EXTRAS", self.additional_airflow_extras)
+ self._opt_arg("ADDITIONAL_DEV_APT_COMMAND", self.additional_dev_apt_command)
+ self._opt_arg("ADDITIONAL_DEV_APT_DEPS", self.additional_dev_apt_deps)
+ self._opt_arg("ADDITIONAL_DEV_APT_ENV", self.additional_dev_apt_env)
+ self._opt_arg("ADDITIONAL_PIP_INSTALL_FLAGS", self.additional_pip_install_flags)
+ self._opt_arg("ADDITIONAL_PYTHON_DEPS", self.additional_python_deps)
+ self._opt_arg("DEV_APT_COMMAND", self.dev_apt_command)
+ self._opt_arg("DEV_APT_DEPS", self.dev_apt_deps)
+ self._opt_arg("ADDITIONAL_DEV_APT_COMMAND", self.additional_dev_apt_command)
+ self._opt_arg("ADDITIONAL_DEV_APT_DEPS", self.additional_dev_apt_deps)
+ self._opt_arg("ADDITIONAL_DEV_APT_ENV", self.additional_dev_apt_env)
+ self._opt_arg("ADDITIONAL_AIRFLOW_EXTRAS", self.additional_airflow_extras)
+ self._opt_arg("ADDITIONAL_PIP_INSTALL_FLAGS", self.additional_pip_install_flags)
+ self._opt_arg("ADDITIONAL_PYTHON_DEPS", self.additional_python_deps)
+ self._opt_arg("VERSION_SUFFIX_FOR_PYPI", self.version_suffix_for_pypi)
+ self._opt_arg("COMMIT_SHA", self.commit_sha)
+ self._opt_arg("BUILD_PROGRESS", self.build_progress)
+ # Convert to build args
+ build_args = self._to_build_args()
+ # Add cache directive
+ return build_args
diff --git a/dev/breeze/src/airflow_breeze/params/build_prod_params.py b/dev/breeze/src/airflow_breeze/params/build_prod_params.py
index 585c7dc0c9..ed9a6b6741 100644
--- a/dev/breeze/src/airflow_breeze/params/build_prod_params.py
+++ b/dev/breeze/src/airflow_breeze/params/build_prod_params.py
@@ -38,9 +38,9 @@ class BuildProdParams(CommonBuildParams):
PROD build parameters. Those parameters are used to determine command issued to build PROD image.
"""
- additional_runtime_apt_command: str = ""
- additional_runtime_apt_deps: str = ""
- additional_runtime_apt_env: str = ""
+ additional_runtime_apt_command: str | None = None
+ additional_runtime_apt_deps: str | None = None
+ additional_runtime_apt_env: str | None = None
airflow_constraints_mode: str = "constraints"
airflow_constraints_reference: str = DEFAULT_AIRFLOW_CONSTRAINTS_BRANCH
cleanup_context: bool = False
@@ -49,13 +49,13 @@ class BuildProdParams(CommonBuildParams):
disable_mssql_client_installation: bool = False
disable_mysql_client_installation: bool = False
disable_postgres_client_installation: bool = False
- install_airflow_reference: str = ""
- install_airflow_version: str = ""
+ install_airflow_reference: str | None = None
+ install_airflow_version: str | None = None
install_packages_from_context: bool = False
use_constraints_for_context_packages: bool = False
installation_method: str = "."
- runtime_apt_command: str = ""
- runtime_apt_deps: str = ""
+ runtime_apt_command: str | None = None
+ runtime_apt_deps: str | None = None
@property
def airflow_version(self) -> str:
@@ -110,24 +110,20 @@ class BuildProdParams(CommonBuildParams):
self.airflow_branch_for_pypi_preloading = AIRFLOW_BRANCH
return build_args
- @property
- def extra_docker_build_flags(self) -> list[str]:
+ def _extra_prod_docker_build_flags(self) -> list[str]:
extra_build_flags = []
if self.install_airflow_reference:
- AIRFLOW_INSTALLATION_METHOD = (
- "https://github.com/apache/airflow/archive/"
- + self.install_airflow_reference
- + ".tar.gz#egg=apache-airflow"
- )
extra_build_flags.extend(
[
"--build-arg",
- AIRFLOW_INSTALLATION_METHOD,
+ "https://github.com/apache/airflow/archive/"
+ + self.install_airflow_reference
+ + ".tar.gz#egg=apache-airflow",
]
)
extra_build_flags.extend(self.args_for_remote_install)
elif self.install_airflow_version:
- if not re.match(r"^[0-9\.]+((a|b|rc|alpha|beta|pre)[0-9]+)?$", self.install_airflow_version):
+ if not re.match(r"^[0-9.]+((a|b|rc|alpha|beta|pre)[0-9]+)?$", self.install_airflow_version):
get_console().print(
f"\n[error]ERROR: Bad value for install-airflow-version:{self.install_airflow_version}"
)
@@ -175,7 +171,7 @@ class BuildProdParams(CommonBuildParams):
f"io.artifacthub.package.logo-url={logo_url}",
]
)
- return super().extra_docker_build_flags + extra_build_flags
+ return extra_build_flags
@property
def airflow_pre_cached_pip_packages(self) -> str:
@@ -201,46 +197,45 @@ class BuildProdParams(CommonBuildParams):
def airflow_image_kubernetes(self) -> str:
return f"{self.airflow_image_name}-kubernetes"
- @property
- def required_image_args(self) -> list[str]:
- return [
- "airflow_branch",
- "airflow_constraints_mode",
- "airflow_extras",
- "airflow_image_date_created",
- "airflow_image_readme_url",
- "airflow_image_repository",
- "airflow_pre_cached_pip_packages",
- "airflow_version",
- "build_id",
- "constraints_github_repository",
- "docker_context_files",
- "install_mssql_client",
- "install_mysql_client",
- "install_packages_from_context",
- "install_postgres_client",
- "install_providers_from_sources",
- "python_base_image",
- ]
-
- @property
- def optional_image_args(self) -> list[str]:
- return [
- "additional_airflow_extras",
- "additional_dev_apt_command",
- "additional_dev_apt_deps",
- "additional_dev_apt_env",
- "additional_pip_install_flags",
- "additional_python_deps",
- "additional_runtime_apt_command",
- "additional_runtime_apt_deps",
- "additional_runtime_apt_env",
- "dev_apt_command",
- "dev_apt_deps",
- "runtime_apt_command",
- "runtime_apt_deps",
- "version_suffix_for_pypi",
- "commit_sha",
- "build_progress",
- "use_constraints_for_context_packages",
- ]
+ def prepare_arguments_for_docker_build_command(self) -> list[str]:
+ self.build_arg_values: list[str] = []
+ # Required build args
+ self._req_arg("AIRFLOW_BRANCH", self.airflow_branch)
+ self._req_arg("AIRFLOW_CONSTRAINTS_MODE", self.airflow_constraints_mode)
+ self._req_arg("AIRFLOW_EXTRAS", self.airflow_extras)
+ self._req_arg("AIRFLOW_IMAGE_DATE_CREATED", self.airflow_image_date_created)
+ self._req_arg("AIRFLOW_IMAGE_README_URL", self.airflow_image_readme_url)
+ self._req_arg("AIRFLOW_IMAGE_REPOSITORY", self.airflow_image_repository)
+ self._req_arg("AIRFLOW_PRE_CACHED_PIP_PACKAGES", self.airflow_pre_cached_pip_packages)
+ self._req_arg("AIRFLOW_VERSION", self.airflow_version)
+ self._req_arg("BUILD_ID", self.build_id)
+ self._req_arg("CONSTRAINTS_GITHUB_REPOSITORY", self.constraints_github_repository)
+ self._req_arg("DOCKER_CONTEXT_FILES", self.docker_context_files)
+ self._req_arg("INSTALL_MSSQL_CLIENT", self.install_mssql_client)
+ self._req_arg("INSTALL_MYSQL_CLIENT", self.install_mysql_client)
+ self._req_arg("INSTALL_PACKAGES_FROM_CONTEXT", self.install_packages_from_context)
+ self._req_arg("INSTALL_POSTGRES_CLIENT", self.install_postgres_client)
+ self._req_arg("INSTALL_PROVIDERS_FROM_SOURCES", self.install_providers_from_sources)
+ self._req_arg("PYTHON_BASE_IMAGE", self.python_base_image)
+ # optional build args
+ self._opt_arg("AIRFLOW_CONSTRAINTS_LOCATION", self.airflow_constraints_location)
+ self._opt_arg("ADDITIONAL_AIRFLOW_EXTRAS", self.additional_airflow_extras)
+ self._opt_arg("ADDITIONAL_DEV_APT_COMMAND", self.additional_dev_apt_command)
+ self._opt_arg("ADDITIONAL_DEV_APT_DEPS", self.additional_dev_apt_deps)
+ self._opt_arg("ADDITIONAL_DEV_APT_ENV", self.additional_dev_apt_env)
+ self._opt_arg("ADDITIONAL_PIP_INSTALL_FLAGS", self.additional_pip_install_flags)
+ self._opt_arg("ADDITIONAL_PYTHON_DEPS", self.additional_python_deps)
+ self._opt_arg("ADDITIONAL_RUNTIME_APT_COMMAND", self.additional_runtime_apt_command)
+ self._opt_arg("ADDITIONAL_RUNTIME_APT_DEPS", self.additional_runtime_apt_deps)
+ self._opt_arg("ADDITIONAL_RUNTIME_APT_ENV", self.additional_runtime_apt_env)
+ self._opt_arg("DEV_APT_COMMAND", self.dev_apt_command)
+ self._opt_arg("DEV_APT_DEPS", self.dev_apt_deps)
+ self._opt_arg("RUNTIME_APT_COMMAND", self.runtime_apt_command)
+ self._opt_arg("RUNTIME_APT_DEPS", self.runtime_apt_deps)
+ self._opt_arg("VERSION_SUFFIX_FOR_PYPI", self.version_suffix_for_pypi)
+ self._opt_arg("COMMIT_SHA", self.commit_sha)
+ self._opt_arg("BUILD_PROGRESS", self.build_progress)
+ self._opt_arg("USE_CONSTRAINTS_FOR_CONTEXT_PACKAGES", self.use_constraints_for_context_packages)
+ build_args = self._to_build_args()
+ build_args.extend(self._extra_prod_docker_build_flags())
+ return build_args
diff --git a/dev/breeze/src/airflow_breeze/params/common_build_params.py b/dev/breeze/src/airflow_breeze/params/common_build_params.py
index 782d01bca4..29e748f8c8 100644
--- a/dev/breeze/src/airflow_breeze/params/common_build_params.py
+++ b/dev/breeze/src/airflow_breeze/params/common_build_params.py
@@ -18,8 +18,9 @@ from __future__ import annotations
import os
import sys
-from dataclasses import dataclass
+from dataclasses import dataclass, field
from datetime import datetime
+from typing import Any
from airflow_breeze.branch_defaults import AIRFLOW_BRANCH, DEFAULT_AIRFLOW_CONSTRAINTS_BRANCH
from airflow_breeze.global_constants import (
@@ -37,24 +38,23 @@ class CommonBuildParams:
Common build parameters. Those parameters are common parameters for CI And PROD build.
"""
- additional_airflow_extras: str = ""
- additional_dev_apt_command: str = ""
- additional_dev_apt_deps: str = ""
- additional_dev_apt_env: str = ""
- additional_python_deps: str = ""
- additional_pip_install_flags: str = ""
+ additional_airflow_extras: str | None = None
+ additional_dev_apt_command: str | None = None
+ additional_dev_apt_deps: str | None = None
+ additional_dev_apt_env: str | None = None
+ additional_python_deps: str | None = None
+ additional_pip_install_flags: str | None = None
airflow_branch: str = os.environ.get("DEFAULT_BRANCH", AIRFLOW_BRANCH)
default_constraints_branch: str = os.environ.get(
"DEFAULT_CONSTRAINTS_BRANCH", DEFAULT_AIRFLOW_CONSTRAINTS_BRANCH
)
- airflow_constraints_location: str = ""
- build_id: int = 0
+ airflow_constraints_location: str | None = None
builder: str = "autodetect"
build_progress: str = ALLOWED_BUILD_PROGRESS[0]
constraints_github_repository: str = APACHE_AIRFLOW_GITHUB_REPOSITORY
- commit_sha: str = ""
- dev_apt_command: str = ""
- dev_apt_deps: str = ""
+ commit_sha: str | None = None
+ dev_apt_command: str | None = None
+ dev_apt_deps: str | None = None
docker_cache: str = "registry"
github_actions: str = os.environ.get("GITHUB_ACTIONS", "false")
github_repository: str = APACHE_AIRFLOW_GITHUB_REPOSITORY
@@ -68,14 +68,19 @@ class CommonBuildParams:
python: str = "3.8"
tag_as_latest: bool = False
dry_run: bool = False
- version_suffix_for_pypi: str = ""
+ version_suffix_for_pypi: str | None = None
verbose: bool = False
debian_version: str = "bookworm"
+ build_arg_values: list[str] = field(default_factory=list)
@property
def airflow_version(self):
raise NotImplementedError()
+ @property
+ def build_id(self) -> str:
+ return os.environ.get("CI_BUILD_ID", "0")
+
@property
def image_type(self) -> str:
raise NotImplementedError()
@@ -99,23 +104,14 @@ class CommonBuildParams:
return image
@property
- def extra_docker_build_flags(self) -> list[str]:
- extra_flass = []
- if self.build_progress:
- extra_flass.append(f"--progress={self.build_progress}")
- return extra_flass
-
- @property
- def docker_cache_directive(self) -> list[str]:
- docker_cache_directive = []
+ def common_docker_build_flags(self) -> list[str]:
+ extra_flags = []
+ extra_flags.append(f"--progress={self.build_progress}")
if self.docker_cache == "registry":
- for platform in self.platforms:
- docker_cache_directive.append(f"--cache-from={self.get_cache(platform)}")
+ extra_flags.append(f"--cache-from={self.get_cache(self.platform)}")
elif self.docker_cache == "disabled":
- docker_cache_directive.append("--no-cache")
- else:
- docker_cache_directive = []
- return docker_cache_directive
+ extra_flags.append("--no-cache")
+ return extra_flags
@property
def python_base_image(self):
@@ -169,13 +165,31 @@ class CommonBuildParams:
def platforms(self) -> list[str]:
return self.platform.split(",")
- @property
- def required_image_args(self) -> list[str]:
- raise NotImplementedError()
+ def _build_arg(self, name: str, value: Any, optional: bool):
+ if value is None or "":
+ if optional:
+ return
+ else:
+ raise ValueError(f"Value for {name} cannot be empty or None")
+ if value is True:
+ str_value = "true"
+ elif value is False:
+ str_value = "false"
+ else:
+ str_value = str(value) if value is not None else ""
+ self.build_arg_values.append(f"{name}={str_value}")
- @property
- def optional_image_args(self) -> list[str]:
+ def _req_arg(self, name: str, value: Any):
+ self._build_arg(name, value, False)
+
+ def _opt_arg(self, name: str, value: Any):
+ self._build_arg(name, value, True)
+
+ def prepare_arguments_for_docker_build_command(self) -> list[str]:
raise NotImplementedError()
- def __post_init__(self):
- pass
+ def _to_build_args(self):
+ build_args = []
+ for arg in self.build_arg_values:
+ build_args.extend(["--build-arg", arg])
+ return build_args
diff --git a/dev/breeze/src/airflow_breeze/utils/common_options.py b/dev/breeze/src/airflow_breeze/utils/common_options.py
index 2a001f2757..249faaafd1 100644
--- a/dev/breeze/src/airflow_breeze/utils/common_options.py
+++ b/dev/breeze/src/airflow_breeze/utils/common_options.py
@@ -293,7 +293,7 @@ option_upgrade_on_failure = click.option(
help="When set, attempt to run upgrade to newer dependencies when regular build fails.",
envvar="UPGRADE_ON_FAILURE",
)
-option_additional_extras = click.option(
+option_additional_airflow_extras = click.option(
"--additional-airflow-extras",
help="Additional extra package while installing Airflow in the image.",
envvar="ADDITIONAL_AIRFLOW_EXTRAS",
@@ -413,7 +413,13 @@ option_load_default_connection = click.option(
option_version_suffix_for_pypi = click.option(
"--version-suffix-for-pypi",
help="Version suffix used for PyPI packages (alpha, beta, rc1, etc.).",
- default="",
+ envvar="VERSION_SUFFIX_FOR_PYPI",
+)
+option_version_suffix_for_pypi_ci = click.option(
+ "--version-suffix-for-pypi",
+ help="Version suffix used for PyPI packages (alpha, beta, rc1, etc.).",
+ default="dev0",
+ show_default=True,
envvar="VERSION_SUFFIX_FOR_PYPI",
)
option_package_format = click.option(
@@ -478,7 +484,6 @@ option_airflow_constraints_reference = click.option(
option_airflow_constraints_location = click.option(
"--airflow-constraints-location",
type=str,
- default="",
help="If specified, it is used instead of calculating reference to the constraint file. "
"It could be full remote URL to the location file, or local file placed in `docker-context-files` "
"(in this case it has to start with /opt/airflow/docker-context-files).",
@@ -615,7 +620,6 @@ option_historical_python_version = click.option(
)
option_commit_sha = click.option(
"--commit-sha",
- default=None,
show_default=True,
envvar="COMMIT_SHA",
help="Commit SHA that is used to build the images.",
diff --git a/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py b/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
index 36c976b2fe..78a8d69754 100644
--- a/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
+++ b/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
@@ -388,35 +388,6 @@ def get_env_variable_value(arg_name: str, params: CommonBuildParams | ShellParam
return value
-def prepare_arguments_for_docker_build_command(image_params: CommonBuildParams) -> list[str]:
- """
- Constructs docker compose command arguments list based on parameters passed. Maps arguments to
- argument values.
-
- It maps:
- * all the truthy/falsy values are converted to "true" / "false" respectively
- * if upgrade_to_newer_dependencies is set to True, it is replaced by a random string to account
- for the need of always triggering upgrade for docker build.
-
- :param image_params: parameters of the image
- :return: list of `--build-arg` commands to use for the parameters passed
- """
-
- args_command = []
- for required_arg in image_params.required_image_args:
- args_command.append("--build-arg")
- args_command.append(
- required_arg.upper() + "=" + get_env_variable_value(arg_name=required_arg, params=image_params)
- )
- for optional_arg in image_params.optional_image_args:
- param_value = get_env_variable_value(optional_arg, params=image_params)
- if param_value:
- args_command.append("--build-arg")
- args_command.append(optional_arg.upper() + "=" + param_value)
- args_command.extend(image_params.docker_cache_directive)
- return args_command
-
-
def prepare_docker_build_cache_command(
image_params: CommonBuildParams,
) -> list[str]:
@@ -427,16 +398,14 @@ def prepare_docker_build_cache_command(
:return: Command to run as list of string
"""
- arguments = prepare_arguments_for_docker_build_command(image_params)
- build_flags = image_params.extra_docker_build_flags
final_command = []
final_command.extend(["docker"])
final_command.extend(
["buildx", "build", "--builder", get_and_use_docker_context(image_params.builder), "--progress=auto"]
)
- final_command.extend(build_flags)
+ final_command.extend(image_params.common_docker_build_flags)
final_command.extend(["--pull"])
- final_command.extend(arguments)
+ final_command.extend(image_params.prepare_arguments_for_docker_build_command())
final_command.extend(["--target", "main", "."])
final_command.extend(
["-f", "Dockerfile" if isinstance(image_params, BuildProdParams) else "Dockerfile.ci"]
@@ -470,7 +439,6 @@ def prepare_base_build_command(image_params: CommonBuildParams) -> list[str]:
"build",
"--builder",
get_and_use_docker_context(image_params.builder),
- "--progress=auto",
"--push" if image_params.push else "--load",
]
)
@@ -488,17 +456,15 @@ def prepare_docker_build_command(
:return: Command to run as list of string
"""
- arguments = prepare_arguments_for_docker_build_command(image_params)
build_command = prepare_base_build_command(
image_params=image_params,
)
- build_flags = image_params.extra_docker_build_flags
final_command = []
final_command.extend(["docker"])
final_command.extend(build_command)
- final_command.extend(build_flags)
+ final_command.extend(image_params.common_docker_build_flags)
final_command.extend(["--pull"])
- final_command.extend(arguments)
+ final_command.extend(image_params.prepare_arguments_for_docker_build_command())
final_command.extend(["-t", image_params.airflow_image_name_with_tag, "--target", "main", "."])
final_command.extend(
["-f", "Dockerfile" if isinstance(image_params, BuildProdParams) else "Dockerfile.ci"]
diff --git a/images/breeze/output_ci-image_build.svg b/images/breeze/output_ci-image_build.svg
index 2a7147c2fd..57fb8ac402 100644
--- a/images/breeze/output_ci-image_build.svg
+++ b/images/breeze/output_ci-image_build.svg
@@ -1,4 +1,4 @@
-<svg class="rich-terminal" viewBox="0 0 1482 2172.7999999999997" xmlns="http://www.w3.org/2000/svg">
+<svg class="rich-terminal" viewBox="0 0 1482 2270.4" xmlns="http://www.w3.org/2000/svg">
<!-- Generated with Rich https://www.textualize.io -->
<style>
@@ -43,7 +43,7 @@
<defs>
<clipPath id="breeze-ci-image-build-clip-terminal">
- <rect x="0" y="0" width="1463.0" height="2121.7999999999997" />
+ <rect x="0" y="0" width="1463.0" height="2219.4" />
</clipPath>
<clipPath id="breeze-ci-image-build-line-0">
<rect x="0" y="1.5" width="1464" height="24.65"/>
@@ -303,9 +303,21 @@
<clipPath id="breeze-ci-image-build-line-85">
<rect x="0" y="2075.5" width="1464" height="24.65"/>
</clipPath>
+<clipPath id="breeze-ci-image-build-line-86">
+ <rect x="0" y="2099.9" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-ci-image-build-line-87">
+ <rect x="0" y="2124.3" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-ci-image-build-line-88">
+ <rect x="0" y="2148.7" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-ci-image-build-line-89">
+ <rect x="0" y="2173.1" width="1464" height="24.65"/>
+ </clipPath>
</defs>
- <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="2170.8" rx="8"/><text class="breeze-ci-image-build-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: ci-image build</text>
+ <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="2268.4" rx="8"/><text class="breeze-ci-image-build-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: ci-image build</text>
<g transform="translate(26,22)">
<circle cx="0" cy="0" r="7" fill="#ff5f57"/>
<circle cx="22" cy="0" r="7" fill="#febc2e"/>
@@ -332,76 +344,80 @@
</text><text class="breeze-ci-image-build-r5" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-14)">│</text><text class="breeze-ci-image-build-r1" x="475.8" y="361.6" textLength="414.8" clip-path="url(#breeze-ci-image-build-line-14)">when you build or pull image with </text><text class="breeze-ci-image-build-r4" x="890.6" y="361.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-14)">-</text><text class="breeze-c [...]
</text><text class="breeze-ci-image-build-r5" x="0" y="386" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-15)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="386" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-15)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="386" textLength="85.4" clip-path="url(#breeze-ci-image-build-line-15)">-docker</text><text class="breeze-ci-image-build-r4" x="122" y="386" textLength="73.2" clip-path="url(#bre [...]
</text><text class="breeze-ci-image-build-r5" x="0" y="410.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-16)">│</text><text class="breeze-ci-image-build-r5" x="475.8" y="410.4" textLength="549" clip-path="url(#breeze-ci-image-build-line-16)">[default: registry]                          </text><text class="breeze-ci-image-build-r5" x="145 [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="434.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-17)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="434.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-17)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="434.8" textLength="73.2" clip-path="url(#breeze-ci-image-build-line-17)">-force</text><text class="breeze-ci-image-build-r4" x="109.8" y="434.8" textLength="73.2" clip-path= [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="459.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-18)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="459.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-18)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="459.2" textLength="73.2" clip-path="url(#breeze-ci-image-build-line-18)">-build</text><text class="breeze-ci-image-build-r4" x="109.8" y="459.2" textLength="109.8" clip-path [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="483.6" textLength="1464" clip-path="url(#breeze-ci-image-build-line-19)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-build-r1" x="1464" y="483.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-19)">
-</text><text class="breeze-ci-image-build-r5" x="0" y="508" textLength="24.4" clip-path="url(#breeze-ci-image-build-line-20)">╭─</text><text class="breeze-ci-image-build-r5" x="24.4" y="508" textLength="353.8" clip-path="url(#breeze-ci-image-build-line-20)"> Building images in parallel </text><text class="breeze-ci-image-build-r5" x="378.2" y="508" textLength="1061.4" clip-path="url(#breeze-ci-image-build-line-20)">──────────────────────────────────────────────── [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-21)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="532.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-21)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="532.4" textLength="48.8" clip-path="url(#breeze-ci-image-build-line-21)">-run</text><text class="breeze-ci-image-build-r4" x="85.4" y="532.4" textLength="146.4" clip-path="u [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="556.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-22)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="556.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-22)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="556.8" textLength="146.4" clip-path="url(#breeze-ci-image-build-line-22)">-parallelism</text><text class="breeze-ci-image-build-r1" x="378.2" y="556.8" textLength="915" clip [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="581.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-23)">│</text><text class="breeze-ci-image-build-r7" x="378.2" y="581.2" textLength="915" clip-path="url(#breeze-ci-image-build-line-23)">(INTEGER RANGE)                                   &# [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="605.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-24)">│</text><text class="breeze-ci-image-build-r5" x="378.2" y="605.6" textLength="915" clip-path="url(#breeze-ci-image-build-line-24)">[default: 4; 1<=x<=8]                                 [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="630" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-25)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="630" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-25)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="630" textLength="85.4" clip-path="url(#breeze-ci-image-build-line-25)">-python</text><text class="breeze-ci-image-build-r4" x="122" y="630" textLength="109.8" clip-path="url(#br [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="654.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-26)">│</text><text class="breeze-ci-image-build-r5" x="378.2" y="654.4" textLength="951.6" clip-path="url(#breeze-ci-image-build-line-26)">[default: 3.8 3.9 3.10 3.11]                              &# [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="678.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-27)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="678.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-27)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="678.8" textLength="61" clip-path="url(#breeze-ci-image-build-line-27)">-skip</text><text class="breeze-ci-image-build-r4" x="97.6" y="678.8" textLength="97.6" clip-path="url [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="703.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-28)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="703.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-28)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="703.2" textLength="73.2" clip-path="url(#breeze-ci-image-build-line-28)">-debug</text><text class="breeze-ci-image-build-r4" x="109.8" y="703.2" textLength="122" clip-path=" [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="727.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-29)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="727.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-29)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="727.6" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-29)">-include</text><text class="breeze-ci-image-build-r4" x="134.2" y="727.6" textLength="195.2" clip-pa [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="752" textLength="1464" clip-path="url(#breeze-ci-image-build-line-30)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-build-r1" x="1464" y="752" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-30)">
-</text><text class="breeze-ci-image-build-r5" x="0" y="776.4" textLength="24.4" clip-path="url(#breeze-ci-image-build-line-31)">╭─</text><text class="breeze-ci-image-build-r5" x="24.4" y="776.4" textLength="439.2" clip-path="url(#breeze-ci-image-build-line-31)"> Advanced options (for power users) </text><text class="breeze-ci-image-build-r5" x="463.6" y="776.4" textLength="976" clip-path="url(#breeze-ci-image-build-line-31)">───────────────────────────────── [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="800.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-32)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="800.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-32)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="800.8" textLength="85.4" clip-path="url(#breeze-ci-image-build-line-32)">-debian</text><text class="breeze-ci-image-build-r4" x="122" y="800.8" textLength="97.6" clip-path=" [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="825.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-33)">│</text><text class="breeze-ci-image-build-r7" x="463.6" y="825.2" textLength="793" clip-path="url(#breeze-ci-image-build-line-33)">(bookworm | bullseye)                                  [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="849.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-34)">│</text><text class="breeze-ci-image-build-r5" x="463.6" y="849.6" textLength="793" clip-path="url(#breeze-ci-image-build-line-34)">[default: bookworm]                                   [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="874" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-35)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="874" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-35)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="874" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-35)">-install</text><text class="breeze-ci-image-build-r4" x="134.2" y="874" textLength="280.6" clip-path="url( [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="898.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-36)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="898.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-36)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="898.4" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-36)">-airflow</text><text class="breeze-ci-image-build-r4" x="134.2" y="898.4" textLength="256.2" clip-pa [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="922.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-37)">│</text><text class="breeze-ci-image-build-r1" x="463.6" y="922.8" textLength="976" clip-path="url(#breeze-ci-image-build-line-37)">file. It could be full remote URL to the location file, or local file placed in </text><text class="breeze-ci-image-build-r5" x="1451.8" y="922.8" textL [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="947.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-38)">│</text><text class="breeze-ci-image-build-r1" x="463.6" y="947.2" textLength="976" clip-path="url(#breeze-ci-image-build-line-38)">`docker-context-files` (in this case it has to start with                       [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="971.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-39)">│</text><text class="breeze-ci-image-build-r1" x="463.6" y="971.6" textLength="976" clip-path="url(#breeze-ci-image-build-line-39)">/opt/airflow/docker-context-files).                                  [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="996" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-40)">│</text><text class="breeze-ci-image-build-r7" x="463.6" y="996" textLength="976" clip-path="url(#breeze-ci-image-build-line-40)">(TEXT)                                      &# [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1020.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-41)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1020.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-41)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1020.4" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-41)">-airflow</text><text class="breeze-ci-image-build-r4" x="134.2" y="1020.4" textLength="207.4" cli [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1044.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-42)">│</text><text class="breeze-ci-image-build-r7" x="463.6" y="1044.8" textLength="866.2" clip-path="url(#breeze-ci-image-build-line-42)">(constraints-source-providers | constraints | constraints-no-providers)</text><text class="breeze-ci-image-build-r5" x="1451.8" y="1044.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-42)">│</ [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1069.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-43)">│</text><text class="breeze-ci-image-build-r5" x="463.6" y="1069.2" textLength="866.2" clip-path="url(#breeze-ci-image-build-line-43)">[default: constraints-source-providers]                               [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1093.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-44)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1093.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-44)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1093.6" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-44)">-airflow</text><text class="breeze-ci-image-build-r4" x="134.2" y="1093.6" textLength="268.4" cli [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1118" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-45)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1118" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-45)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1118" textLength="85.4" clip-path="url(#breeze-ci-image-build-line-45)">-python</text><text class="breeze-ci-image-build-r4" x="122" y="1118" textLength="73.2" clip-path="url( [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1142.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-46)">│</text><text class="breeze-ci-image-build-r1" x="463.6" y="1142.4" textLength="976" clip-path="url(#breeze-ci-image-build-line-46)">something like: python:VERSION-slim-bookworm.                             & [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1166.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-47)">│</text><text class="breeze-ci-image-build-r7" x="463.6" y="1166.8" textLength="976" clip-path="url(#breeze-ci-image-build-line-47)">(TEXT)                                     &# [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1191.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-48)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1191.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-48)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1191.2" textLength="134.2" clip-path="url(#breeze-ci-image-build-line-48)">-additional</text><text class="breeze-ci-image-build-r4" x="170.8" y="1191.2" textLength="146.4" [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1215.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-49)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1215.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-49)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1215.6" textLength="134.2" clip-path="url(#breeze-ci-image-build-line-49)">-additional</text><text class="breeze-ci-image-build-r4" x="170.8" y="1215.6" textLength="183" c [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1240" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-50)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1240" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-50)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1240" textLength="134.2" clip-path="url(#breeze-ci-image-build-line-50)">-additional</text><text class="breeze-ci-image-build-r4" x="170.8" y="1240" textLength="219.6" clip-pa [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1264.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-51)">│</text><text class="breeze-ci-image-build-r1" x="463.6" y="1264.4" textLength="976" clip-path="url(#breeze-ci-image-build-line-51)">itself).                                      [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1288.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-52)">│</text><text class="breeze-ci-image-build-r7" x="463.6" y="1288.8" textLength="976" clip-path="url(#breeze-ci-image-build-line-52)">(TEXT)                                     &# [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1313.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-53)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1313.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-53)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1313.2" textLength="134.2" clip-path="url(#breeze-ci-image-build-line-53)">-additional</text><text class="breeze-ci-image-build-r4" x="170.8" y="1313.2" textLength="158.6" [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1337.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-54)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1337.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-54)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1337.6" textLength="134.2" clip-path="url(#breeze-ci-image-build-line-54)">-additional</text><text class="breeze-ci-image-build-r4" x="170.8" y="1337.6" textLength="146.4" [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1362" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-55)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1362" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-55)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1362" textLength="134.2" clip-path="url(#breeze-ci-image-build-line-55)">-additional</text><text class="breeze-ci-image-build-r4" x="170.8" y="1362" textLength="195.2" clip-pa [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1386.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-56)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1386.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-56)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1386.4" textLength="48.8" clip-path="url(#breeze-ci-image-build-line-56)">-dev</text><text class="breeze-ci-image-build-r4" x="85.4" y="1386.4" textLength="109.8" clip-pat [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1410.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-57)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1410.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-57)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1410.8" textLength="48.8" clip-path="url(#breeze-ci-image-build-line-57)">-dev</text><text class="breeze-ci-image-build-r4" x="85.4" y="1410.8" textLength="146.4" clip-pat [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1435.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-58)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1435.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-58)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1435.2" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-58)">-version</text><text class="breeze-ci-image-build-r4" x="134.2" y="1435.2" textLength="195.2" cli [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1459.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-59)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1459.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-59)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1459.6" textLength="85.4" clip-path="url(#breeze-ci-image-build-line-59)">-commit</text><text class="breeze-ci-image-build-r4" x="122" y="1459.6" textLength="48.8" clip-pa [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1484" textLength="1464" clip-path="url(#breeze-ci-image-build-line-60)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-build-r1" x="1464" y="1484" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-60)">
-</text><text class="breeze-ci-image-build-r5" x="0" y="1508.4" textLength="24.4" clip-path="url(#breeze-ci-image-build-line-61)">╭─</text><text class="breeze-ci-image-build-r5" x="24.4" y="1508.4" textLength="268.4" clip-path="url(#breeze-ci-image-build-line-61)"> Backtracking options </text><text class="breeze-ci-image-build-r5" x="292.8" y="1508.4" textLength="1146.8" clip-path="url(#breeze-ci-image-build-line-61)">──────────────────────────────────────────────────────── [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1532.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-62)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1532.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-62)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1532.8" textLength="73.2" clip-path="url(#breeze-ci-image-build-line-62)">-build</text><text class="breeze-ci-image-build-r4" x="109.8" y="1532.8" textLength="195.2" clip- [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1557.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-63)">│</text><text class="breeze-ci-image-build-r1" x="549" y="1557.2" textLength="890.6" clip-path="url(#breeze-ci-image-build-line-63)">backtracking problems.                                   [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1581.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-64)">│</text><text class="breeze-ci-image-build-r7" x="549" y="1581.6" textLength="890.6" clip-path="url(#breeze-ci-image-build-line-64)">(INTEGER)                                      [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1606" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-65)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1606" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-65)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1606" textLength="73.2" clip-path="url(#breeze-ci-image-build-line-65)">-eager</text><text class="breeze-ci-image-build-r4" x="109.8" y="1606" textLength="390.4" clip-path="ur [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1630.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-66)">│</text><text class="breeze-ci-image-build-r1" x="549" y="1630.4" textLength="890.6" clip-path="url(#breeze-ci-image-build-line-66)">(see `breeze ci find-backtracking-candidates`).                          </text><text [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1654.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-67)">│</text><text class="breeze-ci-image-build-r7" x="549" y="1654.8" textLength="890.6" clip-path="url(#breeze-ci-image-build-line-67)">(TEXT)                                     &# [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1679.2" textLength="1464" clip-path="url(#breeze-ci-image-build-line-68)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-build-r1" x="1464" y="1679.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-68)">
-</text><text class="breeze-ci-image-build-r5" x="0" y="1703.6" textLength="24.4" clip-path="url(#breeze-ci-image-build-line-69)">╭─</text><text class="breeze-ci-image-build-r5" x="24.4" y="1703.6" textLength="622.2" clip-path="url(#breeze-ci-image-build-line-69)"> Preparing cache and push (for maintainers and CI) </text><text class="breeze-ci-image-build-r5" x="646.6" y="1703.6" textLength="793" clip-path="url(#breeze-ci-image-build-line-69)"> [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1728" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-70)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1728" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-70)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1728" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-70)">-builder</text><text class="breeze-ci-image-build-r1" x="341.6" y="1728" textLength="756.4" clip-path=" [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1752.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-71)">│</text><text class="breeze-ci-image-build-r5" x="341.6" y="1752.4" textLength="756.4" clip-path="url(#breeze-ci-image-build-line-71)">[default: autodetect]                                  [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1776.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-72)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1776.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-72)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1776.8" textLength="109.8" clip-path="url(#breeze-ci-image-build-line-72)">-platform</text><text class="breeze-ci-image-build-r1" x="341.6" y="1776.8" textLength="329.4" c [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1801.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-73)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1801.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-73)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1801.2" textLength="61" clip-path="url(#breeze-ci-image-build-line-73)">-push</text><text class="breeze-ci-image-build-r1" x="341.6" y="1801.2" textLength="353.8" clip-pat [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1825.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-74)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1825.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-74)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1825.6" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-74)">-prepare</text><text class="breeze-ci-image-build-r4" x="134.2" y="1825.6" textLength="158.6" cli [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1850" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-75)">│</text><text class="breeze-ci-image-build-r1" x="341.6" y="1850" textLength="1098" clip-path="url(#breeze-ci-image-build-line-75)">image).                                      [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1874.4" textLength="1464" clip-path="url(#breeze-ci-image-build-line-76)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-build-r1" x="1464" y="1874.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-76)">
-</text><text class="breeze-ci-image-build-r5" x="0" y="1898.8" textLength="24.4" clip-path="url(#breeze-ci-image-build-line-77)">╭─</text><text class="breeze-ci-image-build-r5" x="24.4" y="1898.8" textLength="280.6" clip-path="url(#breeze-ci-image-build-line-77)"> Github authentication </text><text class="breeze-ci-image-build-r5" x="305" y="1898.8" textLength="1134.6" clip-path="url(#breeze-ci-image-build-line-77)">───────────────────────────────────────────────────────── [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1923.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-78)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1923.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-78)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1923.2" textLength="85.4" clip-path="url(#breeze-ci-image-build-line-78)">-github</text><text class="breeze-ci-image-build-r4" x="122" y="1923.2" textLength="134.2" clip-p [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1947.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-79)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1947.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-79)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1947.6" textLength="85.4" clip-path="url(#breeze-ci-image-build-line-79)">-github</text><text class="breeze-ci-image-build-r4" x="122" y="1947.6" textLength="73.2" clip-pa [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="434.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-17)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="434.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-17)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="434.8" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-17)">-version</text><text class="breeze-ci-image-build-r4" x="134.2" y="434.8" textLength="195.2" clip-pa [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="459.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-18)">│</text><text class="breeze-ci-image-build-r5" x="475.8" y="459.2" textLength="768.6" clip-path="url(#breeze-ci-image-build-line-18)">[default: dev0]                                    [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-19)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="483.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-19)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="483.6" textLength="73.2" clip-path="url(#breeze-ci-image-build-line-19)">-build</text><text class="breeze-ci-image-build-r4" x="109.8" y="483.6" textLength="109.8" clip-path [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="508" textLength="1464" clip-path="url(#breeze-ci-image-build-line-20)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-build-r1" x="1464" y="508" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-20)">
+</text><text class="breeze-ci-image-build-r5" x="0" y="532.4" textLength="24.4" clip-path="url(#breeze-ci-image-build-line-21)">╭─</text><text class="breeze-ci-image-build-r5" x="24.4" y="532.4" textLength="353.8" clip-path="url(#breeze-ci-image-build-line-21)"> Building images in parallel </text><text class="breeze-ci-image-build-r5" x="378.2" y="532.4" textLength="1061.4" clip-path="url(#breeze-ci-image-build-line-21)">────────────────────────────────────────── [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="556.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-22)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="556.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-22)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="556.8" textLength="48.8" clip-path="url(#breeze-ci-image-build-line-22)">-run</text><text class="breeze-ci-image-build-r4" x="85.4" y="556.8" textLength="146.4" clip-path="u [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="581.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-23)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="581.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-23)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="581.2" textLength="146.4" clip-path="url(#breeze-ci-image-build-line-23)">-parallelism</text><text class="breeze-ci-image-build-r1" x="378.2" y="581.2" textLength="915" clip [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="605.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-24)">│</text><text class="breeze-ci-image-build-r7" x="378.2" y="605.6" textLength="915" clip-path="url(#breeze-ci-image-build-line-24)">(INTEGER RANGE)                                   &# [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="630" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-25)">│</text><text class="breeze-ci-image-build-r5" x="378.2" y="630" textLength="915" clip-path="url(#breeze-ci-image-build-line-25)">[default: 4; 1<=x<=8]                                 & [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="654.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-26)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="654.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-26)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="654.4" textLength="85.4" clip-path="url(#breeze-ci-image-build-line-26)">-python</text><text class="breeze-ci-image-build-r4" x="122" y="654.4" textLength="109.8" clip-path= [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="678.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-27)">│</text><text class="breeze-ci-image-build-r5" x="378.2" y="678.8" textLength="951.6" clip-path="url(#breeze-ci-image-build-line-27)">[default: 3.8 3.9 3.10 3.11]                              &# [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="703.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-28)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="703.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-28)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="703.2" textLength="61" clip-path="url(#breeze-ci-image-build-line-28)">-skip</text><text class="breeze-ci-image-build-r4" x="97.6" y="703.2" textLength="97.6" clip-path="url [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="727.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-29)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="727.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-29)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="727.6" textLength="73.2" clip-path="url(#breeze-ci-image-build-line-29)">-debug</text><text class="breeze-ci-image-build-r4" x="109.8" y="727.6" textLength="122" clip-path=" [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="752" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-30)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="752" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-30)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="752" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-30)">-include</text><text class="breeze-ci-image-build-r4" x="134.2" y="752" textLength="195.2" clip-path="url( [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="776.4" textLength="1464" clip-path="url(#breeze-ci-image-build-line-31)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-build-r1" x="1464" y="776.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-31)">
+</text><text class="breeze-ci-image-build-r5" x="0" y="800.8" textLength="24.4" clip-path="url(#breeze-ci-image-build-line-32)">╭─</text><text class="breeze-ci-image-build-r5" x="24.4" y="800.8" textLength="512.4" clip-path="url(#breeze-ci-image-build-line-32)"> Advanced build options (for power users) </text><text class="breeze-ci-image-build-r5" x="536.8" y="800.8" textLength="902.8" clip-path="url(#breeze-ci-image-build-line-32)">──────────────────── [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="825.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-33)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="825.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-33)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="825.2" textLength="85.4" clip-path="url(#breeze-ci-image-build-line-33)">-debian</text><text class="breeze-ci-image-build-r4" x="122" y="825.2" textLength="97.6" clip-path=" [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="849.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-34)">│</text><text class="breeze-ci-image-build-r7" x="463.6" y="849.6" textLength="793" clip-path="url(#breeze-ci-image-build-line-34)">(bookworm | bullseye)                                  [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="874" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-35)">│</text><text class="breeze-ci-image-build-r5" x="463.6" y="874" textLength="793" clip-path="url(#breeze-ci-image-build-line-35)">[default: bookworm]                                   &# [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="898.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-36)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="898.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-36)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="898.4" textLength="85.4" clip-path="url(#breeze-ci-image-build-line-36)">-python</text><text class="breeze-ci-image-build-r4" x="122" y="898.4" textLength="73.2" clip-path=" [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="922.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-37)">│</text><text class="breeze-ci-image-build-r1" x="463.6" y="922.8" textLength="976" clip-path="url(#breeze-ci-image-build-line-37)">something like: python:VERSION-slim-bookworm.                              [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="947.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-38)">│</text><text class="breeze-ci-image-build-r7" x="463.6" y="947.2" textLength="976" clip-path="url(#breeze-ci-image-build-line-38)">(TEXT)                                      [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="971.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-39)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="971.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-39)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="971.6" textLength="85.4" clip-path="url(#breeze-ci-image-build-line-39)">-commit</text><text class="breeze-ci-image-build-r4" x="122" y="971.6" textLength="48.8" clip-path=" [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="996" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-40)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="996" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-40)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="996" textLength="134.2" clip-path="url(#breeze-ci-image-build-line-40)">-additional</text><text class="breeze-ci-image-build-r4" x="170.8" y="996" textLength="219.6" clip-path=" [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1020.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-41)">│</text><text class="breeze-ci-image-build-r1" x="463.6" y="1020.4" textLength="976" clip-path="url(#breeze-ci-image-build-line-41)">itself).                                      [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1044.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-42)">│</text><text class="breeze-ci-image-build-r7" x="463.6" y="1044.8" textLength="976" clip-path="url(#breeze-ci-image-build-line-42)">(TEXT)                                     &# [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1069.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-43)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1069.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-43)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1069.2" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-43)">-install</text><text class="breeze-ci-image-build-r4" x="134.2" y="1069.2" textLength="280.6" cli [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1093.6" textLength="1464" clip-path="url(#breeze-ci-image-build-line-44)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-build-r1" x="1464" y="1093.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-44)">
+</text><text class="breeze-ci-image-build-r5" x="0" y="1118" textLength="24.4" clip-path="url(#breeze-ci-image-build-line-45)">╭─</text><text class="breeze-ci-image-build-r5" x="24.4" y="1118" textLength="597.8" clip-path="url(#breeze-ci-image-build-line-45)"> Selecting constraint location (for power users) </text><text class="breeze-ci-image-build-r5" x="622.2" y="1118" textLength="817.4" clip-path="url(#breeze-ci-image-build-line-45)">──────────────── [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1142.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-46)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1142.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-46)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1142.4" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-46)">-airflow</text><text class="breeze-ci-image-build-r4" x="134.2" y="1142.4" textLength="256.2" cli [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1166.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-47)">│</text><text class="breeze-ci-image-build-r1" x="451.4" y="1166.8" textLength="988.2" clip-path="url(#breeze-ci-image-build-line-47)">It could be full remote URL to the location file, or local file placed in        </text><text class="breeze-ci-image-b [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1191.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-48)">│</text><text class="breeze-ci-image-build-r1" x="451.4" y="1191.2" textLength="988.2" clip-path="url(#breeze-ci-image-build-line-48)">`docker-context-files` (in this case it has to start with                       [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1215.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-49)">│</text><text class="breeze-ci-image-build-r1" x="451.4" y="1215.6" textLength="988.2" clip-path="url(#breeze-ci-image-build-line-49)">/opt/airflow/docker-context-files).                                & [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1240" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-50)">│</text><text class="breeze-ci-image-build-r7" x="451.4" y="1240" textLength="988.2" clip-path="url(#breeze-ci-image-build-line-50)">(TEXT)                                      [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1264.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-51)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1264.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-51)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1264.4" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-51)">-airflow</text><text class="breeze-ci-image-build-r4" x="134.2" y="1264.4" textLength="207.4" cli [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1288.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-52)">│</text><text class="breeze-ci-image-build-r7" x="451.4" y="1288.8" textLength="866.2" clip-path="url(#breeze-ci-image-build-line-52)">(constraints-source-providers | constraints | constraints-no-providers)</text><text class="breeze-ci-image-build-r5" x="1451.8" y="1288.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-52)">│</ [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1313.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-53)">│</text><text class="breeze-ci-image-build-r5" x="451.4" y="1313.2" textLength="866.2" clip-path="url(#breeze-ci-image-build-line-53)">[default: constraints-source-providers]                               [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1337.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-54)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1337.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-54)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1337.6" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-54)">-airflow</text><text class="breeze-ci-image-build-r4" x="134.2" y="1337.6" textLength="268.4" cli [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1362" textLength="1464" clip-path="url(#breeze-ci-image-build-line-55)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-build-r1" x="1464" y="1362" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-55)">
+</text><text class="breeze-ci-image-build-r5" x="0" y="1386.4" textLength="24.4" clip-path="url(#breeze-ci-image-build-line-56)">╭─</text><text class="breeze-ci-image-build-r5" x="24.4" y="1386.4" textLength="634.4" clip-path="url(#breeze-ci-image-build-line-56)"> Choosing dependencies and extras (for power users) </text><text class="breeze-ci-image-build-r5" x="658.8" y="1386.4" textLength="780.8" clip-path="url(#breeze-ci-image-build-line-56)">── [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1410.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-57)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1410.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-57)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1410.8" textLength="134.2" clip-path="url(#breeze-ci-image-build-line-57)">-additional</text><text class="breeze-ci-image-build-r4" x="170.8" y="1410.8" textLength="183" c [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1435.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-58)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1435.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-58)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1435.2" textLength="134.2" clip-path="url(#breeze-ci-image-build-line-58)">-additional</text><text class="breeze-ci-image-build-r4" x="170.8" y="1435.2" textLength="146.4" [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1459.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-59)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1459.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-59)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1459.6" textLength="48.8" clip-path="url(#breeze-ci-image-build-line-59)">-dev</text><text class="breeze-ci-image-build-r4" x="85.4" y="1459.6" textLength="109.8" clip-pat [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1484" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-60)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1484" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-60)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1484" textLength="134.2" clip-path="url(#breeze-ci-image-build-line-60)">-additional</text><text class="breeze-ci-image-build-r4" x="170.8" y="1484" textLength="158.6" clip-pa [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1508.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-61)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1508.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-61)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1508.4" textLength="48.8" clip-path="url(#breeze-ci-image-build-line-61)">-dev</text><text class="breeze-ci-image-build-r4" x="85.4" y="1508.4" textLength="146.4" clip-pat [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1532.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-62)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1532.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-62)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1532.8" textLength="134.2" clip-path="url(#breeze-ci-image-build-line-62)">-additional</text><text class="breeze-ci-image-build-r4" x="170.8" y="1532.8" textLength="195.2" [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1557.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-63)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1557.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-63)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1557.2" textLength="134.2" clip-path="url(#breeze-ci-image-build-line-63)">-additional</text><text class="breeze-ci-image-build-r4" x="170.8" y="1557.2" textLength="146.4" [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1581.6" textLength="1464" clip-path="url(#breeze-ci-image-build-line-64)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-build-r1" x="1464" y="1581.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-64)">
+</text><text class="breeze-ci-image-build-r5" x="0" y="1606" textLength="24.4" clip-path="url(#breeze-ci-image-build-line-65)">╭─</text><text class="breeze-ci-image-build-r5" x="24.4" y="1606" textLength="268.4" clip-path="url(#breeze-ci-image-build-line-65)"> Backtracking options </text><text class="breeze-ci-image-build-r5" x="292.8" y="1606" textLength="1146.8" clip-path="url(#breeze-ci-image-build-line-65)">────────────────────────────────────────────────────────────── [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1630.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-66)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1630.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-66)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1630.4" textLength="73.2" clip-path="url(#breeze-ci-image-build-line-66)">-build</text><text class="breeze-ci-image-build-r4" x="109.8" y="1630.4" textLength="195.2" clip- [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1654.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-67)">│</text><text class="breeze-ci-image-build-r1" x="549" y="1654.8" textLength="890.6" clip-path="url(#breeze-ci-image-build-line-67)">backtracking problems.                                   [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1679.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-68)">│</text><text class="breeze-ci-image-build-r7" x="549" y="1679.2" textLength="890.6" clip-path="url(#breeze-ci-image-build-line-68)">(INTEGER)                                      [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1703.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-69)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1703.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-69)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1703.6" textLength="73.2" clip-path="url(#breeze-ci-image-build-line-69)">-eager</text><text class="breeze-ci-image-build-r4" x="109.8" y="1703.6" textLength="390.4" clip- [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1728" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-70)">│</text><text class="breeze-ci-image-build-r1" x="549" y="1728" textLength="890.6" clip-path="url(#breeze-ci-image-build-line-70)">(see `breeze ci find-backtracking-candidates`).                          </text><text cla [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1752.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-71)">│</text><text class="breeze-ci-image-build-r7" x="549" y="1752.4" textLength="890.6" clip-path="url(#breeze-ci-image-build-line-71)">(TEXT)                                     &# [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1776.8" textLength="1464" clip-path="url(#breeze-ci-image-build-line-72)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-build-r1" x="1464" y="1776.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-72)">
+</text><text class="breeze-ci-image-build-r5" x="0" y="1801.2" textLength="24.4" clip-path="url(#breeze-ci-image-build-line-73)">╭─</text><text class="breeze-ci-image-build-r5" x="24.4" y="1801.2" textLength="622.2" clip-path="url(#breeze-ci-image-build-line-73)"> Preparing cache and push (for maintainers and CI) </text><text class="breeze-ci-image-build-r5" x="646.6" y="1801.2" textLength="793" clip-path="url(#breeze-ci-image-build-line-73)"> [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1825.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-74)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1825.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-74)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1825.6" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-74)">-builder</text><text class="breeze-ci-image-build-r1" x="341.6" y="1825.6" textLength="756.4" cli [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1850" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-75)">│</text><text class="breeze-ci-image-build-r5" x="341.6" y="1850" textLength="756.4" clip-path="url(#breeze-ci-image-build-line-75)">[default: autodetect]                                  &# [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1874.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-76)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1874.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-76)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1874.4" textLength="109.8" clip-path="url(#breeze-ci-image-build-line-76)">-platform</text><text class="breeze-ci-image-build-r1" x="341.6" y="1874.4" textLength="329.4" c [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1898.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-77)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1898.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-77)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1898.8" textLength="61" clip-path="url(#breeze-ci-image-build-line-77)">-push</text><text class="breeze-ci-image-build-r1" x="341.6" y="1898.8" textLength="353.8" clip-pat [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1923.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-78)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1923.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-78)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1923.2" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-78)">-prepare</text><text class="breeze-ci-image-build-r4" x="134.2" y="1923.2" textLength="158.6" cli [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1947.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-79)">│</text><text class="breeze-ci-image-build-r1" x="341.6" y="1947.6" textLength="1098" clip-path="url(#breeze-ci-image-build-line-79)">image).                                      [...]
</text><text class="breeze-ci-image-build-r5" x="0" y="1972" textLength="1464" clip-path="url(#breeze-ci-image-build-line-80)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-build-r1" x="1464" y="1972" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-80)">
-</text><text class="breeze-ci-image-build-r5" x="0" y="1996.4" textLength="24.4" clip-path="url(#breeze-ci-image-build-line-81)">╭─</text><text class="breeze-ci-image-build-r5" x="24.4" y="1996.4" textLength="195.2" clip-path="url(#breeze-ci-image-build-line-81)"> Common options </text><text class="breeze-ci-image-build-r5" x="219.6" y="1996.4" textLength="1220" clip-path="url(#breeze-ci-image-build-line-81)">──────────────────────────────────────────────────────────────── [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="2020.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-82)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="2020.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-82)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="2020.8" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-82)">-verbose</text><text class="breeze-ci-image-build-r6" x="158.6" y="2020.8" textLength="24.4" clip [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="2045.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-83)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="2045.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-83)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="2045.2" textLength="48.8" clip-path="url(#breeze-ci-image-build-line-83)">-dry</text><text class="breeze-ci-image-build-r4" x="85.4" y="2045.2" textLength="48.8" clip-path [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="2069.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-84)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="2069.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-84)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="2069.6" textLength="85.4" clip-path="url(#breeze-ci-image-build-line-84)">-answer</text><text class="breeze-ci-image-build-r6" x="158.6" y="2069.6" textLength="24.4" clip- [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="2094" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-85)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="2094" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-85)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="2094" textLength="61" clip-path="url(#breeze-ci-image-build-line-85)">-help</text><text class="breeze-ci-image-build-r6" x="158.6" y="2094" textLength="24.4" clip-path="url(#b [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="2118.4" textLength="1464" clip-path="url(#breeze-ci-image-build-line-86)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-build-r1" x="1464" y="2118.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-86)">
+</text><text class="breeze-ci-image-build-r5" x="0" y="1996.4" textLength="24.4" clip-path="url(#breeze-ci-image-build-line-81)">╭─</text><text class="breeze-ci-image-build-r5" x="24.4" y="1996.4" textLength="280.6" clip-path="url(#breeze-ci-image-build-line-81)"> Github authentication </text><text class="breeze-ci-image-build-r5" x="305" y="1996.4" textLength="1134.6" clip-path="url(#breeze-ci-image-build-line-81)">───────────────────────────────────────────────────────── [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="2020.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-82)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="2020.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-82)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="2020.8" textLength="85.4" clip-path="url(#breeze-ci-image-build-line-82)">-github</text><text class="breeze-ci-image-build-r4" x="122" y="2020.8" textLength="134.2" clip-p [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="2045.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-83)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="2045.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-83)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="2045.2" textLength="85.4" clip-path="url(#breeze-ci-image-build-line-83)">-github</text><text class="breeze-ci-image-build-r4" x="122" y="2045.2" textLength="73.2" clip-pa [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="2069.6" textLength="1464" clip-path="url(#breeze-ci-image-build-line-84)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-build-r1" x="1464" y="2069.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-84)">
+</text><text class="breeze-ci-image-build-r5" x="0" y="2094" textLength="24.4" clip-path="url(#breeze-ci-image-build-line-85)">╭─</text><text class="breeze-ci-image-build-r5" x="24.4" y="2094" textLength="195.2" clip-path="url(#breeze-ci-image-build-line-85)"> Common options </text><text class="breeze-ci-image-build-r5" x="219.6" y="2094" textLength="1220" clip-path="url(#breeze-ci-image-build-line-85)">────────────────────────────────────────────────────────────────────── [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="2118.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-86)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="2118.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-86)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="2118.4" textLength="97.6" clip-path="url(#breeze-ci-image-build-line-86)">-verbose</text><text class="breeze-ci-image-build-r6" x="158.6" y="2118.4" textLength="24.4" clip [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="2142.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-87)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="2142.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-87)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="2142.8" textLength="48.8" clip-path="url(#breeze-ci-image-build-line-87)">-dry</text><text class="breeze-ci-image-build-r4" x="85.4" y="2142.8" textLength="48.8" clip-path [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="2167.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-88)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="2167.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-88)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="2167.2" textLength="85.4" clip-path="url(#breeze-ci-image-build-line-88)">-answer</text><text class="breeze-ci-image-build-r6" x="158.6" y="2167.2" textLength="24.4" clip- [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="2191.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-89)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="2191.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-89)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="2191.6" textLength="61" clip-path="url(#breeze-ci-image-build-line-89)">-help</text><text class="breeze-ci-image-build-r6" x="158.6" y="2191.6" textLength="24.4" clip-path [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="2216" textLength="1464" clip-path="url(#breeze-ci-image-build-line-90)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-build-r1" x="1464" y="2216" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-90)">
</text>
</g>
</g>
diff --git a/images/breeze/output_ci-image_build.txt b/images/breeze/output_ci-image_build.txt
index f659f125e8..9c882681f1 100644
--- a/images/breeze/output_ci-image_build.txt
+++ b/images/breeze/output_ci-image_build.txt
@@ -1 +1 @@
-c405885e9a457ff1ca8caa8a3ebab6e5
+3109e8a0e9ff98e6d0b802e6ce09192a
diff --git a/images/breeze/output_prod-image_build.svg b/images/breeze/output_prod-image_build.svg
index 079f3f39e6..729eafed98 100644
--- a/images/breeze/output_prod-image_build.svg
+++ b/images/breeze/output_prod-image_build.svg
@@ -1,4 +1,4 @@
-<svg class="rich-terminal" viewBox="0 0 1482 2441.2" xmlns="http://www.w3.org/2000/svg">
+<svg class="rich-terminal" viewBox="0 0 1482 2563.2" xmlns="http://www.w3.org/2000/svg">
<!-- Generated with Rich https://www.textualize.io -->
<style>
@@ -43,7 +43,7 @@
<defs>
<clipPath id="breeze-prod-image-build-clip-terminal">
- <rect x="0" y="0" width="1463.0" height="2390.2" />
+ <rect x="0" y="0" width="1463.0" height="2512.2" />
</clipPath>
<clipPath id="breeze-prod-image-build-line-0">
<rect x="0" y="1.5" width="1464" height="24.65"/>
@@ -336,9 +336,24 @@
<clipPath id="breeze-prod-image-build-line-96">
<rect x="0" y="2343.9" width="1464" height="24.65"/>
</clipPath>
+<clipPath id="breeze-prod-image-build-line-97">
+ <rect x="0" y="2368.3" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-prod-image-build-line-98">
+ <rect x="0" y="2392.7" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-prod-image-build-line-99">
+ <rect x="0" y="2417.1" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-prod-image-build-line-100">
+ <rect x="0" y="2441.5" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-prod-image-build-line-101">
+ <rect x="0" y="2465.9" width="1464" height="24.65"/>
+ </clipPath>
</defs>
- <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="2439.2" rx="8"/><text class="breeze-prod-image-build-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: prod-image build</text>
+ <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="2561.2" rx="8"/><text class="breeze-prod-image-build-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: prod-image build</text>
<g transform="translate(26,22)">
<circle cx="0" cy="0" r="7" fill="#ff5f57"/>
<circle cx="22" cy="0" r="7" fill="#febc2e"/>
@@ -363,89 +378,94 @@
</text><text class="breeze-prod-image-build-r5" x="0" y="312.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-12)">│</text><text class="breeze-prod-image-build-r1" x="402.6" y="312.8" textLength="353.8" clip-path="url(#breeze-prod-image-build-line-12)">you build or pull image with </text><text class="breeze-prod-image-build-r4" x="756.4" y="312.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-12)">-</text><text class="breeze [...]
</text><text class="breeze-prod-image-build-r5" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-13)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="337.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-13)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="337.2" textLength="85.4" clip-path="url(#breeze-prod-image-build-line-13)">-docker</text><text class="breeze-prod-image-build-r4" x="122" y="337.2" textLength="73. [...]
</text><text class="breeze-prod-image-build-r5" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-14)">│</text><text class="breeze-prod-image-build-r5" x="402.6" y="361.6" textLength="549" clip-path="url(#breeze-prod-image-build-line-14)">[default: registry]                          </text><text class="breeze-prod-image-build- [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="386" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-15)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="386" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-15)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="386" textLength="73.2" clip-path="url(#breeze-prod-image-build-line-15)">-build</text><text class="breeze-prod-image-build-r4" x="109.8" y="386" textLength="109.8" cli [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="410.4" textLength="1464" clip-path="url(#breeze-prod-image-build-line-16)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-build-r1" x="1464" y="410.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-16)">
-</text><text class="breeze-prod-image-build-r5" x="0" y="434.8" textLength="24.4" clip-path="url(#breeze-prod-image-build-line-17)">╭─</text><text class="breeze-prod-image-build-r5" x="24.4" y="434.8" textLength="353.8" clip-path="url(#breeze-prod-image-build-line-17)"> Building images in parallel </text><text class="breeze-prod-image-build-r5" x="378.2" y="434.8" textLength="1061.4" clip-path="url(#breeze-prod-image-build-line-17)">────────────────────────────── [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="459.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-18)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="459.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-18)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="459.2" textLength="48.8" clip-path="url(#breeze-prod-image-build-line-18)">-run</text><text class="breeze-prod-image-build-r4" x="85.4" y="459.2" textLength="146.4 [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-19)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="483.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-19)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="483.6" textLength="146.4" clip-path="url(#breeze-prod-image-build-line-19)">-parallelism</text><text class="breeze-prod-image-build-r1" x="378.2" y="483.6" textLen [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="508" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-20)">│</text><text class="breeze-prod-image-build-r7" x="378.2" y="508" textLength="915" clip-path="url(#breeze-prod-image-build-line-20)">(INTEGER RANGE)                                   [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-21)">│</text><text class="breeze-prod-image-build-r5" x="378.2" y="532.4" textLength="915" clip-path="url(#breeze-prod-image-build-line-21)">[default: 4; 1<=x<=8]                               & [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="556.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-22)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="556.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-22)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="556.8" textLength="85.4" clip-path="url(#breeze-prod-image-build-line-22)">-python</text><text class="breeze-prod-image-build-r4" x="122" y="556.8" textLength="109 [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="581.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-23)">│</text><text class="breeze-prod-image-build-r5" x="378.2" y="581.2" textLength="951.6" clip-path="url(#breeze-prod-image-build-line-23)">[default: 3.8 3.9 3.10 3.11]                              [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="605.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-24)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="605.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-24)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="605.6" textLength="61" clip-path="url(#breeze-prod-image-build-line-24)">-skip</text><text class="breeze-prod-image-build-r4" x="97.6" y="605.6" textLength="97.6" [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="630" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-25)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="630" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-25)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="630" textLength="73.2" clip-path="url(#breeze-prod-image-build-line-25)">-debug</text><text class="breeze-prod-image-build-r4" x="109.8" y="630" textLength="122" clip- [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="654.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-26)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="654.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-26)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="654.4" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-26)">-include</text><text class="breeze-prod-image-build-r4" x="134.2" y="654.4" textLength=" [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="678.8" textLength="1464" clip-path="url(#breeze-prod-image-build-line-27)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-build-r1" x="1464" y="678.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-27)">
-</text><text class="breeze-prod-image-build-r5" x="0" y="703.2" textLength="24.4" clip-path="url(#breeze-prod-image-build-line-28)">╭─</text><text class="breeze-prod-image-build-r5" x="24.4" y="703.2" textLength="390.4" clip-path="url(#breeze-prod-image-build-line-28)"> Options for customizing images </text><text class="breeze-prod-image-build-r5" x="414.8" y="703.2" textLength="1024.8" clip-path="url(#breeze-prod-image-build-line-28)">─────────────────────────── [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="727.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-29)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="727.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-29)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="727.6" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-29)">-install</text><text class="breeze-prod-image-build-r4" x="134.2" y="727.6" textLength=" [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="752" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-30)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="752" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-30)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="752" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-30)">-airflow</text><text class="breeze-prod-image-build-r4" x="134.2" y="752" textLength="85.4" cl [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="776.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-31)">│</text><text class="breeze-prod-image-build-r7" x="463.6" y="776.4" textLength="976" clip-path="url(#breeze-prod-image-build-line-31)">(TEXT)                                    &# [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="800.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-32)">│</text><text class="breeze-prod-image-build-r5" x="463.6" y="800.8" textLength="976" clip-path="url(#breeze-prod-image-build-line-32)">[default:                                     [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="825.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-33)">│</text><text class="breeze-prod-image-build-r5" x="463.6" y="825.2" textLength="976" clip-path="url(#breeze-prod-image-build-line-33)">aiobotocore,amazon,async,celery,cncf.kubernetes,daskexecutor,docker,elasticsear…</text><text class="breeze-prod-image-build-r5" x="1451.8" y="825.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-33)">│</text [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="849.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-34)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="849.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-34)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="849.6" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-34)">-airflow</text><text class="breeze-prod-image-build-r4" x="134.2" y="849.6" textLength=" [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="874" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-35)">│</text><text class="breeze-prod-image-build-r1" x="463.6" y="874" textLength="976" clip-path="url(#breeze-prod-image-build-line-35)">file. It could be full remote URL to the location file, or local file placed in </text><text class="breeze-prod-image-build-r5" x="1451.8" y="874" t [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="898.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-36)">│</text><text class="breeze-prod-image-build-r1" x="463.6" y="898.4" textLength="976" clip-path="url(#breeze-prod-image-build-line-36)">`docker-context-files` (in this case it has to start with                     & [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="922.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-37)">│</text><text class="breeze-prod-image-build-r1" x="463.6" y="922.8" textLength="976" clip-path="url(#breeze-prod-image-build-line-37)">/opt/airflow/docker-context-files).                                [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="947.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-38)">│</text><text class="breeze-prod-image-build-r7" x="463.6" y="947.2" textLength="976" clip-path="url(#breeze-prod-image-build-line-38)">(TEXT)                                    &# [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="971.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-39)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="971.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-39)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="971.6" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-39)">-airflow</text><text class="breeze-prod-image-build-r4" x="134.2" y="971.6" textLength=" [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="996" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-40)">│</text><text class="breeze-prod-image-build-r7" x="463.6" y="996" textLength="866.2" clip-path="url(#breeze-prod-image-build-line-40)">(constraints | constraints-no-providers | constraints-source-providers)</text><text class="breeze-prod-image-build-r5" x="1451.8" y="996" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-40)"> [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1020.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-41)">│</text><text class="breeze-prod-image-build-r5" x="463.6" y="1020.4" textLength="866.2" clip-path="url(#breeze-prod-image-build-line-41)">[default: constraints]                                & [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1044.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-42)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1044.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-42)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1044.8" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-42)">-airflow</text><text class="breeze-prod-image-build-r4" x="134.2" y="1044.8" textLeng [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1069.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-43)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1069.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-43)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1069.2" textLength="85.4" clip-path="url(#breeze-prod-image-build-line-43)">-python</text><text class="breeze-prod-image-build-r4" x="122" y="1069.2" textLength= [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1093.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-44)">│</text><text class="breeze-prod-image-build-r1" x="463.6" y="1093.6" textLength="976" clip-path="url(#breeze-prod-image-build-line-44)">something like: python:VERSION-slim-bookworm.                             [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1118" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-45)">│</text><text class="breeze-prod-image-build-r7" x="463.6" y="1118" textLength="976" clip-path="url(#breeze-prod-image-build-line-45)">(TEXT)                                     [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1142.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-46)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1142.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-46)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1142.4" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-46)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1142.4" text [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1166.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-47)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1166.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-47)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1166.8" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-47)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1166.8" text [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1191.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-48)">│</text><text class="breeze-prod-image-build-r1" x="463.6" y="1191.2" textLength="976" clip-path="url(#breeze-prod-image-build-line-48)">itself).                                    [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1215.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-49)">│</text><text class="breeze-prod-image-build-r7" x="463.6" y="1215.6" textLength="976" clip-path="url(#breeze-prod-image-build-line-49)">(TEXT)                                     [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1240" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-50)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1240" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-50)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1240" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-50)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1240" textLength=" [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1264.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-51)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1264.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-51)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1264.4" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-51)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1264.4" text [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1288.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-52)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1288.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-52)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1288.8" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-52)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1288.8" text [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1313.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-53)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1313.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-53)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1313.2" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-53)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1313.2" text [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1337.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-54)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1337.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-54)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1337.6" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-54)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1337.6" text [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1362" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-55)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1362" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-55)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1362" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-55)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1362" textLength=" [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1386.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-56)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1386.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-56)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1386.4" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-56)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1386.4" text [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1410.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-57)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1410.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-57)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1410.8" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-57)">-runtime</text><text class="breeze-prod-image-build-r4" x="134.2" y="1410.8" textLeng [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1435.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-58)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1435.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-58)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1435.2" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-58)">-runtime</text><text class="breeze-prod-image-build-r4" x="134.2" y="1435.2" textLeng [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1459.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-59)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1459.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-59)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1459.6" textLength="48.8" clip-path="url(#breeze-prod-image-build-line-59)">-dev</text><text class="breeze-prod-image-build-r4" x="85.4" y="1459.6" textLength="1 [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1484" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-60)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1484" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-60)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1484" textLength="48.8" clip-path="url(#breeze-prod-image-build-line-60)">-dev</text><text class="breeze-prod-image-build-r4" x="85.4" y="1484" textLength="146.4" cl [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1508.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-61)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1508.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-61)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1508.4" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-61)">-version</text><text class="breeze-prod-image-build-r4" x="134.2" y="1508.4" textLeng [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1532.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-62)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1532.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-62)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1532.8" textLength="85.4" clip-path="url(#breeze-prod-image-build-line-62)">-commit</text><text class="breeze-prod-image-build-r4" x="122" y="1532.8" textLength= [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1557.2" textLength="1464" clip-path="url(#breeze-prod-image-build-line-63)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-build-r1" x="1464" y="1557.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-63)">
-</text><text class="breeze-prod-image-build-r5" x="0" y="1581.6" textLength="24.4" clip-path="url(#breeze-prod-image-build-line-64)">╭─</text><text class="breeze-prod-image-build-r5" x="24.4" y="1581.6" textLength="817.4" clip-path="url(#breeze-prod-image-build-line-64)"> Advanced customization options (for specific customization needs) </text><text class="breeze-prod-image-build-r5" x="841.8" y="1581.6" textLength="597.8" clip-path="url(#breeze-pr [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1606" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-65)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1606" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-65)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1606" textLength="85.4" clip-path="url(#breeze-prod-image-build-line-65)">-debian</text><text class="breeze-prod-image-build-r4" x="122" y="1606" textLength="97.6" c [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1630.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-66)">│</text><text class="breeze-prod-image-build-r7" x="536.8" y="1630.4" textLength="793" clip-path="url(#breeze-prod-image-build-line-66)">(bookworm | bullseye)                                 [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1654.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-67)">│</text><text class="breeze-prod-image-build-r5" x="536.8" y="1654.8" textLength="793" clip-path="url(#breeze-prod-image-build-line-67)">[default: bookworm]                                  [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1679.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-68)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1679.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-68)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1679.2" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-68)">-install</text><text class="breeze-prod-image-build-r4" x="134.2" y="1679.2" textLeng [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1703.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-69)">│</text><text class="breeze-prod-image-build-r1" x="536.8" y="1703.6" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-69)">Implies </text><text class="breeze-prod-image-build-r4" x="634.4" y="1703.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-69)">-</text><text class="breeze-prod-image-build-r4" x="646.6" y="1703.6" t [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1728" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-70)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1728" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-70)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1728" textLength="48.8" clip-path="url(#breeze-prod-image-build-line-70)">-use</text><text class="breeze-prod-image-build-r4" x="85.4" y="1728" textLength="402.6" cl [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1752.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-71)">│</text><text class="breeze-prod-image-build-r1" x="536.8" y="1752.4" textLength="902.8" clip-path="url(#breeze-prod-image-build-line-71)">constraints store in docker-context-files or from github.                 </text><text class="breeze-prod [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1776.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-72)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1776.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-72)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1776.8" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-72)">-cleanup</text><text class="breeze-prod-image-build-r4" x="134.2" y="1776.8" textLeng [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1801.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-73)">│</text><text class="breeze-prod-image-build-r1" x="536.8" y="1801.2" textLength="170.8" clip-path="url(#breeze-prod-image-build-line-73)">together with </text><text class="breeze-prod-image-build-r4" x="707.6" y="1801.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-73)">-</text><text class="breeze-prod-image-build-r4" x="719.8" [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1825.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-74)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1825.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-74)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1825.6" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-74)">-disable</text><text class="breeze-prod-image-build-r4" x="134.2" y="1825.6" textLeng [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1850" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-75)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1850" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-75)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1850" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-75)">-disable</text><text class="breeze-prod-image-build-r4" x="134.2" y="1850" textLength="317. [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1874.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-76)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1874.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-76)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1874.4" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-76)">-disable</text><text class="breeze-prod-image-build-r4" x="134.2" y="1874.4" textLeng [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1898.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-77)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1898.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-77)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1898.8" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-77)">-disable</text><text class="breeze-prod-image-build-r4" x="134.2" y="1898.8" textLeng [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1923.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-78)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1923.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-78)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1923.2" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-78)">-install</text><text class="breeze-prod-image-build-r4" x="134.2" y="1923.2" textLeng [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1947.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-79)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1947.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-79)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1947.6" textLength="158.6" clip-path="url(#breeze-prod-image-build-line-79)">-installation</text><text class="breeze-prod-image-build-r4" x="195.2" y="1947.6" te [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1972" textLength="1464" clip-path="url(#breeze-prod-image-build-line-80)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-build-r1" x="1464" y="1972" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-80)">
-</text><text class="breeze-prod-image-build-r5" x="0" y="1996.4" textLength="24.4" clip-path="url(#breeze-prod-image-build-line-81)">╭─</text><text class="breeze-prod-image-build-r5" x="24.4" y="1996.4" textLength="622.2" clip-path="url(#breeze-prod-image-build-line-81)"> Preparing cache and push (for maintainers and CI) </text><text class="breeze-prod-image-build-r5" x="646.6" y="1996.4" textLength="793" clip-path="url(#breeze-prod-image-buil [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2020.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-82)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2020.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-82)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2020.8" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-82)">-builder</text><text class="breeze-prod-image-build-r1" x="341.6" y="2020.8" textLeng [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2045.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-83)">│</text><text class="breeze-prod-image-build-r5" x="341.6" y="2045.2" textLength="756.4" clip-path="url(#breeze-prod-image-build-line-83)">[default: autodetect]                                &# [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2069.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-84)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2069.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-84)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2069.6" textLength="109.8" clip-path="url(#breeze-prod-image-build-line-84)">-platform</text><text class="breeze-prod-image-build-r1" x="341.6" y="2069.6" textLe [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2094" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-85)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2094" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-85)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2094" textLength="61" clip-path="url(#breeze-prod-image-build-line-85)">-push</text><text class="breeze-prod-image-build-r1" x="341.6" y="2094" textLength="353.8" cl [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2118.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-86)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2118.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-86)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2118.4" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-86)">-prepare</text><text class="breeze-prod-image-build-r4" x="134.2" y="2118.4" textLeng [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2142.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-87)">│</text><text class="breeze-prod-image-build-r1" x="341.6" y="2142.8" textLength="1098" clip-path="url(#breeze-prod-image-build-line-87)">image).                                    [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2167.2" textLength="1464" clip-path="url(#breeze-prod-image-build-line-88)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-build-r1" x="1464" y="2167.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-88)">
-</text><text class="breeze-prod-image-build-r5" x="0" y="2191.6" textLength="24.4" clip-path="url(#breeze-prod-image-build-line-89)">╭─</text><text class="breeze-prod-image-build-r5" x="24.4" y="2191.6" textLength="280.6" clip-path="url(#breeze-prod-image-build-line-89)"> Github authentication </text><text class="breeze-prod-image-build-r5" x="305" y="2191.6" textLength="1134.6" clip-path="url(#breeze-prod-image-build-line-89)">───────────────────────────────────────────── [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2216" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-90)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2216" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-90)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2216" textLength="85.4" clip-path="url(#breeze-prod-image-build-line-90)">-github</text><text class="breeze-prod-image-build-r4" x="122" y="2216" textLength="134.2" [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2240.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-91)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2240.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-91)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2240.4" textLength="85.4" clip-path="url(#breeze-prod-image-build-line-91)">-github</text><text class="breeze-prod-image-build-r4" x="122" y="2240.4" textLength= [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="386" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-15)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="386" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-15)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="386" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-15)">-version</text><text class="breeze-prod-image-build-r4" x="134.2" y="386" textLength="195.2" c [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="410.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-16)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="410.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-16)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="410.4" textLength="73.2" clip-path="url(#breeze-prod-image-build-line-16)">-build</text><text class="breeze-prod-image-build-r4" x="109.8" y="410.4" textLength="10 [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="434.8" textLength="1464" clip-path="url(#breeze-prod-image-build-line-17)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-build-r1" x="1464" y="434.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-17)">
+</text><text class="breeze-prod-image-build-r5" x="0" y="459.2" textLength="24.4" clip-path="url(#breeze-prod-image-build-line-18)">╭─</text><text class="breeze-prod-image-build-r5" x="24.4" y="459.2" textLength="353.8" clip-path="url(#breeze-prod-image-build-line-18)"> Building images in parallel </text><text class="breeze-prod-image-build-r5" x="378.2" y="459.2" textLength="1061.4" clip-path="url(#breeze-prod-image-build-line-18)">────────────────────────────── [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-19)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="483.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-19)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="483.6" textLength="48.8" clip-path="url(#breeze-prod-image-build-line-19)">-run</text><text class="breeze-prod-image-build-r4" x="85.4" y="483.6" textLength="146.4 [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="508" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-20)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="508" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-20)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="508" textLength="146.4" clip-path="url(#breeze-prod-image-build-line-20)">-parallelism</text><text class="breeze-prod-image-build-r1" x="378.2" y="508" textLength="915 [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-21)">│</text><text class="breeze-prod-image-build-r7" x="378.2" y="532.4" textLength="915" clip-path="url(#breeze-prod-image-build-line-21)">(INTEGER RANGE)                                   [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="556.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-22)">│</text><text class="breeze-prod-image-build-r5" x="378.2" y="556.8" textLength="915" clip-path="url(#breeze-prod-image-build-line-22)">[default: 4; 1<=x<=8]                               & [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="581.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-23)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="581.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-23)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="581.2" textLength="85.4" clip-path="url(#breeze-prod-image-build-line-23)">-python</text><text class="breeze-prod-image-build-r4" x="122" y="581.2" textLength="109 [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="605.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-24)">│</text><text class="breeze-prod-image-build-r5" x="378.2" y="605.6" textLength="951.6" clip-path="url(#breeze-prod-image-build-line-24)">[default: 3.8 3.9 3.10 3.11]                              [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="630" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-25)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="630" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-25)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="630" textLength="61" clip-path="url(#breeze-prod-image-build-line-25)">-skip</text><text class="breeze-prod-image-build-r4" x="97.6" y="630" textLength="97.6" clip-pat [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="654.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-26)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="654.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-26)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="654.4" textLength="73.2" clip-path="url(#breeze-prod-image-build-line-26)">-debug</text><text class="breeze-prod-image-build-r4" x="109.8" y="654.4" textLength="12 [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="678.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-27)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="678.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-27)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="678.8" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-27)">-include</text><text class="breeze-prod-image-build-r4" x="134.2" y="678.8" textLength=" [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="703.2" textLength="1464" clip-path="url(#breeze-prod-image-build-line-28)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-build-r1" x="1464" y="703.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-28)">
+</text><text class="breeze-prod-image-build-r5" x="0" y="727.6" textLength="24.4" clip-path="url(#breeze-prod-image-build-line-29)">╭─</text><text class="breeze-prod-image-build-r5" x="24.4" y="727.6" textLength="512.4" clip-path="url(#breeze-prod-image-build-line-29)"> Advanced build options (for power users) </text><text class="breeze-prod-image-build-r5" x="536.8" y="727.6" textLength="902.8" clip-path="url(#breeze-prod-image-build-line-29)">──────── [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="752" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-30)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="752" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-30)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="752" textLength="85.4" clip-path="url(#breeze-prod-image-build-line-30)">-debian</text><text class="breeze-prod-image-build-r4" x="122" y="752" textLength="97.6" clip- [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="776.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-31)">│</text><text class="breeze-prod-image-build-r7" x="463.6" y="776.4" textLength="793" clip-path="url(#breeze-prod-image-build-line-31)">(bookworm | bullseye)                                & [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="800.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-32)">│</text><text class="breeze-prod-image-build-r5" x="463.6" y="800.8" textLength="793" clip-path="url(#breeze-prod-image-build-line-32)">[default: bookworm]                                 &# [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="825.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-33)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="825.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-33)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="825.2" textLength="85.4" clip-path="url(#breeze-prod-image-build-line-33)">-python</text><text class="breeze-prod-image-build-r4" x="122" y="825.2" textLength="73. [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="849.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-34)">│</text><text class="breeze-prod-image-build-r1" x="463.6" y="849.6" textLength="976" clip-path="url(#breeze-prod-image-build-line-34)">something like: python:VERSION-slim-bookworm.                            & [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="874" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-35)">│</text><text class="breeze-prod-image-build-r7" x="463.6" y="874" textLength="976" clip-path="url(#breeze-prod-image-build-line-35)">(TEXT)                                      [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="898.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-36)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="898.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-36)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="898.4" textLength="85.4" clip-path="url(#breeze-prod-image-build-line-36)">-commit</text><text class="breeze-prod-image-build-r4" x="122" y="898.4" textLength="48. [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="922.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-37)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="922.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-37)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="922.8" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-37)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="922.8" textLeng [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="947.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-38)">│</text><text class="breeze-prod-image-build-r1" x="463.6" y="947.2" textLength="976" clip-path="url(#breeze-prod-image-build-line-38)">itself).                                     [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="971.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-39)">│</text><text class="breeze-prod-image-build-r7" x="463.6" y="971.6" textLength="976" clip-path="url(#breeze-prod-image-build-line-39)">(TEXT)                                    &# [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="996" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-40)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="996" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-40)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="996" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-40)">-install</text><text class="breeze-prod-image-build-r4" x="134.2" y="996" textLength="280.6" c [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1020.4" textLength="1464" clip-path="url(#breeze-prod-image-build-line-41)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-build-r1" x="1464" y="1020.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-41)">
+</text><text class="breeze-prod-image-build-r5" x="0" y="1044.8" textLength="24.4" clip-path="url(#breeze-prod-image-build-line-42)">╭─</text><text class="breeze-prod-image-build-r5" x="24.4" y="1044.8" textLength="597.8" clip-path="url(#breeze-prod-image-build-line-42)"> Selecting constraint location (for power users) </text><text class="breeze-prod-image-build-r5" x="622.2" y="1044.8" textLength="817.4" clip-path="url(#breeze-prod-image-build-line-42) [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1069.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-43)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1069.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-43)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1069.2" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-43)">-airflow</text><text class="breeze-prod-image-build-r4" x="134.2" y="1069.2" textLeng [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1093.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-44)">│</text><text class="breeze-prod-image-build-r1" x="451.4" y="1093.6" textLength="988.2" clip-path="url(#breeze-prod-image-build-line-44)">It could be full remote URL to the location file, or local file placed in        </text><text class="breeze-pr [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1118" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-45)">│</text><text class="breeze-prod-image-build-r1" x="451.4" y="1118" textLength="988.2" clip-path="url(#breeze-prod-image-build-line-45)">`docker-context-files` (in this case it has to start with                     & [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1142.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-46)">│</text><text class="breeze-prod-image-build-r1" x="451.4" y="1142.4" textLength="988.2" clip-path="url(#breeze-prod-image-build-line-46)">/opt/airflow/docker-context-files).                                [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1166.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-47)">│</text><text class="breeze-prod-image-build-r7" x="451.4" y="1166.8" textLength="988.2" clip-path="url(#breeze-prod-image-build-line-47)">(TEXT)                                    [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1191.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-48)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1191.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-48)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1191.2" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-48)">-airflow</text><text class="breeze-prod-image-build-r4" x="134.2" y="1191.2" textLeng [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1215.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-49)">│</text><text class="breeze-prod-image-build-r7" x="451.4" y="1215.6" textLength="866.2" clip-path="url(#breeze-prod-image-build-line-49)">(constraints | constraints-no-providers | constraints-source-providers)</text><text class="breeze-prod-image-build-r5" x="1451.8" y="1215.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-l [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1240" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-50)">│</text><text class="breeze-prod-image-build-r5" x="451.4" y="1240" textLength="866.2" clip-path="url(#breeze-prod-image-build-line-50)">[default: constraints]                                  [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1264.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-51)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1264.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-51)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1264.4" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-51)">-airflow</text><text class="breeze-prod-image-build-r4" x="134.2" y="1264.4" textLeng [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1288.8" textLength="1464" clip-path="url(#breeze-prod-image-build-line-52)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-build-r1" x="1464" y="1288.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-52)">
+</text><text class="breeze-prod-image-build-r5" x="0" y="1313.2" textLength="24.4" clip-path="url(#breeze-prod-image-build-line-53)">╭─</text><text class="breeze-prod-image-build-r5" x="24.4" y="1313.2" textLength="634.4" clip-path="url(#breeze-prod-image-build-line-53)"> Choosing dependencies and extras (for power users) </text><text class="breeze-prod-image-build-r5" x="658.8" y="1313.2" textLength="780.8" clip-path="url(#breeze-prod-image-build- [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1337.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-54)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1337.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-54)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1337.6" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-54)">-airflow</text><text class="breeze-prod-image-build-r4" x="134.2" y="1337.6" textLeng [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1362" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-55)">│</text><text class="breeze-prod-image-build-r7" x="463.6" y="1362" textLength="976" clip-path="url(#breeze-prod-image-build-line-55)">(TEXT)                                     [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1386.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-56)">│</text><text class="breeze-prod-image-build-r5" x="463.6" y="1386.4" textLength="976" clip-path="url(#breeze-prod-image-build-line-56)">[default:                                    [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1410.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-57)">│</text><text class="breeze-prod-image-build-r5" x="463.6" y="1410.8" textLength="976" clip-path="url(#breeze-prod-image-build-line-57)">aiobotocore,amazon,async,celery,cncf.kubernetes,daskexecutor,docker,elasticsear…</text><text class="breeze-prod-image-build-r5" x="1451.8" y="1410.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-57)">│</t [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1435.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-58)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1435.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-58)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1435.2" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-58)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1435.2" text [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1459.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-59)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1459.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-59)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1459.6" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-59)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1459.6" text [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1484" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-60)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1484" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-60)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1484" textLength="48.8" clip-path="url(#breeze-prod-image-build-line-60)">-dev</text><text class="breeze-prod-image-build-r4" x="85.4" y="1484" textLength="109.8" cl [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1508.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-61)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1508.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-61)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1508.4" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-61)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1508.4" text [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1532.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-62)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1532.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-62)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1532.8" textLength="48.8" clip-path="url(#breeze-prod-image-build-line-62)">-dev</text><text class="breeze-prod-image-build-r4" x="85.4" y="1532.8" textLength="1 [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1557.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-63)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1557.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-63)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1557.2" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-63)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1557.2" text [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1581.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-64)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1581.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-64)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1581.6" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-64)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1581.6" text [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1606" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-65)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1606" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-65)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1606" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-65)">-runtime</text><text class="breeze-prod-image-build-r4" x="134.2" y="1606" textLength="109. [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1630.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-66)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1630.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-66)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1630.4" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-66)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1630.4" text [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1654.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-67)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1654.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-67)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1654.8" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-67)">-runtime</text><text class="breeze-prod-image-build-r4" x="134.2" y="1654.8" textLeng [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1679.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-68)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1679.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-68)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1679.2" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-68)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1679.2" text [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1703.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-69)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1703.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-69)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1703.6" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-69)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1703.6" text [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1728" textLength="1464" clip-path="url(#breeze-prod-image-build-line-70)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-build-r1" x="1464" y="1728" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-70)">
+</text><text class="breeze-prod-image-build-r5" x="0" y="1752.4" textLength="24.4" clip-path="url(#breeze-prod-image-build-line-71)">╭─</text><text class="breeze-prod-image-build-r5" x="24.4" y="1752.4" textLength="817.4" clip-path="url(#breeze-prod-image-build-line-71)"> Advanced customization options (for specific customization needs) </text><text class="breeze-prod-image-build-r5" x="841.8" y="1752.4" textLength="597.8" clip-path="url(#breeze-pr [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1776.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-72)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1776.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-72)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1776.8" textLength="158.6" clip-path="url(#breeze-prod-image-build-line-72)">-installation</text><text class="breeze-prod-image-build-r4" x="195.2" y="1776.8" te [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1801.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-73)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1801.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-73)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1801.2" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-73)">-install</text><text class="breeze-prod-image-build-r4" x="134.2" y="1801.2" textLeng [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1825.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-74)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1825.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-74)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1825.6" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-74)">-install</text><text class="breeze-prod-image-build-r4" x="134.2" y="1825.6" textLeng [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1850" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-75)">│</text><text class="breeze-prod-image-build-r1" x="536.8" y="1850" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-75)">Implies </text><text class="breeze-prod-image-build-r4" x="634.4" y="1850" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-75)">-</text><text class="breeze-prod-image-build-r4" x="646.6" y="1850" textLengt [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1874.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-76)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1874.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-76)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1874.4" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-76)">-cleanup</text><text class="breeze-prod-image-build-r4" x="134.2" y="1874.4" textLeng [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1898.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-77)">│</text><text class="breeze-prod-image-build-r1" x="536.8" y="1898.8" textLength="170.8" clip-path="url(#breeze-prod-image-build-line-77)">together with </text><text class="breeze-prod-image-build-r4" x="707.6" y="1898.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-77)">-</text><text class="breeze-prod-image-build-r4" x="719.8" [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1923.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-78)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1923.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-78)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1923.2" textLength="48.8" clip-path="url(#breeze-prod-image-build-line-78)">-use</text><text class="breeze-prod-image-build-r4" x="85.4" y="1923.2" textLength="4 [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1947.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-79)">│</text><text class="breeze-prod-image-build-r1" x="536.8" y="1947.6" textLength="902.8" clip-path="url(#breeze-prod-image-build-line-79)">constraints store in docker-context-files or from github.                 </text><text class="breeze-prod [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1972" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-80)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1972" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-80)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1972" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-80)">-disable</text><text class="breeze-prod-image-build-r4" x="134.2" y="1972" textLength="231. [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1996.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-81)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1996.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-81)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1996.4" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-81)">-disable</text><text class="breeze-prod-image-build-r4" x="134.2" y="1996.4" textLeng [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2020.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-82)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2020.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-82)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2020.8" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-82)">-disable</text><text class="breeze-prod-image-build-r4" x="134.2" y="2020.8" textLeng [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2045.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-83)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2045.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-83)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2045.2" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-83)">-disable</text><text class="breeze-prod-image-build-r4" x="134.2" y="2045.2" textLeng [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2069.6" textLength="1464" clip-path="url(#breeze-prod-image-build-line-84)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-build-r1" x="1464" y="2069.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-84)">
+</text><text class="breeze-prod-image-build-r5" x="0" y="2094" textLength="24.4" clip-path="url(#breeze-prod-image-build-line-85)">╭─</text><text class="breeze-prod-image-build-r5" x="24.4" y="2094" textLength="622.2" clip-path="url(#breeze-prod-image-build-line-85)"> Preparing cache and push (for maintainers and CI) </text><text class="breeze-prod-image-build-r5" x="646.6" y="2094" textLength="793" clip-path="url(#breeze-prod-image-build-line [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2118.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-86)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2118.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-86)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2118.4" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-86)">-builder</text><text class="breeze-prod-image-build-r1" x="341.6" y="2118.4" textLeng [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2142.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-87)">│</text><text class="breeze-prod-image-build-r5" x="341.6" y="2142.8" textLength="756.4" clip-path="url(#breeze-prod-image-build-line-87)">[default: autodetect]                                &# [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2167.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-88)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2167.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-88)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2167.2" textLength="109.8" clip-path="url(#breeze-prod-image-build-line-88)">-platform</text><text class="breeze-prod-image-build-r1" x="341.6" y="2167.2" textLe [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2191.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-89)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2191.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-89)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2191.6" textLength="61" clip-path="url(#breeze-prod-image-build-line-89)">-push</text><text class="breeze-prod-image-build-r1" x="341.6" y="2191.6" textLength="3 [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2216" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-90)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2216" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-90)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2216" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-90)">-prepare</text><text class="breeze-prod-image-build-r4" x="134.2" y="2216" textLength="158. [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2240.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-91)">│</text><text class="breeze-prod-image-build-r1" x="341.6" y="2240.4" textLength="1098" clip-path="url(#breeze-prod-image-build-line-91)">image).                                    [...]
</text><text class="breeze-prod-image-build-r5" x="0" y="2264.8" textLength="1464" clip-path="url(#breeze-prod-image-build-line-92)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-build-r1" x="1464" y="2264.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-92)">
-</text><text class="breeze-prod-image-build-r5" x="0" y="2289.2" textLength="24.4" clip-path="url(#breeze-prod-image-build-line-93)">╭─</text><text class="breeze-prod-image-build-r5" x="24.4" y="2289.2" textLength="195.2" clip-path="url(#breeze-prod-image-build-line-93)"> Common options </text><text class="breeze-prod-image-build-r5" x="219.6" y="2289.2" textLength="1220" clip-path="url(#breeze-prod-image-build-line-93)">──────────────────────────────────────────────────── [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2313.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-94)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2313.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-94)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2313.6" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-94)">-verbose</text><text class="breeze-prod-image-build-r6" x="158.6" y="2313.6" textLeng [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2338" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-95)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2338" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-95)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2338" textLength="48.8" clip-path="url(#breeze-prod-image-build-line-95)">-dry</text><text class="breeze-prod-image-build-r4" x="85.4" y="2338" textLength="48.8" cli [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2362.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-96)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2362.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-96)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2362.4" textLength="61" clip-path="url(#breeze-prod-image-build-line-96)">-help</text><text class="breeze-prod-image-build-r6" x="158.6" y="2362.4" textLength="2 [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="2386.8" textLength="1464" clip-path="url(#breeze-prod-image-build-line-97)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-build-r1" x="1464" y="2386.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-97)">
+</text><text class="breeze-prod-image-build-r5" x="0" y="2289.2" textLength="24.4" clip-path="url(#breeze-prod-image-build-line-93)">╭─</text><text class="breeze-prod-image-build-r5" x="24.4" y="2289.2" textLength="280.6" clip-path="url(#breeze-prod-image-build-line-93)"> Github authentication </text><text class="breeze-prod-image-build-r5" x="305" y="2289.2" textLength="1134.6" clip-path="url(#breeze-prod-image-build-line-93)">───────────────────────────────────────────── [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2313.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-94)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2313.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-94)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2313.6" textLength="85.4" clip-path="url(#breeze-prod-image-build-line-94)">-github</text><text class="breeze-prod-image-build-r4" x="122" y="2313.6" textLength= [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2338" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-95)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2338" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-95)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2338" textLength="85.4" clip-path="url(#breeze-prod-image-build-line-95)">-github</text><text class="breeze-prod-image-build-r4" x="122" y="2338" textLength="73.2" c [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2362.4" textLength="1464" clip-path="url(#breeze-prod-image-build-line-96)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-build-r1" x="1464" y="2362.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-96)">
+</text><text class="breeze-prod-image-build-r5" x="0" y="2386.8" textLength="24.4" clip-path="url(#breeze-prod-image-build-line-97)">╭─</text><text class="breeze-prod-image-build-r5" x="24.4" y="2386.8" textLength="195.2" clip-path="url(#breeze-prod-image-build-line-97)"> Common options </text><text class="breeze-prod-image-build-r5" x="219.6" y="2386.8" textLength="1220" clip-path="url(#breeze-prod-image-build-line-97)">──────────────────────────────────────────────────── [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2411.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-98)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2411.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-98)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2411.2" textLength="97.6" clip-path="url(#breeze-prod-image-build-line-98)">-verbose</text><text class="breeze-prod-image-build-r6" x="158.6" y="2411.2" textLeng [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2435.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-99)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2435.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-99)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2435.6" textLength="48.8" clip-path="url(#breeze-prod-image-build-line-99)">-dry</text><text class="breeze-prod-image-build-r4" x="85.4" y="2435.6" textLength="4 [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2460" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-100)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2460" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-100)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2460" textLength="85.4" clip-path="url(#breeze-prod-image-build-line-100)">-answer</text><text class="breeze-prod-image-build-r6" x="158.6" y="2460" textLength="24 [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2484.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-101)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="2484.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-101)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="2484.4" textLength="61" clip-path="url(#breeze-prod-image-build-line-101)">-help</text><text class="breeze-prod-image-build-r6" x="158.6" y="2484.4" textLength [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="2508.8" textLength="1464" clip-path="url(#breeze-prod-image-build-line-102)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-prod-image-build-r1" x="1464" y="2508.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-102)">
</text>
</g>
</g>
diff --git a/images/breeze/output_prod-image_build.txt b/images/breeze/output_prod-image_build.txt
index aac6b4aab7..f93523394a 100644
--- a/images/breeze/output_prod-image_build.txt
+++ b/images/breeze/output_prod-image_build.txt
@@ -1 +1 @@
-1eed0951812bd758082ffe34daed932b
+5ae644dee2b69e91dbcd593cdeebd3e7
diff --git a/images/breeze/output_release-management_prepare-airflow-package.txt b/images/breeze/output_release-management_prepare-airflow-package.txt
index 644026b1cf..bb4e71586a 100644
--- a/images/breeze/output_release-management_prepare-airflow-package.txt
+++ b/images/breeze/output_release-management_prepare-airflow-package.txt
@@ -1 +1 @@
-5077bcfc4272c969f8b80ea4438a438a
+91b0782d87b4e0ed7ce6e8cb231adafd
diff --git a/images/breeze/output_release-management_prepare-provider-packages.txt b/images/breeze/output_release-management_prepare-provider-packages.txt
index 872a3881ba..cf50eb5d41 100644
--- a/images/breeze/output_release-management_prepare-provider-packages.txt
+++ b/images/breeze/output_release-management_prepare-provider-packages.txt
@@ -1 +1 @@
-1b7d25cb5f379cee89c5345c60be1992
+2bb361f519c8165e884439722e76e945
(airflow) 03/37: Remove backcompat with Airflow 2.3/2.4 in providers (#35727)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 619855884356abbd2e6d8a4f7956e4a0510b8c9e
Author: Andrey Anshin <An...@taragol.is>
AuthorDate: Mon Nov 20 18:11:54 2023 +0400
Remove backcompat with Airflow 2.3/2.4 in providers (#35727)
* Remove backcompat with Airflow 2.3/2.4 in providers
* Revert changes in sql.py
---
airflow/providers/google/cloud/hooks/gcs.py | 9 +--------
airflow/providers/google/cloud/secrets/secret_manager.py | 14 ++++++--------
airflow/providers/microsoft/azure/secrets/key_vault.py | 14 ++++++--------
3 files changed, 13 insertions(+), 24 deletions(-)
diff --git a/airflow/providers/google/cloud/hooks/gcs.py b/airflow/providers/google/cloud/hooks/gcs.py
index d8bb36037f..1a74d09d55 100644
--- a/airflow/providers/google/cloud/hooks/gcs.py
+++ b/airflow/providers/google/cloud/hooks/gcs.py
@@ -45,6 +45,7 @@ from airflow.exceptions import AirflowException, AirflowProviderDeprecationWarni
from airflow.providers.google.cloud.utils.helpers import normalize_directory_path
from airflow.providers.google.common.consts import CLIENT_INFO
from airflow.providers.google.common.hooks.base_google import GoogleBaseAsyncHook, GoogleBaseHook
+from airflow.typing_compat import ParamSpec
from airflow.utils import timezone
from airflow.version import version
@@ -54,14 +55,6 @@ if TYPE_CHECKING:
from aiohttp import ClientSession
from google.api_core.retry import Retry
-try:
- # Airflow 2.3 doesn't have this yet
- from airflow.typing_compat import ParamSpec
-except ImportError:
- try:
- from typing import ParamSpec # type: ignore[no-redef, attr-defined]
- except ImportError:
- from typing_extensions import ParamSpec
RT = TypeVar("RT")
T = TypeVar("T", bound=Callable)
diff --git a/airflow/providers/google/cloud/secrets/secret_manager.py b/airflow/providers/google/cloud/secrets/secret_manager.py
index fd8b8e33e2..a40c6bfbe5 100644
--- a/airflow/providers/google/cloud/secrets/secret_manager.py
+++ b/airflow/providers/google/cloud/secrets/secret_manager.py
@@ -28,7 +28,6 @@ from airflow.providers.google.cloud._internal_client.secret_manager_client impor
from airflow.providers.google.cloud.utils.credentials_provider import get_credentials_and_project_id
from airflow.secrets import BaseSecretsBackend
from airflow.utils.log.logging_mixin import LoggingMixin
-from airflow.version import version as airflow_version
log = logging.getLogger(__name__)
@@ -154,13 +153,12 @@ class CloudSecretManagerBackend(BaseSecretsBackend, LoggingMixin):
:param conn_id: the connection id
:return: deserialized Connection
"""
- if _parse_version(airflow_version) >= (2, 3):
- warnings.warn(
- f"Method `{self.__class__.__name__}.get_conn_uri` is deprecated and will be removed "
- "in a future release. Please use method `get_conn_value` instead.",
- AirflowProviderDeprecationWarning,
- stacklevel=2,
- )
+ warnings.warn(
+ f"Method `{self.__class__.__name__}.get_conn_uri` is deprecated and will be removed "
+ "in a future release. Please use method `get_conn_value` instead.",
+ AirflowProviderDeprecationWarning,
+ stacklevel=2,
+ )
return self.get_conn_value(conn_id)
def get_variable(self, key: str) -> str | None:
diff --git a/airflow/providers/microsoft/azure/secrets/key_vault.py b/airflow/providers/microsoft/azure/secrets/key_vault.py
index 794788206c..bfa9117b11 100644
--- a/airflow/providers/microsoft/azure/secrets/key_vault.py
+++ b/airflow/providers/microsoft/azure/secrets/key_vault.py
@@ -38,7 +38,6 @@ from airflow.exceptions import AirflowProviderDeprecationWarning
from airflow.providers.microsoft.azure.utils import get_sync_default_azure_credential
from airflow.secrets import BaseSecretsBackend
from airflow.utils.log.logging_mixin import LoggingMixin
-from airflow.version import version as airflow_version
def _parse_version(val):
@@ -170,13 +169,12 @@ class AzureKeyVaultBackend(BaseSecretsBackend, LoggingMixin):
:param conn_id: the connection id
:return: deserialized Connection
"""
- if _parse_version(airflow_version) >= (2, 3):
- warnings.warn(
- f"Method `{self.__class__.__name__}.get_conn_uri` is deprecated and will be removed "
- "in a future release. Please use method `get_conn_value` instead.",
- AirflowProviderDeprecationWarning,
- stacklevel=2,
- )
+ warnings.warn(
+ f"Method `{self.__class__.__name__}.get_conn_uri` is deprecated and will be removed "
+ "in a future release. Please use method `get_conn_value` instead.",
+ AirflowProviderDeprecationWarning,
+ stacklevel=2,
+ )
return self.get_conn_value(conn_id)
def get_variable(self, key: str) -> str | None:
(airflow) 35/37: Add borderWidthRight to grid for Firefox scrollbar (#35346)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 55914e14cbb587e6712c027e039929bb5bb105c0
Author: Victor Chiapaikeo <vc...@gmail.com>
AuthorDate: Wed Nov 22 11:58:10 2023 -0500
Add borderWidthRight to grid for Firefox scrollbar (#35346)
---
airflow/www/static/js/dag/grid/index.tsx | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/airflow/www/static/js/dag/grid/index.tsx b/airflow/www/static/js/dag/grid/index.tsx
index 638f4268c0..dc73aecee3 100644
--- a/airflow/www/static/js/dag/grid/index.tsx
+++ b/airflow/www/static/js/dag/grid/index.tsx
@@ -169,7 +169,7 @@ const Grid = ({
mt={8}
overscrollBehavior="contain"
>
- <Table pr="10px">
+ <Table pr="10px" borderRightWidth="14px" borderColor="transparent">
<Thead>
<DagRuns
groups={groups}
(airflow) 18/37: Add OpenLineage support to GCSToBigQueryOperator (#35778)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit b51aaf59d2280e4341807ed39e3a110a26627426
Author: Kacper Muda <mu...@gmail.com>
AuthorDate: Tue Nov 21 18:14:36 2023 +0100
Add OpenLineage support to GCSToBigQueryOperator (#35778)
---
.../google/cloud/transfers/gcs_to_bigquery.py | 77 +++++
.../google/cloud/transfers/test_gcs_to_bigquery.py | 380 ++++++++++++++++++++-
2 files changed, 455 insertions(+), 2 deletions(-)
diff --git a/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py b/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py
index 94c233d6c5..798bc8a52b 100644
--- a/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py
+++ b/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py
@@ -41,6 +41,7 @@ from airflow.providers.google.cloud.hooks.bigquery import BigQueryHook, BigQuery
from airflow.providers.google.cloud.hooks.gcs import GCSHook
from airflow.providers.google.cloud.links.bigquery import BigQueryTableLink
from airflow.providers.google.cloud.triggers.bigquery import BigQueryInsertJobTrigger
+from airflow.utils.helpers import merge_dicts
if TYPE_CHECKING:
from google.api_core.retry import Retry
@@ -294,6 +295,8 @@ class GCSToBigQueryOperator(BaseOperator):
self.reattach_states: set[str] = reattach_states or set()
self.cancel_on_kill = cancel_on_kill
+ self.source_uris: list[str] = []
+
def _submit_job(
self,
hook: BigQueryHook,
@@ -731,3 +734,77 @@ class GCSToBigQueryOperator(BaseOperator):
self.hook.cancel_job(job_id=self.job_id, location=self.location) # type: ignore[union-attr]
else:
self.log.info("Skipping to cancel job: %s.%s", self.location, self.job_id)
+
+ def get_openlineage_facets_on_complete(self, task_instance):
+ """Implementing on_complete as we will include final BQ job id."""
+ from pathlib import Path
+
+ from openlineage.client.facet import (
+ ExternalQueryRunFacet,
+ SymlinksDatasetFacet,
+ SymlinksDatasetFacetIdentifiers,
+ )
+ from openlineage.client.run import Dataset
+
+ from airflow.providers.google.cloud.hooks.gcs import _parse_gcs_url
+ from airflow.providers.google.cloud.utils.openlineage import (
+ get_facets_from_bq_table,
+ get_identity_column_lineage_facet,
+ )
+ from airflow.providers.openlineage.extractors import OperatorLineage
+
+ table_object = self.hook.get_client(self.hook.project_id).get_table(
+ self.destination_project_dataset_table
+ )
+
+ output_dataset_facets = get_facets_from_bq_table(table_object)
+
+ input_dataset_facets = {
+ "schema": output_dataset_facets["schema"],
+ }
+ input_datasets = []
+ for uri in sorted(self.source_uris):
+ bucket, blob = _parse_gcs_url(uri)
+ additional_facets = {}
+
+ if "*" in blob:
+ # If wildcard ("*") is used in gcs path, we want the name of dataset to be directory name,
+ # but we create a symlink to the full object path with wildcard.
+ additional_facets = {
+ "symlink": SymlinksDatasetFacet(
+ identifiers=[
+ SymlinksDatasetFacetIdentifiers(
+ namespace=f"gs://{bucket}", name=blob, type="file"
+ )
+ ]
+ ),
+ }
+ blob = Path(blob).parent.as_posix()
+ if blob == ".":
+ # blob path does not have leading slash, but we need root dataset name to be "/"
+ blob = "/"
+
+ dataset = Dataset(
+ namespace=f"gs://{bucket}",
+ name=blob,
+ facets=merge_dicts(input_dataset_facets, additional_facets),
+ )
+ input_datasets.append(dataset)
+
+ output_dataset_facets["columnLineage"] = get_identity_column_lineage_facet(
+ field_names=[field.name for field in table_object.schema], input_datasets=input_datasets
+ )
+
+ output_dataset = Dataset(
+ namespace="bigquery",
+ name=str(table_object.reference),
+ facets=output_dataset_facets,
+ )
+
+ run_facets = {}
+ if self.job_id:
+ run_facets = {
+ "externalQuery": ExternalQueryRunFacet(externalQueryId=self.job_id, source="bigquery"),
+ }
+
+ return OperatorLineage(inputs=input_datasets, outputs=[output_dataset], run_facets=run_facets)
diff --git a/tests/providers/google/cloud/transfers/test_gcs_to_bigquery.py b/tests/providers/google/cloud/transfers/test_gcs_to_bigquery.py
index a67759ff85..b7087b6785 100644
--- a/tests/providers/google/cloud/transfers/test_gcs_to_bigquery.py
+++ b/tests/providers/google/cloud/transfers/test_gcs_to_bigquery.py
@@ -22,8 +22,20 @@ from unittest import mock
from unittest.mock import MagicMock, call
import pytest
-from google.cloud.bigquery import DEFAULT_RETRY
+from google.cloud.bigquery import DEFAULT_RETRY, Table
from google.cloud.exceptions import Conflict
+from openlineage.client.facet import (
+ ColumnLineageDatasetFacet,
+ ColumnLineageDatasetFacetFieldsAdditional,
+ ColumnLineageDatasetFacetFieldsAdditionalInputFields,
+ DocumentationDatasetFacet,
+ ExternalQueryRunFacet,
+ SchemaDatasetFacet,
+ SchemaField,
+ SymlinksDatasetFacet,
+ SymlinksDatasetFacetIdentifiers,
+)
+from openlineage.client.run import Dataset
from airflow.exceptions import AirflowException, TaskDeferred
from airflow.models import DAG
@@ -37,6 +49,9 @@ from airflow.utils.types import DagRunType
TASK_ID = "test-gcs-to-bq-operator"
TEST_EXPLICIT_DEST = "test-project.dataset.table"
TEST_BUCKET = "test-bucket"
+TEST_FOLDER = "test-folder"
+TEST_OBJECT_NO_WILDCARD = "file.extension"
+TEST_OBJECT_WILDCARD = "file_*.extension"
PROJECT_ID = "test-project"
DATASET = "dataset"
TABLE = "table"
@@ -59,7 +74,21 @@ TEST_SOURCE_OBJECTS = "test/objects/test.csv"
TEST_SOURCE_OBJECTS_JSON = "test/objects/test.json"
LABELS = {"k1": "v1"}
DESCRIPTION = "Test Description"
-
+TEST_TABLE: Table = Table.from_api_repr(
+ {
+ "tableReference": {"projectId": PROJECT_ID, "datasetId": DATASET, "tableId": TABLE},
+ "description": DESCRIPTION,
+ "schema": {
+ "fields": [
+ {"name": "field1", "type": "STRING", "description": "field1 description"},
+ {"name": "field2", "type": "INTEGER"},
+ ]
+ },
+ }
+)
+TEST_EMPTY_TABLE: Table = Table.from_api_repr(
+ {"tableReference": {"projectId": PROJECT_ID, "datasetId": DATASET, "tableId": TABLE}}
+)
job_id = "123456"
hash_ = "hash"
pytest.real_job_id = f"{job_id}_{hash_}"
@@ -1209,6 +1238,353 @@ class TestGCSToBigQueryOperator:
},
)
+ @pytest.mark.parametrize(
+ ("source_object", "expected_dataset_name"),
+ (
+ (f"{TEST_FOLDER}/{TEST_OBJECT_NO_WILDCARD}", f"{TEST_FOLDER}/{TEST_OBJECT_NO_WILDCARD}"),
+ (TEST_OBJECT_NO_WILDCARD, TEST_OBJECT_NO_WILDCARD),
+ (f"{TEST_FOLDER}/{TEST_OBJECT_WILDCARD}", TEST_FOLDER),
+ (f"{TEST_OBJECT_WILDCARD}", "/"),
+ (f"{TEST_FOLDER}/*", TEST_FOLDER),
+ ),
+ )
+ @mock.patch(GCS_TO_BQ_PATH.format("BigQueryHook"))
+ def test_get_openlineage_facets_on_complete_gcs_dataset_name(
+ self, hook, source_object, expected_dataset_name
+ ):
+ hook.return_value.insert_job.return_value = MagicMock(job_id=pytest.real_job_id, error_result=False)
+ hook.return_value.split_tablename.return_value = (PROJECT_ID, DATASET, TABLE)
+ operator = GCSToBigQueryOperator(
+ project_id=JOB_PROJECT_ID,
+ task_id=TASK_ID,
+ bucket=TEST_BUCKET,
+ source_objects=[source_object],
+ destination_project_dataset_table=TEST_EXPLICIT_DEST,
+ )
+
+ expected_symlink = SymlinksDatasetFacet(
+ identifiers=[
+ SymlinksDatasetFacetIdentifiers(
+ namespace=f"gs://{TEST_BUCKET}",
+ name=source_object,
+ type="file",
+ )
+ ]
+ )
+ operator.execute(context=mock.MagicMock())
+
+ lineage = operator.get_openlineage_facets_on_complete(None)
+ assert len(lineage.inputs) == 1
+ assert lineage.inputs[0].name == expected_dataset_name
+ if "*" in source_object:
+ assert lineage.inputs[0].facets.get("symlink")
+ assert lineage.inputs[0].facets.get("symlink") == expected_symlink
+
+ @mock.patch(GCS_TO_BQ_PATH.format("BigQueryHook"))
+ def test_get_openlineage_facets_on_complete_gcs_multiple_uris(self, hook):
+ hook.return_value.insert_job.return_value = MagicMock(job_id=pytest.real_job_id, error_result=False)
+ hook.return_value.split_tablename.return_value = (PROJECT_ID, DATASET, TABLE)
+ operator = GCSToBigQueryOperator(
+ project_id=JOB_PROJECT_ID,
+ task_id=TASK_ID,
+ bucket=TEST_BUCKET,
+ source_objects=[
+ TEST_OBJECT_NO_WILDCARD,
+ TEST_OBJECT_WILDCARD,
+ f"{TEST_FOLDER}1/{TEST_OBJECT_NO_WILDCARD}",
+ f"{TEST_FOLDER}2/{TEST_OBJECT_WILDCARD}",
+ ],
+ destination_project_dataset_table=TEST_EXPLICIT_DEST,
+ )
+
+ operator.execute(context=mock.MagicMock())
+
+ lineage = operator.get_openlineage_facets_on_complete(None)
+ assert len(lineage.inputs) == 4
+ assert lineage.inputs[0].name == TEST_OBJECT_NO_WILDCARD
+ assert lineage.inputs[1].name == "/"
+ assert lineage.inputs[1].facets.get("symlink") == SymlinksDatasetFacet(
+ identifiers=[
+ SymlinksDatasetFacetIdentifiers(
+ namespace=f"gs://{TEST_BUCKET}",
+ name=TEST_OBJECT_WILDCARD,
+ type="file",
+ )
+ ]
+ )
+ assert lineage.inputs[2].name == f"{TEST_FOLDER}1/{TEST_OBJECT_NO_WILDCARD}"
+ assert lineage.inputs[3].name == f"{TEST_FOLDER}2"
+ assert lineage.inputs[3].facets.get("symlink") == SymlinksDatasetFacet(
+ identifiers=[
+ SymlinksDatasetFacetIdentifiers(
+ namespace=f"gs://{TEST_BUCKET}",
+ name=f"{TEST_FOLDER}2/{TEST_OBJECT_WILDCARD}",
+ type="file",
+ )
+ ]
+ )
+
+ @mock.patch(GCS_TO_BQ_PATH.format("BigQueryHook"))
+ def test_get_openlineage_facets_on_complete_bq_dataset(self, hook):
+ hook.return_value.insert_job.return_value = MagicMock(job_id=pytest.real_job_id, error_result=False)
+ hook.return_value.split_tablename.return_value = (PROJECT_ID, DATASET, TABLE)
+ hook.return_value.get_client.return_value.get_table.return_value = TEST_TABLE
+
+ expected_output_dataset_facets = {
+ "schema": SchemaDatasetFacet(
+ fields=[
+ SchemaField(name="field1", type="STRING", description="field1 description"),
+ SchemaField(name="field2", type="INTEGER"),
+ ]
+ ),
+ "documentation": DocumentationDatasetFacet(description="Test Description"),
+ "columnLineage": ColumnLineageDatasetFacet(
+ fields={
+ "field1": ColumnLineageDatasetFacetFieldsAdditional(
+ inputFields=[
+ ColumnLineageDatasetFacetFieldsAdditionalInputFields(
+ namespace=f"gs://{TEST_BUCKET}", name=TEST_OBJECT_NO_WILDCARD, field="field1"
+ )
+ ],
+ transformationType="IDENTITY",
+ transformationDescription="identical",
+ ),
+ "field2": ColumnLineageDatasetFacetFieldsAdditional(
+ inputFields=[
+ ColumnLineageDatasetFacetFieldsAdditionalInputFields(
+ namespace=f"gs://{TEST_BUCKET}", name=TEST_OBJECT_NO_WILDCARD, field="field2"
+ )
+ ],
+ transformationType="IDENTITY",
+ transformationDescription="identical",
+ ),
+ }
+ ),
+ }
+
+ operator = GCSToBigQueryOperator(
+ project_id=JOB_PROJECT_ID,
+ task_id=TASK_ID,
+ bucket=TEST_BUCKET,
+ source_objects=[TEST_OBJECT_NO_WILDCARD],
+ destination_project_dataset_table=TEST_EXPLICIT_DEST,
+ )
+
+ operator.execute(context=mock.MagicMock())
+
+ lineage = operator.get_openlineage_facets_on_complete(None)
+ assert len(lineage.outputs) == 1
+ assert lineage.outputs[0] == Dataset(
+ namespace="bigquery",
+ name=TEST_EXPLICIT_DEST,
+ facets=expected_output_dataset_facets,
+ )
+
+ @mock.patch(GCS_TO_BQ_PATH.format("BigQueryHook"))
+ def test_get_openlineage_facets_on_complete_bq_dataset_multiple_gcs_uris(self, hook):
+ hook.return_value.insert_job.return_value = MagicMock(job_id=pytest.real_job_id, error_result=False)
+ hook.return_value.split_tablename.return_value = (PROJECT_ID, DATASET, TABLE)
+ hook.return_value.get_client.return_value.get_table.return_value = TEST_TABLE
+
+ expected_output_dataset_facets = {
+ "schema": SchemaDatasetFacet(
+ fields=[
+ SchemaField(name="field1", type="STRING", description="field1 description"),
+ SchemaField(name="field2", type="INTEGER"),
+ ]
+ ),
+ "documentation": DocumentationDatasetFacet(description="Test Description"),
+ "columnLineage": ColumnLineageDatasetFacet(
+ fields={
+ "field1": ColumnLineageDatasetFacetFieldsAdditional(
+ inputFields=[
+ ColumnLineageDatasetFacetFieldsAdditionalInputFields(
+ namespace=f"gs://{TEST_BUCKET}", name=TEST_OBJECT_NO_WILDCARD, field="field1"
+ ),
+ ColumnLineageDatasetFacetFieldsAdditionalInputFields(
+ namespace=f"gs://{TEST_BUCKET}", name="/", field="field1"
+ ),
+ ],
+ transformationType="IDENTITY",
+ transformationDescription="identical",
+ ),
+ "field2": ColumnLineageDatasetFacetFieldsAdditional(
+ inputFields=[
+ ColumnLineageDatasetFacetFieldsAdditionalInputFields(
+ namespace=f"gs://{TEST_BUCKET}", name=TEST_OBJECT_NO_WILDCARD, field="field2"
+ ),
+ ColumnLineageDatasetFacetFieldsAdditionalInputFields(
+ namespace=f"gs://{TEST_BUCKET}", name="/", field="field2"
+ ),
+ ],
+ transformationType="IDENTITY",
+ transformationDescription="identical",
+ ),
+ }
+ ),
+ }
+
+ operator = GCSToBigQueryOperator(
+ project_id=JOB_PROJECT_ID,
+ task_id=TASK_ID,
+ bucket=TEST_BUCKET,
+ source_objects=[TEST_OBJECT_NO_WILDCARD, TEST_OBJECT_WILDCARD],
+ destination_project_dataset_table=TEST_EXPLICIT_DEST,
+ )
+
+ operator.execute(context=mock.MagicMock())
+
+ lineage = operator.get_openlineage_facets_on_complete(None)
+ assert len(lineage.outputs) == 1
+ assert lineage.outputs[0] == Dataset(
+ namespace="bigquery",
+ name=TEST_EXPLICIT_DEST,
+ facets=expected_output_dataset_facets,
+ )
+
+ @mock.patch(GCS_TO_BQ_PATH.format("BigQueryHook"))
+ def test_get_openlineage_facets_on_complete_empty_table(self, hook):
+ hook.return_value.insert_job.return_value = MagicMock(job_id=pytest.real_job_id, error_result=False)
+ hook.return_value.split_tablename.return_value = (PROJECT_ID, DATASET, TABLE)
+ hook.return_value.get_client.return_value.get_table.return_value = TEST_EMPTY_TABLE
+
+ expected_output_dataset_facets = {
+ "schema": SchemaDatasetFacet(fields=[]),
+ "documentation": DocumentationDatasetFacet(description=""),
+ "columnLineage": ColumnLineageDatasetFacet(fields={}),
+ }
+
+ operator = GCSToBigQueryOperator(
+ project_id=JOB_PROJECT_ID,
+ task_id=TASK_ID,
+ bucket=TEST_BUCKET,
+ source_objects=[TEST_OBJECT_NO_WILDCARD, TEST_OBJECT_WILDCARD],
+ destination_project_dataset_table=TEST_EXPLICIT_DEST,
+ )
+
+ operator.execute(context=mock.MagicMock())
+
+ lineage = operator.get_openlineage_facets_on_complete(None)
+ assert len(lineage.inputs) == 2
+ assert len(lineage.outputs) == 1
+ assert lineage.outputs[0] == Dataset(
+ namespace="bigquery",
+ name=TEST_EXPLICIT_DEST,
+ facets=expected_output_dataset_facets,
+ )
+ assert lineage.inputs[0] == Dataset(
+ namespace=f"gs://{TEST_BUCKET}",
+ name=TEST_OBJECT_NO_WILDCARD,
+ facets={"schema": SchemaDatasetFacet(fields=[])},
+ )
+ assert lineage.inputs[1] == Dataset(
+ namespace=f"gs://{TEST_BUCKET}",
+ name="/",
+ facets={
+ "schema": SchemaDatasetFacet(fields=[]),
+ "symlink": SymlinksDatasetFacet(
+ identifiers=[
+ SymlinksDatasetFacetIdentifiers(
+ namespace=f"gs://{TEST_BUCKET}",
+ name=TEST_OBJECT_WILDCARD,
+ type="file",
+ )
+ ]
+ ),
+ },
+ )
+
+ @mock.patch(GCS_TO_BQ_PATH.format("BigQueryHook"))
+ def test_get_openlineage_facets_on_complete_full_table_multiple_gcs_uris(self, hook):
+ hook.return_value.insert_job.return_value = MagicMock(job_id=pytest.real_job_id, error_result=False)
+ hook.return_value.split_tablename.return_value = (PROJECT_ID, DATASET, TABLE)
+ hook.return_value.get_client.return_value.get_table.return_value = TEST_TABLE
+ hook.return_value.generate_job_id.return_value = pytest.real_job_id
+
+ schema_facet = SchemaDatasetFacet(
+ fields=[
+ SchemaField(name="field1", type="STRING", description="field1 description"),
+ SchemaField(name="field2", type="INTEGER"),
+ ]
+ )
+
+ expected_input_wildcard_dataset_facets = {
+ "schema": schema_facet,
+ "symlink": SymlinksDatasetFacet(
+ identifiers=[
+ SymlinksDatasetFacetIdentifiers(
+ namespace=f"gs://{TEST_BUCKET}",
+ name=TEST_OBJECT_WILDCARD,
+ type="file",
+ )
+ ]
+ ),
+ }
+ expected_input_no_wildcard_dataset_facets = {"schema": schema_facet}
+
+ expected_output_dataset_facets = {
+ "schema": schema_facet,
+ "documentation": DocumentationDatasetFacet(description="Test Description"),
+ "columnLineage": ColumnLineageDatasetFacet(
+ fields={
+ "field1": ColumnLineageDatasetFacetFieldsAdditional(
+ inputFields=[
+ ColumnLineageDatasetFacetFieldsAdditionalInputFields(
+ namespace=f"gs://{TEST_BUCKET}", name=TEST_OBJECT_NO_WILDCARD, field="field1"
+ ),
+ ColumnLineageDatasetFacetFieldsAdditionalInputFields(
+ namespace=f"gs://{TEST_BUCKET}", name="/", field="field1"
+ ),
+ ],
+ transformationType="IDENTITY",
+ transformationDescription="identical",
+ ),
+ "field2": ColumnLineageDatasetFacetFieldsAdditional(
+ inputFields=[
+ ColumnLineageDatasetFacetFieldsAdditionalInputFields(
+ namespace=f"gs://{TEST_BUCKET}", name=TEST_OBJECT_NO_WILDCARD, field="field2"
+ ),
+ ColumnLineageDatasetFacetFieldsAdditionalInputFields(
+ namespace=f"gs://{TEST_BUCKET}", name="/", field="field2"
+ ),
+ ],
+ transformationType="IDENTITY",
+ transformationDescription="identical",
+ ),
+ }
+ ),
+ }
+
+ operator = GCSToBigQueryOperator(
+ project_id=JOB_PROJECT_ID,
+ task_id=TASK_ID,
+ bucket=TEST_BUCKET,
+ source_objects=[TEST_OBJECT_NO_WILDCARD, TEST_OBJECT_WILDCARD],
+ destination_project_dataset_table=TEST_EXPLICIT_DEST,
+ )
+
+ operator.execute(context=mock.MagicMock())
+
+ lineage = operator.get_openlineage_facets_on_complete(None)
+ assert len(lineage.inputs) == 2
+ assert len(lineage.outputs) == 1
+ assert lineage.outputs[0] == Dataset(
+ namespace="bigquery", name=TEST_EXPLICIT_DEST, facets=expected_output_dataset_facets
+ )
+
+ assert lineage.inputs[0] == Dataset(
+ namespace=f"gs://{TEST_BUCKET}",
+ name=TEST_OBJECT_NO_WILDCARD,
+ facets=expected_input_no_wildcard_dataset_facets,
+ )
+ assert lineage.inputs[1] == Dataset(
+ namespace=f"gs://{TEST_BUCKET}", name="/", facets=expected_input_wildcard_dataset_facets
+ )
+ assert lineage.run_facets == {
+ "externalQuery": ExternalQueryRunFacet(externalQueryId=pytest.real_job_id, source="bigquery")
+ }
+ assert lineage.job_facets == {}
+
class TestAsyncGCSToBigQueryOperator:
@mock.patch(GCS_TO_BQ_PATH.format("BigQueryHook"))
(airflow) 12/37: Remove backcompat inheritance for DbApiHook (#35754)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 73106932771ba3a1e8a08de3905b569d5462c416
Author: Andrey Anshin <An...@taragol.is>
AuthorDate: Tue Nov 21 11:12:56 2023 +0400
Remove backcompat inheritance for DbApiHook (#35754)
* Remove backcompat inheritance for DbApiHook
* jwt_file > jwt__file
* simplify trino test
---
airflow/providers/apache/impala/hooks/impala.py | 3 +-
airflow/providers/common/sql/hooks/sql.py | 18 +---------
airflow/providers/common/sql/hooks/sql.pyi | 4 +--
.../providers/elasticsearch/hooks/elasticsearch.py | 4 +--
airflow/providers/trino/hooks/trino.py | 19 ++++++++---
tests/providers/odbc/hooks/test_odbc.py | 3 +-
tests/providers/trino/hooks/test_trino.py | 38 +++++++++++++++-------
7 files changed, 49 insertions(+), 40 deletions(-)
diff --git a/airflow/providers/apache/impala/hooks/impala.py b/airflow/providers/apache/impala/hooks/impala.py
index ab19865a9e..b8c79b4e25 100644
--- a/airflow/providers/apache/impala/hooks/impala.py
+++ b/airflow/providers/apache/impala/hooks/impala.py
@@ -35,7 +35,8 @@ class ImpalaHook(DbApiHook):
hook_name = "Impala"
def get_conn(self) -> Connection:
- connection = self.get_connection(self.impala_conn_id) # pylint: disable=no-member
+ conn_id: str = getattr(self, self.conn_name_attr)
+ connection = self.get_connection(conn_id)
return connect(
host=connection.host,
port=connection.port,
diff --git a/airflow/providers/common/sql/hooks/sql.py b/airflow/providers/common/sql/hooks/sql.py
index ab4eda5d8e..bb85dedc1c 100644
--- a/airflow/providers/common/sql/hooks/sql.py
+++ b/airflow/providers/common/sql/hooks/sql.py
@@ -34,12 +34,10 @@ from typing import (
from urllib.parse import urlparse
import sqlparse
-from packaging.version import Version
from sqlalchemy import create_engine
from airflow.exceptions import AirflowException
from airflow.hooks.base import BaseHook
-from airflow.version import version
if TYPE_CHECKING:
from pandas import DataFrame
@@ -120,21 +118,7 @@ class ConnectorProtocol(Protocol):
"""
-# In case we are running it on Airflow 2.4+, we should use BaseHook, but on Airflow 2.3 and below
-# We want the DbApiHook to derive from the original DbApiHook from airflow, because otherwise
-# SqlSensor and BaseSqlOperator from "airflow.operators" and "airflow.sensors" will refuse to
-# accept the new Hooks as not derived from the original DbApiHook
-if Version(version) < Version("2.4"):
- try:
- from airflow.hooks.dbapi import DbApiHook as BaseForDbApiHook
- except ImportError:
- # just in case we have a problem with circular import
- BaseForDbApiHook: type[BaseHook] = BaseHook # type: ignore[no-redef]
-else:
- BaseForDbApiHook: type[BaseHook] = BaseHook # type: ignore[no-redef]
-
-
-class DbApiHook(BaseForDbApiHook):
+class DbApiHook(BaseHook):
"""
Abstract base class for sql hooks.
diff --git a/airflow/providers/common/sql/hooks/sql.pyi b/airflow/providers/common/sql/hooks/sql.pyi
index dedac037df..41bd6ebf47 100644
--- a/airflow/providers/common/sql/hooks/sql.pyi
+++ b/airflow/providers/common/sql/hooks/sql.pyi
@@ -32,8 +32,8 @@ Definition of the public interface for airflow.providers.common.sql.hooks.sql
isort:skip_file
"""
from _typeshed import Incomplete
-from airflow.hooks.dbapi import DbApiHook as BaseForDbApiHook
-from typing import Any, Callable, Iterable, Mapping, Sequence
+from airflow.hooks.base import BaseHook as BaseForDbApiHook
+from typing import Any, Callable, Iterable, Mapping, Sequence, Union
from typing_extensions import Protocol
def return_single_query_results(
diff --git a/airflow/providers/elasticsearch/hooks/elasticsearch.py b/airflow/providers/elasticsearch/hooks/elasticsearch.py
index 6c93586892..2d9fca4a97 100644
--- a/airflow/providers/elasticsearch/hooks/elasticsearch.py
+++ b/airflow/providers/elasticsearch/hooks/elasticsearch.py
@@ -108,9 +108,7 @@ class ElasticsearchSQLHook(DbApiHook):
if conn.extra_dejson.get("timeout", False):
conn_args["timeout"] = conn.extra_dejson["timeout"]
- conn = connect(**conn_args)
-
- return conn
+ return connect(**conn_args)
def get_uri(self) -> str:
conn_id = getattr(self, self.conn_name_attr)
diff --git a/airflow/providers/trino/hooks/trino.py b/airflow/providers/trino/hooks/trino.py
index 03195fe452..798109dc3f 100644
--- a/airflow/providers/trino/hooks/trino.py
+++ b/airflow/providers/trino/hooks/trino.py
@@ -19,6 +19,7 @@ from __future__ import annotations
import json
import os
+from pathlib import Path
from typing import TYPE_CHECKING, Any, Iterable, Mapping, TypeVar
import trino
@@ -28,6 +29,7 @@ from trino.transaction import IsolationLevel
from airflow.configuration import conf
from airflow.exceptions import AirflowException
from airflow.providers.common.sql.hooks.sql import DbApiHook
+from airflow.utils.helpers import exactly_one
from airflow.utils.operator_helpers import AIRFLOW_VAR_NAME_FORMAT_MAPPING, DEFAULT_FORMAT_PREFIX
if TYPE_CHECKING:
@@ -99,11 +101,20 @@ class TrinoHook(DbApiHook):
elif db.password:
auth = trino.auth.BasicAuthentication(db.login, db.password) # type: ignore[attr-defined]
elif extra.get("auth") == "jwt":
- if "jwt__file" in extra:
- with open(extra.get("jwt__file")) as jwt_file:
- token = jwt_file.read()
+ if not exactly_one(jwt_file := "jwt__file" in extra, jwt_token := "jwt__token" in extra):
+ msg = (
+ "When auth set to 'jwt' then expected exactly one parameter 'jwt__file' or 'jwt__token'"
+ " in connection extra, but "
+ )
+ if jwt_file and jwt_token:
+ msg += "provided both."
+ else:
+ msg += "none of them provided."
+ raise ValueError(msg)
+ elif jwt_file:
+ token = Path(extra["jwt__file"]).read_text()
else:
- token = extra.get("jwt__token")
+ token = extra["jwt__token"]
auth = trino.auth.JWTAuthentication(token=token)
elif extra.get("auth") == "certs":
auth = trino.auth.CertificateAuthentication(
diff --git a/tests/providers/odbc/hooks/test_odbc.py b/tests/providers/odbc/hooks/test_odbc.py
index ad763b934b..03e09a8adf 100644
--- a/tests/providers/odbc/hooks/test_odbc.py
+++ b/tests/providers/odbc/hooks/test_odbc.py
@@ -79,7 +79,8 @@ class TestOdbcHook:
class UnitTestOdbcHook(OdbcHook):
conn_name_attr = "test_conn_id"
- def get_connection(self, conn_id: str):
+ @classmethod
+ def get_connection(cls, conn_id: str):
return connection
def get_conn(self):
diff --git a/tests/providers/trino/hooks/test_trino.py b/tests/providers/trino/hooks/test_trino.py
index 8aeb4cbe08..5d61c056bc 100644
--- a/tests/providers/trino/hooks/test_trino.py
+++ b/tests/providers/trino/hooks/test_trino.py
@@ -18,9 +18,7 @@
from __future__ import annotations
import json
-import os
import re
-from tempfile import TemporaryDirectory
from unittest import mock
from unittest.mock import patch
@@ -40,16 +38,10 @@ CERT_AUTHENTICATION = "airflow.providers.trino.hooks.trino.trino.auth.Certificat
@pytest.fixture()
-def jwt_token_file():
- # Couldn't get this working with TemporaryFile, using TemporaryDirectory instead
- # Save a phony jwt to a temporary file for the trino hook to read from
- with TemporaryDirectory() as tmp_dir:
- tmp_jwt_file = os.path.join(tmp_dir, "jwt.json")
-
- with open(tmp_jwt_file, "w") as tmp_file:
- tmp_file.write('{"phony":"jwt"}')
-
- yield tmp_jwt_file
+def jwt_token_file(tmp_path):
+ jwt_file = tmp_path / "jwt.json"
+ jwt_file.write_text('{"phony":"jwt"}')
+ yield jwt_file.__fspath__()
class TestTrinoHookConn:
@@ -140,6 +132,28 @@ class TestTrinoHookConn:
TrinoHook().get_conn()
self.assert_connection_called_with(mock_connect, auth=mock_jwt_auth)
+ @pytest.mark.parametrize(
+ "jwt_file, jwt_token, error_suffix",
+ [
+ pytest.param(True, True, "provided both", id="provided-both-params"),
+ pytest.param(False, False, "none of them provided", id="no-jwt-provided"),
+ ],
+ )
+ @patch(HOOK_GET_CONNECTION)
+ def test_exactly_one_jwt_token(
+ self, mock_get_connection, jwt_file, jwt_token, error_suffix, jwt_token_file
+ ):
+ error_match = f"When auth set to 'jwt'.*{error_suffix}"
+ extras = {"auth": "jwt"}
+ if jwt_file:
+ extras["jwt__file"] = jwt_token_file
+ if jwt_token:
+ extras["jwt__token"] = "TEST_JWT_TOKEN"
+
+ self.set_get_connection_return_value(mock_get_connection, extra=json.dumps(extras))
+ with pytest.raises(ValueError, match=error_match):
+ TrinoHook().get_conn()
+
@patch(CERT_AUTHENTICATION)
@patch(TRINO_DBAPI_CONNECT)
@patch(HOOK_GET_CONNECTION)
(airflow) 10/37: Add basic metrics to stats collector. (#35368)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 5d69fc142d9c475940d11b1bbb56bf1a6c36c7fb
Author: Jakub Dardzinski <ku...@gmail.com>
AuthorDate: Tue Nov 21 06:45:04 2023 +0100
Add basic metrics to stats collector. (#35368)
Signed-off-by: Jakub Dardzinski <ku...@gmail.com>
Co-authored-by: Elad Kalif <45...@users.noreply.github.com>
---
airflow/providers/openlineage/plugins/adapter.py | 5 +-
.../plugins/test_openlineage_adapter.py | 75 +++++++++++++++++++---
2 files changed, 69 insertions(+), 11 deletions(-)
diff --git a/airflow/providers/openlineage/plugins/adapter.py b/airflow/providers/openlineage/plugins/adapter.py
index fabb14eaa3..a925ddf8e6 100644
--- a/airflow/providers/openlineage/plugins/adapter.py
+++ b/airflow/providers/openlineage/plugins/adapter.py
@@ -38,6 +38,7 @@ from openlineage.client.run import Job, Run, RunEvent, RunState
from airflow.configuration import conf
from airflow.providers.openlineage import __version__ as OPENLINEAGE_PROVIDER_VERSION
from airflow.providers.openlineage.utils.utils import OpenLineageRedactor
+from airflow.stats import Stats
from airflow.utils.log.logging_mixin import LoggingMixin
if TYPE_CHECKING:
@@ -113,8 +114,10 @@ class OpenLineageAdapter(LoggingMixin):
self._client = self.get_or_create_openlineage_client()
redacted_event: RunEvent = self._redacter.redact(event, max_depth=20) # type: ignore[assignment]
try:
- return self._client.emit(redacted_event)
+ with Stats.timer("ol.emit.attempts"):
+ return self._client.emit(redacted_event)
except Exception as e:
+ Stats.incr("ol.emit.failed")
self.log.warning("Failed to emit OpenLineage event of id %s", event.run.runId)
self.log.debug("OpenLineage emission failure: %s", e)
diff --git a/tests/providers/openlineage/plugins/test_openlineage_adapter.py b/tests/providers/openlineage/plugins/test_openlineage_adapter.py
index 685e88c725..bcb92b2b9b 100644
--- a/tests/providers/openlineage/plugins/test_openlineage_adapter.py
+++ b/tests/providers/openlineage/plugins/test_openlineage_adapter.py
@@ -39,7 +39,10 @@ from tests.test_utils.config import conf_vars
pytestmark = pytest.mark.db_test
-@patch.dict(os.environ, {"OPENLINEAGE_URL": "http://ol-api:5000", "OPENLINEAGE_API_KEY": "api-key"})
+@patch.dict(
+ os.environ,
+ {"OPENLINEAGE_URL": "http://ol-api:5000", "OPENLINEAGE_API_KEY": "api-key"},
+)
def test_create_client_from_ol_env():
client = OpenLineageAdapter().get_or_create_openlineage_client()
@@ -90,7 +93,11 @@ def test_create_client_from_env_var_config():
@patch.dict(
- os.environ, {"OPENLINEAGE_URL": "http://ol-from-env:5000", "OPENLINEAGE_API_KEY": "api-key-from-env"}
+ os.environ,
+ {
+ "OPENLINEAGE_URL": "http://ol-from-env:5000",
+ "OPENLINEAGE_API_KEY": "api-key-from-env",
+ },
)
@patch.dict(os.environ, {"OPENLINEAGE_CONFIG": "some/config.yml"})
def test_create_client_overrides_env_vars():
@@ -108,7 +115,9 @@ def test_create_client_overrides_env_vars():
assert client.transport.kind == "console"
-def test_emit_start_event():
+@mock.patch("airflow.providers.openlineage.plugins.adapter.Stats.timer")
+@mock.patch("airflow.providers.openlineage.plugins.adapter.Stats.incr")
+def test_emit_start_event(mock_stats_incr, mock_stats_timer):
client = MagicMock()
adapter = OpenLineageAdapter(client)
@@ -138,7 +147,8 @@ def test_emit_start_event():
runId=run_id,
facets={
"nominalTime": NominalTimeRunFacet(
- nominalStartTime="2022-01-01T00:00:00", nominalEndTime="2022-01-01T00:00:00"
+ nominalStartTime="2022-01-01T00:00:00",
+ nominalEndTime="2022-01-01T00:00:00",
),
"processing_engine": ProcessingEngineRunFacet(
version=ANY, name="Airflow", openlineageAdapterVersion=ANY
@@ -158,8 +168,13 @@ def test_emit_start_event():
in client.emit.mock_calls
)
+ mock_stats_incr.assert_not_called()
+ mock_stats_timer.assert_called_with("ol.emit.attempts")
+
-def test_emit_complete_event():
+@mock.patch("airflow.providers.openlineage.plugins.adapter.Stats.timer")
+@mock.patch("airflow.providers.openlineage.plugins.adapter.Stats.incr")
+def test_emit_complete_event(mock_stats_incr, mock_stats_timer):
client = MagicMock()
adapter = OpenLineageAdapter(client)
@@ -187,8 +202,13 @@ def test_emit_complete_event():
in client.emit.mock_calls
)
+ mock_stats_incr.assert_not_called()
+ mock_stats_timer.assert_called_with("ol.emit.attempts")
-def test_emit_failed_event():
+
+@mock.patch("airflow.providers.openlineage.plugins.adapter.Stats.timer")
+@mock.patch("airflow.providers.openlineage.plugins.adapter.Stats.incr")
+def test_emit_failed_event(mock_stats_incr, mock_stats_timer):
client = MagicMock()
adapter = OpenLineageAdapter(client)
@@ -216,9 +236,14 @@ def test_emit_failed_event():
in client.emit.mock_calls
)
+ mock_stats_incr.assert_not_called()
+ mock_stats_timer.assert_called_with("ol.emit.attempts")
+
@mock.patch("airflow.providers.openlineage.plugins.adapter.uuid")
-def test_emit_dag_started_event(uuid):
+@mock.patch("airflow.providers.openlineage.plugins.adapter.Stats.timer")
+@mock.patch("airflow.providers.openlineage.plugins.adapter.Stats.incr")
+def test_emit_dag_started_event(mock_stats_incr, mock_stats_timer, uuid):
random_uuid = "9d3b14f7-de91-40b6-aeef-e887e2c7673e"
client = MagicMock()
adapter = OpenLineageAdapter(client)
@@ -248,7 +273,8 @@ def test_emit_dag_started_event(uuid):
runId=random_uuid,
facets={
"nominalTime": NominalTimeRunFacet(
- nominalStartTime=event_time.isoformat(), nominalEndTime=event_time.isoformat()
+ nominalStartTime=event_time.isoformat(),
+ nominalEndTime=event_time.isoformat(),
)
},
),
@@ -261,9 +287,14 @@ def test_emit_dag_started_event(uuid):
in client.emit.mock_calls
)
+ mock_stats_incr.assert_not_called()
+ mock_stats_timer.assert_called_with("ol.emit.attempts")
+
@mock.patch("airflow.providers.openlineage.plugins.adapter.uuid")
-def test_emit_dag_complete_event(uuid):
+@mock.patch("airflow.providers.openlineage.plugins.adapter.Stats.timer")
+@mock.patch("airflow.providers.openlineage.plugins.adapter.Stats.incr")
+def test_emit_dag_complete_event(mock_stats_incr, mock_stats_timer, uuid):
random_uuid = "9d3b14f7-de91-40b6-aeef-e887e2c7673e"
client = MagicMock()
adapter = OpenLineageAdapter(client)
@@ -298,9 +329,14 @@ def test_emit_dag_complete_event(uuid):
in client.emit.mock_calls
)
+ mock_stats_incr.assert_not_called()
+ mock_stats_timer.assert_called_with("ol.emit.attempts")
+
@mock.patch("airflow.providers.openlineage.plugins.adapter.uuid")
-def test_emit_dag_failed_event(uuid):
+@mock.patch("airflow.providers.openlineage.plugins.adapter.Stats.timer")
+@mock.patch("airflow.providers.openlineage.plugins.adapter.Stats.incr")
+def test_emit_dag_failed_event(mock_stats_incr, mock_stats_timer, uuid):
random_uuid = "9d3b14f7-de91-40b6-aeef-e887e2c7673e"
client = MagicMock()
adapter = OpenLineageAdapter(client)
@@ -341,3 +377,22 @@ def test_emit_dag_failed_event(uuid):
)
in client.emit.mock_calls
)
+
+ mock_stats_incr.assert_not_called()
+ mock_stats_timer.assert_called_with("ol.emit.attempts")
+
+
+@patch("airflow.providers.openlineage.plugins.adapter.OpenLineageAdapter.get_or_create_openlineage_client")
+@patch("airflow.providers.openlineage.plugins.adapter.OpenLineageRedactor")
+@patch("airflow.providers.openlineage.plugins.adapter.Stats.timer")
+@patch("airflow.providers.openlineage.plugins.adapter.Stats.incr")
+def test_openlineage_adapter_stats_emit_failed(
+ mock_stats_incr, mock_stats_timer, mock_redact, mock_get_client
+):
+ adapter = OpenLineageAdapter()
+ mock_get_client.return_value.emit.side_effect = Exception()
+
+ adapter.emit(MagicMock())
+
+ mock_stats_timer.assert_called_with("ol.emit.attempts")
+ mock_stats_incr.assert_has_calls([mock.call("ol.emit.failed")])
(airflow) 37/37: Update RELEASE_NOTES.rst
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 5bbc46005f20c20380d18907899b0ddff33eecde
Author: Ephraim Anierobi <sp...@gmail.com>
AuthorDate: Mon Nov 20 22:51:30 2023 +0100
Update RELEASE_NOTES.rst
---
RELEASE_NOTES.rst | 169 +++++++++++++++++++++++++++++++++++-
newsfragments/35460.significant.rst | 10 ---
2 files changed, 168 insertions(+), 11 deletions(-)
diff --git a/RELEASE_NOTES.rst b/RELEASE_NOTES.rst
index 62183c8b58..7a5883154f 100644
--- a/RELEASE_NOTES.rst
+++ b/RELEASE_NOTES.rst
@@ -21,7 +21,174 @@
.. towncrier release notes start
-Airflow 2.7.3 (2023-11-04)
+Airflow 2.8.0 (2023-12-14)
+--------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+- Raw HTML code in DAG docs and DAG params descriptions is disabled by default
+
+ To ensure that no malicious javascript can be injected with DAG descriptions or trigger UI forms by DAG authors
+ a new parameter ``webserver.allow_raw_html_descriptions`` was added with default value of ``False``.
+ If you trust your DAG authors code and want to allow using raw HTML in DAG descriptions and params, you can restore the previous
+ behavior by setting the configuration value to ``True``.
+
+ To ensure Airflow is secure by default, the raw HTML support in trigger UI has been super-seeded by markdown support via
+ the ``description_md`` attribute. If you have been using ``description_html`` please migrate to ``description_md``.
+ The ``custom_html_form`` is now deprecated. (#35460)
+
+New Features
+""""""""""""
+- AIP-58: Add Airflow ObjectStore (AFS) (`AIP-58 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-58+milestone%3A%22Airflow+2.8.0%22>`_)
+- Add "literal" wrapper to disable field templating (#35017)
+- Add task context logging feature to allow forwarding messages to task logs (#32646, #32693)
+- Add Listener hooks for Datasets (#34418)
+- Allow override of navbar text color (#35505)
+- Add lightweight serialization for deltalake tables (#35462)
+- Add support for serialization of iceberg tables (#35456)
+- ``prev_end_date_success`` method access (#34528)
+- Add task parameter to set custom logger name (#34964)
+- Add pyspark decorator (#35247)
+- Add trigger as a valid option for the db clean command (#34908)
+- Add decorators for external and venv python branching operators (#35043)
+- Allow PythonVenvOperator using other index url (#33017)
+- Add Python Virtualenv Operator Caching (#33355)
+- Introduce a generic export for containerized executor logging (#34903)
+- Add ability to clear downstream tis in ``List Task Instances`` view (#34529)
+- Attribute ``clear_number`` to track DAG run being cleared (#34126)
+- Add BranchPythonVirtualenvOperator (#33356)
+- Allow PythonVenvOperator using other index url (#33017)
+- Add CLI notification commands to providers (#33116)
+
+Improvements
+""""""""""""
+- Move ``BaseOperatorLink`` into the separate module (#35032)
+- Set mark_end_on_close after set_context (#35761)
+- Move external logs links to top of react logs page (#35668)
+- Change terminal mode to ``cbreak`` in ``execute_interactive`` and handle ``SIGINT`` (#35602)
+- Make raw HTML descriptions configurable (#35460)
+- Allow email field to be templated (#35546)
+- Hide logical date and run id in trigger UI form (#35284)
+- Improved instructions for adding dependencies in TaskFlow (#35406)
+- Add optional exit code to list import errors (#35378)
+- Limit query result on DB rather than client in ``synchronize_log_template`` function (#35366)
+- Feature: Allow description to be passed in when using variables CLI (#34791)
+- Allow optional defaults in required fields with manual triggered dags (#31301)
+- Permitting airflow kerberos to run in different modes (#35146)
+- Refactor commands to unify daemon context handling (#34945)
+- Add extra fields to plugins endpoint (#34913)
+- Add description to pools view (#34862)
+- Move cli's Connection export and Variable export command print logic to a separate function (#34647)
+- Extract and reuse get_kerberos_principle func from get_kerberos_principle (#34936)
+- Change type annotation for ``BaseOperatorLink.operators`` (#35003)
+- Optimise and migrate to ``SA2-compatible`` syntax for TaskReschedule (#33720)
+- Consolidate the permissions name in SlaMissModelView (#34949)
+- Add debug log saying what's being run to ``EventScheduler`` (#34808)
+- Increase log reader stream loop sleep duration to 1 second (#34789)
+- Resolve pydantic deprecation warnings re ``update_forward_refs`` (#34657)
+- Unify mapped task group lookup logic (#34637)
+- Allow filtering event logs by attributes (#34417)
+- Make connection login and password TEXT (#32815)
+- Ban import ``Dataset`` from ``airflow`` package in codebase (#34610)
+- Use ``airflow.datasets.Dataset`` in examples and tests (#34605)
+- Enhance task status visibility (#34486)
+- Simplify DAG trigger UI (#34567)
+- Ban import AirflowException from airflow (#34512)
+- Add descriptions for airflow resource config parameters (#34438)
+- Simplify trigger name expression (#34356)
+- Move definition of Pod*Exceptions to pod_generator (#34346)
+- Add deferred tasks to the cluster_activity view Pools Slots (#34275)
+- heartbeat failure log message fix (#34160)
+- Rename variables for dag runs (#34049)
+- Clarify new_state in OpenAPI spec (#34056)
+- Remove ``version`` top-level element from docker compose files (#33831)
+- Remove generic trigger cancelled error log (#33874)
+- Use ``NOT EXISTS`` subquery instead of ``tuple_not_in_condition`` (#33527)
+- Allow context key args to not provide a default (#33430)
+- Order triggers by - TI priority_weight when assign unassigned triggers (#32318)
+- Add metric ``triggerer_heartbeat`` (#33320)
+- Allow ``airflow variables export`` to print to stdout (#33279)
+- Workaround failing deadlock when running backfill (#32991)
+- add dag_run_ids and task_ids filter for the batch task instance API endpoint (#32705)
+- Configurable health check threshold for triggerer (#33089)
+- Rework provider manager to treat Airflow core hooks like other provider hooks (#33051)
+- Ensure DAG-level references are filled on unmap (#33083)
+- Affix webserver access_denied warning to be configurable (#33022)
+- Add support for arrays of different data types in the Trigger Form UI (#32734)
+- Add a mechanism to warn if executors override existing CLI commands (#33423)
+
+Bug Fixes
+"""""""""
+- Add ``borderWidthRight`` to grid for Firefox ``scrollbar`` (#35346)
+- Fix for infinite recursion due to secrets_masker (#35048)
+- Fix write ``processor_subdir`` in serialized_dag table (#35661)
+- Reload configuration for standalone dag file processor (#35725)
+- Long custom operator name overflows in graph view (#35382)
+- Add try_number to extra links query (#35317)
+- Prevent assignment of non JSON serializable values to DagRun.conf dict (#35096)
+- Numeric values in DAG details are incorrectly rendered as timestamps (#35538)
+- Fix Scheduler and triggerer crashes in daemon mode when statsd metrics are enabled (#35181)
+- Infinite UI redirection loop after deactivating an active user (#35486)
+- Bug fix fetch_callback of Partial Subset DAG (#35256)
+- Fix DagRun data interval for DeltaDataIntervalTimetable (#35391)
+- Fix query in ``get_dag_by_pickle`` util function (#35339)
+- Fix TriggerDagRunOperator failing to trigger subsequent runs when reset_dag_run=True (#35429)
+- Fix weight_rule property type in ``mappedoperator`` (#35257)
+- Bugfix/prevent concurrency with cached venv (#35258)
+- Fix dag serialization (#34042)
+- Fix py/url-redirection by replacing request.referrer by get_redirect() (#34237)
+- Fix updating variables during variable imports (#33932)
+- Use Literal from airflow.typing_compat in Airflow core (#33821)
+- Always use ``Literal`` from ``typing_extensions`` (#33794)
+
+Misc/Internal
+"""""""""""""
+- Upgrade to Pydantic v2 (#35551)
+- Bump ``moto`` version to ``>= 4.2.9`` (#35687)
+- Use ``pyarrow-hotfix`` to mitigate CVE-2023-47248 (#35650)
+- Bump ``axios`` from ``0.26.0 to 1.6.0`` in ``/airflow/www/`` (#35624)
+- Make docker decorator's type annotation consistent with operator (#35568)
+- Add default to ``navbar_text_color`` and ``rm`` condition in style (#35553)
+- Avoid initiating session twice in ``dag_next_execution`` (#35539)
+- Work around typing issue in examples and providers (#35494)
+- Enable ``TCH004`` and ``TCH005`` rules (#35475)
+- Humanize log output about retrieved DAG(s) (#35338)
+- Switch from Black to Ruff formatter (#35287)
+- Upgrade to Flask Application Builder 4.3.9 (#35085)
+- D401 Support (#34932, #34933)
+- Use requires_access to check read permission on dag instead of checking it explicitly (#34940)
+- Deprecate lazy import ``AirflowException`` from airflow (#34541)
+- View util refactoring on mapped stuff use cases (#34638)
+- Bump ``postcss`` from ``8.4.25 to 8.4.31`` in ``/airflow/www`` (#34770)
+- Refactor Sqlalchemy queries to 2.0 style (#34763, #34665, #32883, #35120)
+- Change to lazy loading of io in pandas serializer (#34684)
+- Use ``airflow.models.dag.DAG`` in examples (#34617)
+- Use airflow.exceptions.AirflowException in core (#34510)
+- Check that dag_ids passed in request are consistent (#34366)
+- Refactors to make code better (#34278, #34113, #34110, #33838, #34260, #34409, #34377, #34350)
+- Suspend qubole provider (#33889)
+- Generate Python API docs for Google ADS (#33814)
+- Improve importing in modules (#33812, #33811, #33810, #33806, #33807, #33805, #33804, #33803,
+ #33801, #33799, #33800, #33797, #33798, #34406, #33808)
+- Upgrade Elasticsearch to 8 (#33135)
+
+Doc Only Changes
+""""""""""""""""
+- Reflect drop/add support of DB Backends versions in documentation (#35785)
+- More detail on mandatory task arguments (#35740)
+- Indicate usage of the ``re2`` regex engine in the .airflowignore documentation. (#35663)
+- Update ``best-practices.rst`` (#35692)
+- Update ``dag-run.rst`` to mention Airflow's support for extended cron syntax through croniter (#35342)
+- Update ``webserver.rst`` to include information of supported OAuth2 providers (#35237)
+- Add back dag_run to docs (#35142)
+- Fix ``rst`` code block format (#34708)
+- Add typing to concrete taskflow examples (#33417)
+- Add concrete examples for accessing context variables from TaskFlow tasks (#33296)
+- Fix links in security docs (#33329)
+
+
+Airflow 2.7.3 (2023-11-06)
--------------------------
Significant Changes
diff --git a/newsfragments/35460.significant.rst b/newsfragments/35460.significant.rst
deleted file mode 100644
index d29481d219..0000000000
--- a/newsfragments/35460.significant.rst
+++ /dev/null
@@ -1,10 +0,0 @@
-Raw HTML code in DAG docs and DAG params descriptions is disabled by default
-
-To ensure that no malicious javascript can be injected with DAG descriptions or trigger UI forms by DAG authors
-a new parameter ``webserver.allow_raw_html_descriptions`` was added with default value of ``False``.
-If you trust your DAG authors code and want to allow using raw HTML in DAG descriptions and params, you can restore the previous
-behavior by setting the configuration value to ``True``.
-
-To ensure Airflow is secure by default, the raw HTML support in trigger UI has been super-seeded by markdown support via
-the ``description_md`` attribute. If you have been using ``description_html`` please migrate to ``description_md``.
-The ``custom_html_form`` is now deprecated.
(airflow) 34/37: Updated docstring: `check_key_async` is now in line with description of `_check_key_async` (#35799)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 840707cedf75a65b297645e671f467148b704f10
Author: Aman Gupta <am...@gmail.com>
AuthorDate: Wed Nov 22 22:24:29 2023 +0530
Updated docstring: `check_key_async` is now in line with description of `_check_key_async` (#35799)
---
airflow/providers/amazon/aws/hooks/s3.py | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/airflow/providers/amazon/aws/hooks/s3.py b/airflow/providers/amazon/aws/hooks/s3.py
index ef54dc0099..ad25c5b5c5 100644
--- a/airflow/providers/amazon/aws/hooks/s3.py
+++ b/airflow/providers/amazon/aws/hooks/s3.py
@@ -518,7 +518,11 @@ class S3Hook(AwsBaseHook):
wildcard_match: bool,
) -> bool:
"""
- Check for all keys in bucket and returns boolean value.
+ Get a list of files that a key matching a wildcard expression or get the head object.
+
+ If wildcard_match is True get list of files that a key matching a wildcard
+ expression exists in a bucket asynchronously and return the boolean value. If wildcard_match
+ is False get the head object from the bucket and return the boolean value.
:param client: aiobotocore client
:param bucket: the name of the bucket
(airflow) 01/37: Add v2-8 branches to codecov.yml and .asf.yaml (#35750)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 1fc0633f894ed8902b8a73af21421eeb380af839
Author: Ephraim Anierobi <sp...@gmail.com>
AuthorDate: Mon Nov 20 15:10:37 2023 +0100
Add v2-8 branches to codecov.yml and .asf.yaml (#35750)
---
.asf.yaml | 3 +++
codecov.yml | 2 ++
2 files changed, 5 insertions(+)
diff --git a/.asf.yaml b/.asf.yaml
index 6166687a5d..094299d2c4 100644
--- a/.asf.yaml
+++ b/.asf.yaml
@@ -71,6 +71,9 @@ github:
v2-7-stable:
required_pull_request_reviews:
required_approving_review_count: 1
+ v2-8-stable:
+ required_pull_request_reviews:
+ required_approving_review_count: 1
collaborators:
- mhenc
diff --git a/codecov.yml b/codecov.yml
index 67ea777302..d1ed5fb446 100644
--- a/codecov.yml
+++ b/codecov.yml
@@ -55,6 +55,8 @@ coverage:
- v2-6-test
- v2-7-stable
- v2-7-test
+ - v2-8-stable
+ - v2-8-test
if_not_found: success
if_ci_failed: error
informational: true
(airflow) 15/37: Set mark_end_on_close after set_context (#35761)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit f40e1c17ee07ebeb6ebf1494e1fc845a00e599dc
Author: Daniel Standish <15...@users.noreply.github.com>
AuthorDate: Tue Nov 21 05:42:18 2023 -0800
Set mark_end_on_close after set_context (#35761)
In ES task handler, set_context applies its own logic for mark_end_on_close, so it we must set the attr after for our override to persist.
---
airflow/utils/log/task_context_logger.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/airflow/utils/log/task_context_logger.py b/airflow/utils/log/task_context_logger.py
index 0661789f5b..84ed207e3a 100644
--- a/airflow/utils/log/task_context_logger.py
+++ b/airflow/utils/log/task_context_logger.py
@@ -98,9 +98,9 @@ class TaskContextLogger:
task_handler = copy(self.task_handler)
try:
+ task_handler.set_context(ti, identifier=self.component_name)
if hasattr(task_handler, "mark_end_on_close"):
task_handler.mark_end_on_close = False
- task_handler.set_context(ti, identifier=self.component_name)
filename, lineno, func, stackinfo = logger.findCaller()
record = logging.LogRecord(
self.component_name, level, filename, lineno, msg, args, None, func=func
(airflow) 23/37: Fix permission check on menus (#35781)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit b904523c5a2d8e7d0c359d45eef8514297b1f52c
Author: Vincent <97...@users.noreply.github.com>
AuthorDate: Tue Nov 21 14:44:29 2023 -0500
Fix permission check on menus (#35781)
---
airflow/www/security_manager.py | 23 ++++----
tests/www/test_security_manager.py | 115 +++++++++++++++++++++++++++++++++++++
2 files changed, 126 insertions(+), 12 deletions(-)
diff --git a/airflow/www/security_manager.py b/airflow/www/security_manager.py
index 78d8ef00fe..13342ff7f7 100644
--- a/airflow/www/security_manager.py
+++ b/airflow/www/security_manager.py
@@ -137,15 +137,7 @@ class AirflowSecurityManagerV2(LoggingMixin):
user = g.user
is_authorized_method = self._get_auth_manager_is_authorized_method(resource_name)
- if is_authorized_method:
- return is_authorized_method(action_name, resource_pk, user)
- else:
- # This means the page the user is trying to access is specific to the auth manager used
- # Example: the user list view in FabAuthManager
- action_name = ACTION_CAN_READ if action_name == ACTION_CAN_ACCESS_MENU else action_name
- return get_auth_manager().is_authorized_custom_view(
- fab_action_name=action_name, fab_resource_name=resource_name, user=user
- )
+ return is_authorized_method(action_name, resource_pk, user)
def create_admin_standalone(self) -> tuple[str | None, str | None]:
"""Perform the required steps when initializing airflow for standalone mode.
@@ -331,7 +323,7 @@ class AirflowSecurityManagerV2(LoggingMixin):
),
}
- def _get_auth_manager_is_authorized_method(self, fab_resource_name: str) -> Callable | None:
+ def _get_auth_manager_is_authorized_method(self, fab_resource_name: str) -> Callable:
is_authorized_method = self._auth_manager_is_authorized_map.get(fab_resource_name)
if is_authorized_method:
return is_authorized_method
@@ -340,12 +332,19 @@ class AirflowSecurityManagerV2(LoggingMixin):
# least one dropdown child
return self._is_authorized_category_menu(fab_resource_name)
else:
- return None
+ # This means the page the user is trying to access is specific to the auth manager used
+ # Example: the user list view in FabAuthManager
+ return lambda action, resource_pk, user: get_auth_manager().is_authorized_custom_view(
+ fab_action_name=ACTION_CAN_READ if action == ACTION_CAN_ACCESS_MENU else action,
+ fab_resource_name=fab_resource_name,
+ user=user,
+ )
def _is_authorized_category_menu(self, category: str) -> Callable:
items = {item.name for item in self.appbuilder.menu.find(category).childs}
return lambda action, resource_pk, user: any(
- self._get_auth_manager_is_authorized_method(fab_resource_name=item) for item in items
+ self._get_auth_manager_is_authorized_method(fab_resource_name=item)(action, resource_pk, user)
+ for item in items
)
"""
diff --git a/tests/www/test_security_manager.py b/tests/www/test_security_manager.py
new file mode 100644
index 0000000000..fafb539ee3
--- /dev/null
+++ b/tests/www/test_security_manager.py
@@ -0,0 +1,115 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from unittest import mock
+from unittest.mock import Mock
+
+import pytest
+
+from airflow.security.permissions import (
+ ACTION_CAN_READ,
+ RESOURCE_ADMIN_MENU,
+ RESOURCE_BROWSE_MENU,
+ RESOURCE_DOCS_MENU,
+ RESOURCE_VARIABLE,
+)
+from airflow.www import app as application
+
+
+@pytest.fixture()
+def app():
+ return application.create_app(testing=True)
+
+
+@pytest.fixture()
+def app_builder(app):
+ return app.appbuilder
+
+
+@pytest.fixture()
+def security_manager(app_builder):
+ return app_builder.sm
+
+
+@pytest.mark.db_test
+class TestAirflowSecurityManagerV2:
+ @pytest.mark.parametrize(
+ "resource_name, auth_manager_methods, expected",
+ [
+ (RESOURCE_VARIABLE, {"is_authorized_variable": True}, True),
+ (RESOURCE_VARIABLE, {"is_authorized_variable": False}, False),
+ (RESOURCE_DOCS_MENU, {"is_authorized_view": True}, True),
+ (RESOURCE_DOCS_MENU, {"is_authorized_view": False}, False),
+ (
+ RESOURCE_ADMIN_MENU,
+ {
+ "is_authorized_view": False,
+ "is_authorized_variable": False,
+ "is_authorized_connection": True,
+ "is_authorized_dag": False,
+ "is_authorized_configuration": False,
+ "is_authorized_pool": False,
+ },
+ True,
+ ),
+ (
+ RESOURCE_ADMIN_MENU,
+ {
+ "is_authorized_view": False,
+ "is_authorized_variable": False,
+ "is_authorized_connection": False,
+ "is_authorized_dag": False,
+ "is_authorized_configuration": False,
+ "is_authorized_pool": False,
+ },
+ False,
+ ),
+ (
+ RESOURCE_BROWSE_MENU,
+ {
+ "is_authorized_dag": False,
+ "is_authorized_view": False,
+ },
+ False,
+ ),
+ (
+ RESOURCE_BROWSE_MENU,
+ {
+ "is_authorized_dag": False,
+ "is_authorized_view": True,
+ },
+ True,
+ ),
+ ],
+ )
+ @mock.patch("airflow.www.security_manager.get_auth_manager")
+ def test_has_access(
+ self, mock_get_auth_manager, security_manager, resource_name, auth_manager_methods, expected
+ ):
+ user = Mock()
+ auth_manager = Mock()
+ for method_name, method_return in auth_manager_methods.items():
+ method = Mock(return_value=method_return)
+ getattr(auth_manager, method_name).side_effect = method
+ mock_get_auth_manager.return_value = auth_manager
+ result = security_manager.has_access(ACTION_CAN_READ, resource_name, user=user)
+ assert result == expected
+ if len(auth_manager_methods) > 1 and not expected:
+ for method_name in auth_manager_methods:
+ getattr(auth_manager, method_name).assert_called()
(airflow) 24/37: Remove pendulum as dependency of breeze (#35786)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 2a240e65d5a9da93eb1008e8e0a9ef93eb216190
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Tue Nov 21 22:29:37 2023 +0100
Remove pendulum as dependency of breeze (#35786)
For some reason (likely importing some stuff from setup.py in the
old days) pendulum was added as dependency in breeze - which still
caused a problem when `pipx` decided to use Python 3.12 to install
Breeze (despite #35652 that was supposed to supersede #35620).
The #35620 adding a need to specify python additionally when you
install breeze added it's own complexity (which python?), it
turns out that breeze does not need to have pendulum installed
at all now (we stopped depending on airflow being installed
and stopped importing things from setup.py or __version__ in
favour of directly parsing __version__ variable from python code.
This PR removes pendulum entirely as Breeze dependency.
---
dev/breeze/README.md | 2 +-
dev/breeze/pyproject.toml | 1 -
2 files changed, 1 insertion(+), 2 deletions(-)
diff --git a/dev/breeze/README.md b/dev/breeze/README.md
index b2d7130210..9db81c1fd4 100644
--- a/dev/breeze/README.md
+++ b/dev/breeze/README.md
@@ -66,6 +66,6 @@ PLEASE DO NOT MODIFY THE HASH BELOW! IT IS AUTOMATICALLY UPDATED BY PRE-COMMIT.
---------------------------------------------------------------------------------------------------------
-Package config hash: a5878ba073fa5924f21660531f0988f287269f0d3aca741095cad62b3a1f3ccb262f76df203aff1f02cfec691f839da02bc6844342f49e40f896a1c9b3c450d8
+Package config hash: c7d80ab49c6dc4bf2b54957663b0126ab9c8f48df28a34c0eb56340540cb1f52d063ef99ee5f9cacbd375b1a711278884f9ef9aab41e620fa70fffd81f7ece3c
---------------------------------------------------------------------------------------------------------
diff --git a/dev/breeze/pyproject.toml b/dev/breeze/pyproject.toml
index 0f0e47a517..aca59140ac 100644
--- a/dev/breeze/pyproject.toml
+++ b/dev/breeze/pyproject.toml
@@ -54,7 +54,6 @@ dependencies = [
"jinja2>=3.1.0",
"jsonschema>=4.19.1",
"packaging>=23.2",
- "pendulum>=2.1.2,<3",
"pre-commit>=3.5.0",
"psutil>=5.9.6",
"pygithub>=2.1.1",
(airflow) 26/37: Update emr.py (#35787)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit e0736092a89b1b8c46d389554c5e6abadcf8de21
Author: Tony Zhang <45...@users.noreply.github.com>
AuthorDate: Tue Nov 21 15:24:49 2023 -0800
Update emr.py (#35787)
---
airflow/providers/amazon/aws/operators/emr.py | 18 +++++++++++-------
1 file changed, 11 insertions(+), 7 deletions(-)
diff --git a/airflow/providers/amazon/aws/operators/emr.py b/airflow/providers/amazon/aws/operators/emr.py
index 1067464474..1f0b247b75 100644
--- a/airflow/providers/amazon/aws/operators/emr.py
+++ b/airflow/providers/amazon/aws/operators/emr.py
@@ -1270,14 +1270,18 @@ class EmrServerlessStartJobOperator(BaseOperator):
)
self.log.info("Starting job on Application: %s", self.application_id)
self.name = self.name or self.config.pop("name", f"emr_serverless_job_airflow_{uuid4()}")
- response = self.hook.conn.start_job_run(
- clientToken=self.client_request_token,
- applicationId=self.application_id,
- executionRoleArn=self.execution_role_arn,
- jobDriver=self.job_driver,
- configurationOverrides=self.configuration_overrides,
- name=self.name,
+ args = {
+ "clientToken": self.client_request_token,
+ "applicationId": self.application_id,
+ "executionRoleArn": self.execution_role_arn,
+ "jobDriver": self.job_driver,
+ "name": self.name,
**self.config,
+ }
+ if self.configuration_overrides is not None:
+ args["configurationOverrides"] = self.configuration_overrides
+ response = self.hook.conn.start_job_run(
+ **args,
)
if response["ResponseMetadata"]["HTTPStatusCode"] != 200:
(airflow) 32/37: Upgrade to Pydantic v2 (#35551)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit b331cb2e0e3c47e3e2322decba29cba1d6f1aecf
Author: Sin-Woo Bang <si...@gmail.com>
AuthorDate: Wed Nov 22 20:56:43 2023 +0900
Upgrade to Pydantic v2 (#35551)
* Replace deprecated Config with ConfigDict
* Drop Pydantic v1 compatibility as bumping it to 2.3.0
---
airflow/configuration.py | 4 ----
airflow/serialization/pydantic/dag.py | 29 +++++++++-----------------
airflow/serialization/pydantic/dag_run.py | 9 ++------
airflow/serialization/pydantic/dataset.py | 27 +++++-------------------
airflow/serialization/pydantic/job.py | 8 ++-----
airflow/serialization/pydantic/taskinstance.py | 9 ++------
airflow/serialization/serde.py | 9 +-------
airflow/serialization/serialized_objects.py | 5 +----
setup.cfg | 6 +-----
9 files changed, 24 insertions(+), 82 deletions(-)
diff --git a/airflow/configuration.py b/airflow/configuration.py
index 6b03759033..1df62c9b99 100644
--- a/airflow/configuration.py
+++ b/airflow/configuration.py
@@ -62,10 +62,6 @@ if not sys.warnoptions:
warnings.filterwarnings(action="default", category=DeprecationWarning, module="airflow")
warnings.filterwarnings(action="default", category=PendingDeprecationWarning, module="airflow")
- # Temporarily suppress warnings from pydantic until we upgrade minimum version of pydantic to v2
- # Which should happen in Airflow 2.8.0
- warnings.filterwarnings(action="ignore", category=UserWarning, module=r"pydantic._internal._config")
-
_SQLITE3_VERSION_PATTERN = re2.compile(r"(?P<version>^\d+(?:\.\d+)*)\D?.*$")
ConfigType = Union[str, int, float, bool]
diff --git a/airflow/serialization/pydantic/dag.py b/airflow/serialization/pydantic/dag.py
index 6631afdf73..04b2472355 100644
--- a/airflow/serialization/pydantic/dag.py
+++ b/airflow/serialization/pydantic/dag.py
@@ -21,7 +21,13 @@ from datetime import datetime, timedelta
from typing import Any, List, Optional
from dateutil import relativedelta
-from pydantic import BaseModel as BaseModelPydantic, PlainSerializer, PlainValidator, ValidationInfo
+from pydantic import (
+ BaseModel as BaseModelPydantic,
+ ConfigDict,
+ PlainSerializer,
+ PlainValidator,
+ ValidationInfo,
+)
from typing_extensions import Annotated
from airflow import DAG, settings
@@ -86,12 +92,7 @@ class DagOwnerAttributesPydantic(BaseModelPydantic):
owner: str
link: str
- class Config:
- """Make sure it deals automatically with SQLAlchemy ORM classes."""
-
- from_attributes = True
- orm_mode = True # Pydantic 1.x compatibility.
- arbitrary_types_allowed = True
+ model_config = ConfigDict(from_attributes=True, arbitrary_types_allowed=True)
class DagTagPydantic(BaseModelPydantic):
@@ -100,12 +101,7 @@ class DagTagPydantic(BaseModelPydantic):
name: str
dag_id: str
- class Config:
- """Make sure it deals automatically with SQLAlchemy ORM classes."""
-
- from_attributes = True
- orm_mode = True # Pydantic 1.x compatibility.
- arbitrary_types_allowed = True
+ model_config = ConfigDict(from_attributes=True, arbitrary_types_allowed=True)
class DagModelPydantic(BaseModelPydantic):
@@ -141,12 +137,7 @@ class DagModelPydantic(BaseModelPydantic):
_processor_dags_folder: Optional[str] = None
- class Config:
- """Make sure it deals automatically with SQLAlchemy ORM classes."""
-
- from_attributes = True
- orm_mode = True # Pydantic 1.x compatibility.
- arbitrary_types_allowed = True
+ model_config = ConfigDict(from_attributes=True, arbitrary_types_allowed=True)
@property
def relative_fileloc(self) -> pathlib.Path:
diff --git a/airflow/serialization/pydantic/dag_run.py b/airflow/serialization/pydantic/dag_run.py
index aaa4372a50..cd0886ecaf 100644
--- a/airflow/serialization/pydantic/dag_run.py
+++ b/airflow/serialization/pydantic/dag_run.py
@@ -19,7 +19,7 @@ from __future__ import annotations
from datetime import datetime
from typing import TYPE_CHECKING, Iterable, List, Optional
-from pydantic import BaseModel as BaseModelPydantic
+from pydantic import BaseModel as BaseModelPydantic, ConfigDict
from airflow.serialization.pydantic.dag import PydanticDag
from airflow.serialization.pydantic.dataset import DatasetEventPydantic
@@ -56,12 +56,7 @@ class DagRunPydantic(BaseModelPydantic):
dag: Optional[PydanticDag]
consumed_dataset_events: List[DatasetEventPydantic] # noqa
- class Config:
- """Make sure it deals automatically with SQLAlchemy ORM classes."""
-
- from_attributes = True
- orm_mode = True # Pydantic 1.x compatibility.
- arbitrary_types_allowed = True
+ model_config = ConfigDict(from_attributes=True, arbitrary_types_allowed=True)
@property
def logical_date(self) -> datetime:
diff --git a/airflow/serialization/pydantic/dataset.py b/airflow/serialization/pydantic/dataset.py
index 14255c8271..0c233a3fd6 100644
--- a/airflow/serialization/pydantic/dataset.py
+++ b/airflow/serialization/pydantic/dataset.py
@@ -17,7 +17,7 @@
from datetime import datetime
from typing import List, Optional
-from pydantic import BaseModel as BaseModelPydantic
+from pydantic import BaseModel as BaseModelPydantic, ConfigDict
class DagScheduleDatasetReferencePydantic(BaseModelPydantic):
@@ -28,11 +28,7 @@ class DagScheduleDatasetReferencePydantic(BaseModelPydantic):
created_at: datetime
updated_at: datetime
- class Config:
- """Make sure it deals automatically with SQLAlchemy ORM classes."""
-
- from_attributes = True
- orm_mode = True # Pydantic 1.x compatibility.
+ model_config = ConfigDict(from_attributes=True)
class TaskOutletDatasetReferencePydantic(BaseModelPydantic):
@@ -44,11 +40,7 @@ class TaskOutletDatasetReferencePydantic(BaseModelPydantic):
created_at: datetime
updated_at: datetime
- class Config:
- """Make sure it deals automatically with SQLAlchemy ORM classes."""
-
- from_attributes = True
- orm_mode = True # Pydantic 1.x compatibility.
+ model_config = ConfigDict(from_attributes=True)
class DatasetPydantic(BaseModelPydantic):
@@ -64,11 +56,7 @@ class DatasetPydantic(BaseModelPydantic):
consuming_dags: List[DagScheduleDatasetReferencePydantic]
producing_tasks: List[TaskOutletDatasetReferencePydantic]
- class Config:
- """Make sure it deals automatically with SQLAlchemy ORM classes."""
-
- from_attributes = True
- orm_mode = True # Pydantic 1.x compatibility.
+ model_config = ConfigDict(from_attributes=True)
class DatasetEventPydantic(BaseModelPydantic):
@@ -84,9 +72,4 @@ class DatasetEventPydantic(BaseModelPydantic):
timestamp: datetime
dataset: Optional[DatasetPydantic]
- class Config:
- """Make sure it deals automatically with SQLAlchemy ORM classes."""
-
- from_attributes = True
- orm_mode = True # Pydantic 1.x compatibility.
- arbitrary_types_allowed = True
+ model_config = ConfigDict(from_attributes=True, arbitrary_types_allowed=True)
diff --git a/airflow/serialization/pydantic/job.py b/airflow/serialization/pydantic/job.py
index eb92411090..2db30ab2c8 100644
--- a/airflow/serialization/pydantic/job.py
+++ b/airflow/serialization/pydantic/job.py
@@ -18,7 +18,7 @@ import datetime
from functools import cached_property
from typing import Optional
-from pydantic import BaseModel as BaseModelPydantic
+from pydantic import BaseModel as BaseModelPydantic, ConfigDict
from airflow.executors.executor_loader import ExecutorLoader
from airflow.jobs.base_job_runner import BaseJobRunner
@@ -44,11 +44,7 @@ class JobPydantic(BaseModelPydantic):
hostname: Optional[str]
unixname: Optional[str]
- class Config:
- """Make sure it deals automatically with SQLAlchemy ORM classes."""
-
- from_attributes = True
- orm_mode = True # Pydantic 1.x compatibility.
+ model_config = ConfigDict(from_attributes=True)
@cached_property
def executor(self):
diff --git a/airflow/serialization/pydantic/taskinstance.py b/airflow/serialization/pydantic/taskinstance.py
index 0043bfaef0..106a31186e 100644
--- a/airflow/serialization/pydantic/taskinstance.py
+++ b/airflow/serialization/pydantic/taskinstance.py
@@ -19,7 +19,7 @@ from __future__ import annotations
from datetime import datetime
from typing import TYPE_CHECKING, Any, Iterable, Optional
-from pydantic import BaseModel as BaseModelPydantic, PlainSerializer, PlainValidator
+from pydantic import BaseModel as BaseModelPydantic, ConfigDict, PlainSerializer, PlainValidator
from typing_extensions import Annotated
from airflow.models import Operator
@@ -105,12 +105,7 @@ class TaskInstancePydantic(BaseModelPydantic, LoggingMixin):
dag_run: Optional[DagRunPydantic]
dag_model: Optional[DagModelPydantic]
- class Config:
- """Make sure it deals automatically with SQLAlchemy ORM classes."""
-
- from_attributes = True
- orm_mode = True # Pydantic 1.x compatibility.
- arbitrary_types_allowed = True
+ model_config = ConfigDict(from_attributes=True, arbitrary_types_allowed=True)
def init_run_context(self, raw: bool = False) -> None:
"""Set the log context."""
diff --git a/airflow/serialization/serde.py b/airflow/serialization/serde.py
index c86f643505..23d67e6162 100644
--- a/airflow/serialization/serde.py
+++ b/airflow/serialization/serde.py
@@ -319,14 +319,7 @@ def _is_pydantic(cls: Any) -> bool:
Checking is done by attributes as it is significantly faster than
using isinstance.
"""
- return (
- hasattr(cls, "__validators__")
- and hasattr(cls, "__fields__")
- and hasattr(cls, "dict") # Pydantic v1
- or hasattr(cls, "model_config")
- and hasattr(cls, "model_fields")
- and hasattr(cls, "model_fields_set") # Pydantic v2
- )
+ return hasattr(cls, "model_config") and hasattr(cls, "model_fields") and hasattr(cls, "model_fields_set")
def _register():
diff --git a/airflow/serialization/serialized_objects.py b/airflow/serialization/serialized_objects.py
index 889177f4c1..9d7955bf01 100644
--- a/airflow/serialization/serialized_objects.py
+++ b/airflow/serialization/serialized_objects.py
@@ -502,10 +502,7 @@ class BaseSerialization:
elif use_pydantic_models and _ENABLE_AIP_44:
def _pydantic_model_dump(model_cls: type[BaseModel], var: Any) -> dict[str, Any]:
- try:
- return model_cls.model_validate(var).model_dump(mode="json") # type: ignore[attr-defined]
- except AttributeError: # Pydantic 1.x compatibility.
- return model_cls.from_orm(var).dict() # type: ignore[attr-defined]
+ return model_cls.model_validate(var).model_dump(mode="json") # type: ignore[attr-defined]
if isinstance(var, Job):
return cls._encode(_pydantic_model_dump(JobPydantic, var), type_=DAT.BASE_JOB)
diff --git a/setup.cfg b/setup.cfg
index 1bbf5fb8f1..cfde203f43 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -133,11 +133,7 @@ install_requires =
pendulum>=2.0,<3.0
pluggy>=1.0
psutil>=4.2.0
- # We should bump it to at least pydantic>=2.3.0 when we prepare Airflow 2.8.0 release
- # we keep Pydantic < 1 for compatibility with packages that depend on Pydantic 1
- # We should also remove then `filterwarning` for pydantic from airflow/configuration.py
- # and # Pydantic v1 check in airflow/serialization/serde.py
- pydantic>=1.10.0
+ pydantic>=2.3.0
pygments>=2.0.1
pyjwt>=2.0.0
python-daemon>=3.0.0
(airflow) 28/37: added Topic params for schema_settings and message_retention_duration. (#35767)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 08785f50f15a419e91a3eec03a60d9d0b9173343
Author: dmedora <21...@users.noreply.github.com>
AuthorDate: Wed Nov 22 01:21:59 2023 -0800
added Topic params for schema_settings and message_retention_duration. (#35767)
---
airflow/providers/google/cloud/hooks/pubsub.py | 10 ++++++++++
airflow/providers/google/cloud/operators/pubsub.py | 7 +++++++
tests/providers/google/cloud/hooks/test_pubsub.py | 9 ++++++++-
tests/providers/google/cloud/operators/test_pubsub.py | 4 ++++
4 files changed, 29 insertions(+), 1 deletion(-)
diff --git a/airflow/providers/google/cloud/hooks/pubsub.py b/airflow/providers/google/cloud/hooks/pubsub.py
index 1573f32841..0f6e5fcff8 100644
--- a/airflow/providers/google/cloud/hooks/pubsub.py
+++ b/airflow/providers/google/cloud/hooks/pubsub.py
@@ -57,6 +57,7 @@ if TYPE_CHECKING:
PushConfig,
ReceivedMessage,
RetryPolicy,
+ SchemaSettings,
)
@@ -182,6 +183,8 @@ class PubSubHook(GoogleBaseHook):
labels: dict[str, str] | None = None,
message_storage_policy: dict | MessageStoragePolicy = None,
kms_key_name: str | None = None,
+ schema_settings: dict | SchemaSettings = None,
+ message_retention_duration: str | None = None,
retry: Retry | _MethodDefault = DEFAULT,
timeout: float | None = None,
metadata: Sequence[tuple[str, str]] = (),
@@ -206,6 +209,11 @@ class PubSubHook(GoogleBaseHook):
to be used to protect access to messages published on this topic.
The expected format is
``projects/*/locations/*/keyRings/*/cryptoKeys/*``.
+ :param schema_settings: (Optional) Settings for validating messages published against an
+ existing schema. The expected format is ``projects/*/schemas/*``.
+ :param message_retention_duration: (Optional) Indicates the minimum duration to retain a
+ message after it is published to the topic. The expected format is a duration in
+ seconds with up to nine fractional digits, ending with 's'. Example: "3.5s".
:param retry: (Optional) A retry object used to retry requests.
If None is specified, requests will not be retried.
:param timeout: (Optional) The amount of time, in seconds, to wait for the request
@@ -228,6 +236,8 @@ class PubSubHook(GoogleBaseHook):
"labels": labels,
"message_storage_policy": message_storage_policy,
"kms_key_name": kms_key_name,
+ "schema_settings": schema_settings,
+ "message_retention_duration": message_retention_duration,
},
retry=retry,
timeout=timeout,
diff --git a/airflow/providers/google/cloud/operators/pubsub.py b/airflow/providers/google/cloud/operators/pubsub.py
index b8c90be0fe..3751e9a371 100644
--- a/airflow/providers/google/cloud/operators/pubsub.py
+++ b/airflow/providers/google/cloud/operators/pubsub.py
@@ -35,6 +35,7 @@ from google.cloud.pubsub_v1.types import (
PushConfig,
ReceivedMessage,
RetryPolicy,
+ SchemaSettings,
)
from airflow.providers.google.cloud.hooks.pubsub import PubSubHook
@@ -130,6 +131,8 @@ class PubSubCreateTopicOperator(GoogleCloudBaseOperator):
labels: dict[str, str] | None = None,
message_storage_policy: dict | MessageStoragePolicy = None,
kms_key_name: str | None = None,
+ schema_settings: dict | SchemaSettings = None,
+ message_retention_duration: str | None = None,
retry: Retry | _MethodDefault = DEFAULT,
timeout: float | None = None,
metadata: Sequence[tuple[str, str]] = (),
@@ -144,6 +147,8 @@ class PubSubCreateTopicOperator(GoogleCloudBaseOperator):
self.labels = labels
self.message_storage_policy = message_storage_policy
self.kms_key_name = kms_key_name
+ self.schema_settings = schema_settings
+ self.message_retention_duration = message_retention_duration
self.retry = retry
self.timeout = timeout
self.metadata = metadata
@@ -163,6 +168,8 @@ class PubSubCreateTopicOperator(GoogleCloudBaseOperator):
labels=self.labels,
message_storage_policy=self.message_storage_policy,
kms_key_name=self.kms_key_name,
+ schema_settings=self.schema_settings,
+ message_retention_duration=self.message_retention_duration,
retry=self.retry,
timeout=self.timeout,
metadata=self.metadata,
diff --git a/tests/providers/google/cloud/hooks/test_pubsub.py b/tests/providers/google/cloud/hooks/test_pubsub.py
index fe25df6562..04c53cdae2 100644
--- a/tests/providers/google/cloud/hooks/test_pubsub.py
+++ b/tests/providers/google/cloud/hooks/test_pubsub.py
@@ -103,7 +103,14 @@ class TestPubSubHook:
create_method = mock_service.return_value.create_topic
self.pubsub_hook.create_topic(project_id=TEST_PROJECT, topic=TEST_TOPIC)
create_method.assert_called_once_with(
- request=dict(name=EXPANDED_TOPIC, labels=LABELS, message_storage_policy=None, kms_key_name=None),
+ request=dict(
+ name=EXPANDED_TOPIC,
+ labels=LABELS,
+ message_storage_policy=None,
+ kms_key_name=None,
+ schema_settings=None,
+ message_retention_duration=None,
+ ),
retry=DEFAULT,
timeout=None,
metadata=(),
diff --git a/tests/providers/google/cloud/operators/test_pubsub.py b/tests/providers/google/cloud/operators/test_pubsub.py
index f2a9fb0e9b..0598214f10 100644
--- a/tests/providers/google/cloud/operators/test_pubsub.py
+++ b/tests/providers/google/cloud/operators/test_pubsub.py
@@ -59,6 +59,8 @@ class TestPubSubTopicCreateOperator:
labels=None,
message_storage_policy=None,
kms_key_name=None,
+ schema_settings=None,
+ message_retention_duration=None,
retry=DEFAULT,
timeout=None,
metadata=(),
@@ -79,6 +81,8 @@ class TestPubSubTopicCreateOperator:
labels=None,
message_storage_policy=None,
kms_key_name=None,
+ schema_settings=None,
+ message_retention_duration=None,
retry=DEFAULT,
timeout=None,
metadata=(),
(airflow) 11/37: Extend task context logging support for remote logging using Elasticsearch (#32977)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit d9cfdd8131d797735b406554674282ea0f8dcbae
Author: Pankaj Koti <pa...@gmail.com>
AuthorDate: Tue Nov 21 11:55:03 2023 +0530
Extend task context logging support for remote logging using Elasticsearch (#32977)
* Extend task context logging support for remote logging using Elasticsearch
With the addition of task context logging feature in PR #32646,
this PR extends the feature to Elasticsearch when is it set as
remote logging store. Here, backward compatibility is ensured for
older versions of Airflow that do not have the feature included
in Airflow Core.
* update ensure_ti
---------
Co-authored-by: Daniel Standish <15...@users.noreply.github.com>
---
.../providers/elasticsearch/log/es_task_handler.py | 46 +++++++++++++++++++---
1 file changed, 41 insertions(+), 5 deletions(-)
diff --git a/airflow/providers/elasticsearch/log/es_task_handler.py b/airflow/providers/elasticsearch/log/es_task_handler.py
index 79f9ad0b41..1e8c75b7e3 100644
--- a/airflow/providers/elasticsearch/log/es_task_handler.py
+++ b/airflow/providers/elasticsearch/log/es_task_handler.py
@@ -34,7 +34,7 @@ import pendulum
from elasticsearch.exceptions import NotFoundError
from airflow.configuration import conf
-from airflow.exceptions import AirflowProviderDeprecationWarning
+from airflow.exceptions import AirflowException, AirflowProviderDeprecationWarning
from airflow.models.dagrun import DagRun
from airflow.providers.elasticsearch.log.es_json_formatter import ElasticsearchJSONFormatter
from airflow.providers.elasticsearch.log.es_response import ElasticSearchResponse, Hit
@@ -46,7 +46,8 @@ from airflow.utils.session import create_session
if TYPE_CHECKING:
from datetime import datetime
- from airflow.models.taskinstance import TaskInstance
+ from airflow.models.taskinstance import TaskInstance, TaskInstanceKey
+
LOG_LINE_DEFAULTS = {"exc_text": "", "stack_info": ""}
# Elasticsearch hosted log type
@@ -84,6 +85,32 @@ def get_es_kwargs_from_config() -> dict[str, Any]:
return kwargs_dict
+def _ensure_ti(ti: TaskInstanceKey | TaskInstance, session) -> TaskInstance:
+ """Given TI | TIKey, return a TI object.
+
+ Will raise exception if no TI is found in the database.
+ """
+ from airflow.models.taskinstance import TaskInstance, TaskInstanceKey
+
+ if not isinstance(ti, TaskInstanceKey):
+ return ti
+ val = (
+ session.query(TaskInstance)
+ .filter(
+ TaskInstance.task_id == ti.task_id,
+ TaskInstance.dag_id == ti.dag_id,
+ TaskInstance.run_id == ti.run_id,
+ TaskInstance.map_index == ti.map_index,
+ )
+ .one_or_none()
+ )
+ if isinstance(val, TaskInstance):
+ val._try_number = ti.try_number
+ return val
+ else:
+ raise AirflowException(f"Could not find TaskInstance for {ti}")
+
+
class ElasticsearchTaskHandler(FileTaskHandler, ExternalLoggingMixin, LoggingMixin):
"""
ElasticsearchTaskHandler is a python log handler that reads logs from Elasticsearch.
@@ -182,8 +209,12 @@ class ElasticsearchTaskHandler(FileTaskHandler, ExternalLoggingMixin, LoggingMix
return host
- def _render_log_id(self, ti: TaskInstance, try_number: int) -> str:
+ def _render_log_id(self, ti: TaskInstance | TaskInstanceKey, try_number: int) -> str:
+ from airflow.models.taskinstance import TaskInstanceKey
+
with create_session() as session:
+ if isinstance(ti, TaskInstanceKey):
+ ti = _ensure_ti(ti, session)
dag_run = ti.get_dagrun(session=session)
if USE_PER_RUN_LOG_ID:
log_id_template = dag_run.get_log_template(session=session).elasticsearch_id
@@ -377,11 +408,13 @@ class ElasticsearchTaskHandler(FileTaskHandler, ExternalLoggingMixin, LoggingMix
setattr(record, self.offset_field, int(time.time() * (10**9)))
self.handler.emit(record)
- def set_context(self, ti: TaskInstance, **kwargs) -> None:
+ def set_context(self, ti: TaskInstance, *, identifier: str | None = None) -> None:
"""
Provide task_instance context to airflow task handler.
:param ti: task instance object
+ :param identifier: if set, identifies the Airflow component which is relaying logs from
+ exceptional scenarios related to the task instance
"""
is_trigger_log_context = getattr(ti, "is_trigger_log_context", None)
is_ti_raw = getattr(ti, "raw", None)
@@ -410,7 +443,10 @@ class ElasticsearchTaskHandler(FileTaskHandler, ExternalLoggingMixin, LoggingMix
self.handler.setLevel(self.level)
self.handler.setFormatter(self.formatter)
else:
- super().set_context(ti)
+ if getattr(self, "supports_task_context_logging", False):
+ super().set_context(ti, identifier=identifier)
+ else:
+ super().set_context(ti)
self.context_set = True
def close(self) -> None:
(airflow) 13/37: Improve ownership fixing for Breeze (#35759)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 26d5e3f4e72c349b9f041fbb11f2e4791e6da1db
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Tue Nov 21 10:30:50 2023 +0100
Improve ownership fixing for Breeze (#35759)
When we run airflow package preparation in docker containers and
asset compilation, we use docker containers - similarly as breeze
CI image, those constainers use root user in container. This means
that the files created by those containers are owned by root on
Linux - which means that we should change ownership of these files
after they were generated.
Used the opportunity to rewrtie "fix_ownership" part to Python
and remove bash scripts used for it so far.
---
.github/actions/build-ci-images/action.yml | 4 --
.github/actions/build-prod-images/action.yml | 4 --
.github/actions/post_tests_failure/action.yml | 3 -
.github/actions/post_tests_success/action.yml | 3 -
.github/workflows/build-images.yml | 3 -
.github/workflows/ci.yml | 53 --------------
.github/workflows/release_dockerhub_image.yml | 3 -
Dockerfile.ci | 2 -
...-root-ownership-after-exiting-docker-command.md | 60 ++++++++++++++++
.../airflow_breeze/commands/developer_commands.py | 7 ++
.../commands/release_management_commands.py | 21 ++++++
.../airflow_breeze/commands/testing_commands.py | 5 ++
.../airflow_breeze/utils/docker_command_utils.py | 17 +++--
dev/breeze/src/airflow_breeze/utils/parallel.py | 6 ++
dev/breeze/src/airflow_breeze/utils/run_utils.py | 2 +-
scripts/docker/entrypoint_ci.sh | 2 -
scripts/in_container/_in_container_utils.sh | 40 -----------
scripts/in_container/run_fix_ownership.py | 84 ++++++++++++++++++++++
scripts/in_container/run_fix_ownership.sh | 21 ------
.../in_container/run_prepare_airflow_packages.py | 12 ++--
20 files changed, 201 insertions(+), 151 deletions(-)
diff --git a/.github/actions/build-ci-images/action.yml b/.github/actions/build-ci-images/action.yml
index d8c0216bb4..f950c53b4e 100644
--- a/.github/actions/build-ci-images/action.yml
+++ b/.github/actions/build-ci-images/action.yml
@@ -45,7 +45,3 @@ runs:
name: source-constraints
path: ./files/constraints-*/constraints-*.txt
retention-days: 7
- - name: "Fix ownership"
- shell: bash
- run: breeze ci fix-ownership
- if: always()
diff --git a/.github/actions/build-prod-images/action.yml b/.github/actions/build-prod-images/action.yml
index 7572153169..5fdbb795c4 100644
--- a/.github/actions/build-prod-images/action.yml
+++ b/.github/actions/build-prod-images/action.yml
@@ -72,7 +72,3 @@ runs:
env:
COMMIT_SHA: ${{ github.sha }}
if: ${{ inputs.build-provider-packages != 'true' }}
- - name: "Fix ownership"
- shell: bash
- run: breeze ci fix-ownership
- if: always()
diff --git a/.github/actions/post_tests_failure/action.yml b/.github/actions/post_tests_failure/action.yml
index 96e43bbe0e..5b51db97ed 100644
--- a/.github/actions/post_tests_failure/action.yml
+++ b/.github/actions/post_tests_failure/action.yml
@@ -39,6 +39,3 @@ runs:
name: container-logs-${{env.JOB_ID}}
path: "./files/other_logs*"
retention-days: 7
- - name: "Fix ownership"
- shell: bash
- run: breeze ci fix-ownership
diff --git a/.github/actions/post_tests_success/action.yml b/.github/actions/post_tests_success/action.yml
index 1325c5dde3..ac1e4fb8d2 100644
--- a/.github/actions/post_tests_success/action.yml
+++ b/.github/actions/post_tests_success/action.yml
@@ -40,6 +40,3 @@ runs:
name: coverage-${{env.JOB_ID}}
flags: python-${{env.PYTHON_MAJOR_MINOR_VERSION}},${{env.BACKEND}}-${{env.BACKEND_VERSION}}
directory: "./files/coverage-reposts/"
- - name: "Fix ownership"
- shell: bash
- run: breeze ci fix-ownership
diff --git a/.github/workflows/build-images.yml b/.github/workflows/build-images.yml
index 82498ebde0..b29d49de17 100644
--- a/.github/workflows/build-images.yml
+++ b/.github/workflows/build-images.yml
@@ -362,6 +362,3 @@ jobs:
- name: "Stop ARM instance"
run: ./scripts/ci/images/ci_stop_arm_instance.sh
if: always()
- - name: "Fix ownership"
- run: breeze ci fix-ownership
- if: always()
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index da4dbfde53..45e0a9fca0 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -260,11 +260,6 @@ jobs:
if: >
matrix.platform == 'linux/amd64' && needs.build-info.outputs.canary-run == 'true'
&& needs.build-info.outputs.default-branch == 'main'
- - name: "Fix ownership"
- run: breeze ci fix-ownership
- if: >
- always() && needs.build-info.outputs.canary-run == 'true'
- && needs.build-info.outputs.default-branch == 'main'
# Check that after earlier cache push, breeze command will build quickly
check-that-image-builds-quickly:
timeout-minutes: 5
@@ -291,9 +286,6 @@ jobs:
uses: ./.github/actions/breeze
- name: "Check that image builds quickly"
run: breeze shell --max-time 120
- - name: "Fix ownership"
- run: breeze ci fix-ownership
- if: always()
build-ci-images:
timeout-minutes: 80
@@ -385,9 +377,6 @@ jobs:
name: constraints
path: ./files/constraints-*/constraints-*.txt
retention-days: 7
- - name: "Fix ownership"
- run: breeze ci fix-ownership
- if: always()
build-prod-images:
timeout-minutes: 80
@@ -603,9 +592,6 @@ jobs:
env:
PYTHON_VERSIONS: ${{ needs.build-info.outputs.python-versions-list-as-string }}
DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
- - name: "Fix ownership"
- run: breeze ci fix-ownership
- if: always()
static-checks:
timeout-minutes: 45
@@ -645,9 +631,6 @@ jobs:
SKIP_GROUP_OUTPUT: "true"
DEFAULT_BRANCH: ${{ needs.build-info.outputs.default-branch }}
RUFF_FORMAT: "github"
- - name: "Fix ownership"
- run: breeze ci fix-ownership
- if: always()
# Those checks are run if no image needs to be built for checks. This is for simple changes that
# Do not touch any of the python code or any of the important files that might require building
@@ -701,9 +684,6 @@ jobs:
SKIP_IMAGE_PRE_COMMITS: "true"
SKIP: ${{ needs.build-info.outputs.skip-pre-commits }}
COLUMNS: "250"
- - name: "Fix ownership"
- run: breeze ci fix-ownership
- if: always()
build-docs:
timeout-minutes: 60
@@ -768,9 +748,6 @@ jobs:
- name: "Upload documentation to AWS S3"
if: needs.build-info.outputs.canary-run == 'true' && needs.build-info.outputs.default-branch == 'main'
run: aws s3 sync --delete ./files/documentation s3://apache-airflow-docs
- - name: "Fix ownership"
- run: breeze ci fix-ownership
- if: always()
spellcheck-docs:
timeout-minutes: 60
@@ -802,9 +779,6 @@ jobs:
- name: "Spellcheck docs"
run: >
breeze build-docs ${{ needs.build-info.outputs.docs-list-as-string }} --spellcheck-only
- - name: "Fix ownership"
- run: breeze ci fix-ownership
- if: always()
prepare-test-provider-packages-wheel:
timeout-minutes: 80
@@ -857,9 +831,6 @@ jobs:
if: needs.build-info.outputs.affected-providers-list-as-string != ''
env:
SKIP_CONSTRAINTS: "${{ needs.build-info.outputs.upgrade-to-newer-dependencies }}"
- - name: "Fix ownership"
- run: breeze ci fix-ownership
- if: always()
provider-airflow-compatibility-check:
timeout-minutes: 80
@@ -925,9 +896,6 @@ jobs:
if: >
needs.build-info.outputs.affected-providers-list-as-string != '' &&
needs.build-info.outputs.affected-providers-list-as-string != 'openlineage'
- - name: "Fix ownership"
- run: breeze ci fix-ownership
- if: always()
prepare-install-provider-packages-sdist:
timeout-minutes: 80
@@ -974,9 +942,6 @@ jobs:
breeze release-management install-provider-packages
--package-format sdist --use-airflow-version sdist --run-in-parallel
if: needs.build-info.outputs.affected-providers-list-as-string != ''
- - name: "Fix ownership"
- run: breeze ci fix-ownership
- if: always()
test-airflow-release-commands:
@@ -1012,9 +977,6 @@ jobs:
- name: "Check Airflow release process command"
run: |
breeze release-management start-release --release-candidate 2.4.3rc1 --previous-release 2.4.2 -a y
- - name: "Fix ownership"
- run: breeze ci fix-ownership
- if: always()
tests-helm:
timeout-minutes: 80
@@ -1717,9 +1679,6 @@ jobs:
env:
PYTHON_VERSIONS: ${{ needs.build-info.outputs.python-versions-list-as-string }}
DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
- - name: "Fix ownership"
- run: breeze ci fix-ownership
- if: always()
test-docker-compose-quick-start:
@@ -1746,9 +1705,6 @@ jobs:
pull-image-type: 'PROD'
- name: "Test docker-compose quick start"
run: breeze testing docker-compose-tests
- - name: "Fix ownership"
- run: breeze ci fix-ownership
- if: always()
tests-kubernetes:
timeout-minutes: 240
@@ -1816,9 +1772,6 @@ jobs:
- name: "Delete clusters just in case they are left"
run: breeze k8s delete-cluster --all
if: always()
- - name: "Fix ownership"
- run: breeze ci fix-ownership
- if: always()
update-constraints:
permissions:
@@ -1974,9 +1927,6 @@ jobs:
- name: "Stop ARM instance"
run: ./scripts/ci/images/ci_stop_arm_instance.sh
if: always() && matrix.platform == 'linux/arm64'
- - name: "Fix ownership"
- run: breeze ci fix-ownership
- if: always()
# This is only a check if ARM images are successfully building when committer runs PR from
# Apache repository. This is needed in case you want to fix failing cache job in "canary" run
@@ -2033,6 +1983,3 @@ jobs:
- name: "Stop ARM instance"
run: ./scripts/ci/images/ci_stop_arm_instance.sh
if: always()
- - name: "Fix ownership"
- run: breeze ci fix-ownership
- if: always()
diff --git a/.github/workflows/release_dockerhub_image.yml b/.github/workflows/release_dockerhub_image.yml
index 02d7d6fcc0..6889539387 100644
--- a/.github/workflows/release_dockerhub_image.yml
+++ b/.github/workflows/release_dockerhub_image.yml
@@ -150,6 +150,3 @@ jobs:
- name: "Docker logout"
run: docker logout
if: always()
- - name: "Fix ownership"
- run: breeze ci fix-ownership
- if: always()
diff --git a/Dockerfile.ci b/Dockerfile.ci
index 45a1166070..81a9d424a0 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -998,8 +998,6 @@ if [[ ${SKIP_ENVIRONMENT_INITIALIZATION=} != "true" ]]; then
touch /usr/lib/google-cloud-sdk/bin/gcloud
ln -s -f /usr/bin/gcloud /usr/lib/google-cloud-sdk/bin/gcloud
- in_container_fix_ownership
-
if [[ ${SKIP_SSH_SETUP="false"} == "false" ]]; then
# Set up ssh keys
echo 'yes' | ssh-keygen -t rsa -C your_email@youremail.com -m PEM -P '' -f ~/.ssh/id_rsa \
diff --git a/dev/breeze/doc/adr/0014-fix-root-ownership-after-exiting-docker-command.md b/dev/breeze/doc/adr/0014-fix-root-ownership-after-exiting-docker-command.md
new file mode 100644
index 0000000000..5ca0c4681a
--- /dev/null
+++ b/dev/breeze/doc/adr/0014-fix-root-ownership-after-exiting-docker-command.md
@@ -0,0 +1,60 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+
+- [14. Fix root ownership after exiting docker command](#14-fix-root-ownership-after-exiting-docker-command)
+ - [Status](#status)
+ - [Context](#context)
+ - [Decision](#decision)
+ - [Consequences](#consequences)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+# 14. Fix root ownership after exiting docker command
+
+Date: 2023-11-20
+
+## Status
+
+Accepted
+
+Builds on [6. Using root user and fixing ownership for-ci-container](0006-using-root-user-and-fixing-ownership-for-ci-container.md)
+
+## Context
+
+As discussed in [6. Using root user and fixing ownership for-ci-container](0006-using-root-user-and-fixing-ownership-for-ci-container.md)
+we run Breeze CI container as root user. We used to use TRAP to fix the ownership of files created in
+the container by root user. However, this is not fool-proof and using TRAP has its caveats (for example
+it can be overridden by another TRAP command). Also TRAP might be invalidated by signal handling
+or abrupt killing the docker container. Running the cleanup command after docker command completed
+seems to do be much more robust solution.
+
+## Decision
+
+Instead of running the command as TRAP when exiting we simply attempt to clean the ownership whenever
+we exit the container using `breeze`, `breeze shell`, `tests` or `breeze start-airflow` commands that run
+commands in the CI image.
+
+## Consequences
+
+Users running Breeze on Linux will have less problems with root owned files and we can also remove
+dedicated `ci fix-ownership` command in CI.
diff --git a/dev/breeze/src/airflow_breeze/commands/developer_commands.py b/dev/breeze/src/airflow_breeze/commands/developer_commands.py
index 7f4b38d27b..a8c257a2df 100644
--- a/dev/breeze/src/airflow_breeze/commands/developer_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/developer_commands.py
@@ -84,6 +84,7 @@ from airflow_breeze.utils.console import get_console
from airflow_breeze.utils.custom_param_types import BetterChoice
from airflow_breeze.utils.docker_command_utils import (
check_docker_resources,
+ fix_ownership_using_docker,
get_env_variables_for_docker_commands,
get_extra_docker_flags,
perform_environment_checks,
@@ -259,6 +260,7 @@ def shell(
run_db_tests_only=run_db_tests_only,
skip_db_tests=skip_db_tests,
)
+ fix_ownership_using_docker()
sys.exit(result.returncode)
@@ -382,6 +384,7 @@ def start_airflow(
standalone_dag_processor=standalone_dag_processor,
database_isolation=database_isolation,
)
+ fix_ownership_using_docker()
sys.exit(result.returncode)
@@ -426,6 +429,7 @@ def build_docs(
Build documents.
"""
perform_environment_checks()
+ fix_ownership_using_docker()
cleanup_python_generated_files()
params = BuildCiParams(
github_repository=github_repository, python=DEFAULT_PYTHON_MAJOR_MINOR_VERSION, builder=builder
@@ -460,6 +464,7 @@ def build_docs(
*doc_builder.args_doc_builder,
]
process = run_command(cmd, text=True, env=env, check=False)
+ fix_ownership_using_docker()
if process.returncode == 0:
get_console().print(
"[info]Start the webserver in breeze and view the built docs at http://localhost:28080/docs/[/]"
@@ -639,6 +644,7 @@ def static_checks(
text=True,
env=env,
)
+ fix_ownership_using_docker()
if static_checks_result.returncode != 0:
if os.environ.get("CI"):
get_console().print("\n[error]This error means that you have to fix the issues listed above:[/]")
@@ -756,6 +762,7 @@ def enter_shell(**kwargs) -> RunCommandResult:
"""
perform_environment_checks()
+ fix_ownership_using_docker()
cleanup_python_generated_files()
if read_from_cache_file("suppress_asciiart") is None:
get_console().print(ASCIIART, style=ASCIIART_STYLE)
diff --git a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
index c7a29b5aa2..732475d5df 100644
--- a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
@@ -96,6 +96,7 @@ from airflow_breeze.utils.console import MessageType, Output, get_console
from airflow_breeze.utils.custom_param_types import BetterChoice
from airflow_breeze.utils.docker_command_utils import (
check_remote_ghcr_io_commands,
+ fix_ownership_using_docker,
get_env_variables_for_docker_commands,
get_extra_docker_flags,
perform_environment_checks,
@@ -220,6 +221,7 @@ NODE_BUILD_IMAGE_TAG = "node:21.2.0-bookworm-slim"
def _compile_assets_in_docker():
clean_www_assets()
+ get_console().print("[info]Compiling assets in docker container\n")
result = run_command(
[
"docker",
@@ -242,8 +244,14 @@ def _compile_assets_in_docker():
get_console().print("[error]Error compiling assets[/]")
get_console().print(result.stdout)
get_console().print(result.stderr)
+ fix_ownership_using_docker()
sys.exit(result.returncode)
+ get_console().print("[success]compiled assets in docker container\n")
+ get_console().print("[info]Fixing ownership of compiled assets\n")
+ fix_ownership_using_docker()
+ get_console().print("[success]Fixing ownership of compiled assets\n")
+
@release_management.command(
name="prepare-airflow-package",
@@ -267,6 +275,7 @@ def prepare_airflow_packages(
use_container_for_assets_compilation: bool,
):
perform_environment_checks()
+ fix_ownership_using_docker()
cleanup_python_generated_files()
get_console().print("[info]Compiling assets\n")
from sys import platform
@@ -303,6 +312,9 @@ def prepare_airflow_packages(
check=True,
)
get_console().print("[success]Successfully prepared Airflow package!\n\n")
+ get_console().print("\n[info]Cleaning ownership of generated files\n")
+ fix_ownership_using_docker()
+ get_console().print("\n[success]Cleaned ownership of generated files\n")
def provider_action_summary(description: str, message_type: MessageType, packages: list[str]):
@@ -371,6 +383,8 @@ def prepare_provider_documentation(
update_release_notes,
)
+ perform_environment_checks()
+ fix_ownership_using_docker()
cleanup_python_generated_files()
if not provider_packages:
provider_packages = get_available_packages()
@@ -516,6 +530,7 @@ def prepare_provider_packages(
provider_packages: tuple[str, ...],
):
perform_environment_checks()
+ fix_ownership_using_docker()
cleanup_python_generated_files()
packages_list = get_packages_list_to_act_on(package_list_file, provider_packages)
if not skip_tag_check:
@@ -601,6 +616,7 @@ def run_generate_constraints(
output=output,
output_outside_the_group=True,
)
+ fix_ownership_using_docker()
return (
generate_constraints_result.returncode,
f"Constraints {shell_params.airflow_constraints_mode}:{shell_params.python}",
@@ -687,6 +703,7 @@ def generate_constraints(
):
perform_environment_checks()
check_remote_ghcr_io_commands()
+ fix_ownership_using_docker()
cleanup_python_generated_files()
if debug and run_in_parallel:
get_console().print("\n[error]Cannot run --debug and --run-in-parallel at the same time[/]\n")
@@ -854,6 +871,7 @@ def install_provider_packages(
include_success_outputs: bool,
):
perform_environment_checks()
+ fix_ownership_using_docker()
cleanup_python_generated_files()
shell_params = ShellParams(
mount_sources=MOUNT_SELECTED,
@@ -944,6 +962,7 @@ def install_provider_packages(
debug=debug,
output_outside_the_group=True,
)
+ fix_ownership_using_docker()
sys.exit(result_command.returncode)
@@ -977,6 +996,7 @@ def verify_provider_packages(
get_console().print("Forcing use_packages_from_dist as installing selected_providers is set")
use_packages_from_dist = True
perform_environment_checks()
+ fix_ownership_using_docker()
cleanup_python_generated_files()
shell_params = ShellParams(
mount_sources=MOUNT_SELECTED,
@@ -1000,6 +1020,7 @@ def verify_provider_packages(
debug=debug,
output_outside_the_group=True,
)
+ fix_ownership_using_docker()
sys.exit(result_command.returncode)
diff --git a/dev/breeze/src/airflow_breeze/commands/testing_commands.py b/dev/breeze/src/airflow_breeze/commands/testing_commands.py
index 66ffb593c3..fdd712087b 100644
--- a/dev/breeze/src/airflow_breeze/commands/testing_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/testing_commands.py
@@ -70,6 +70,7 @@ from airflow_breeze.utils.common_options import (
from airflow_breeze.utils.console import Output, get_console
from airflow_breeze.utils.custom_param_types import BetterChoice
from airflow_breeze.utils.docker_command_utils import (
+ fix_ownership_using_docker,
get_env_variables_for_docker_commands,
perform_environment_checks,
remove_docker_networks,
@@ -592,6 +593,7 @@ def _run_test_command(
parallel_test_types_list=test_list,
)
rebuild_or_pull_ci_image_if_needed(command_params=exec_shell_params)
+ fix_ownership_using_docker()
cleanup_python_generated_files()
perform_environment_checks()
if run_in_parallel:
@@ -695,6 +697,7 @@ def integration_tests(
github_repository=github_repository,
enable_coverage=enable_coverage,
)
+ fix_ownership_using_docker()
cleanup_python_generated_files()
perform_environment_checks()
returncode, _ = _run_test(
@@ -752,6 +755,7 @@ def helm_tests(
if helm_test_package != "all":
env_variables["HELM_TEST_PACKAGE"] = helm_test_package
perform_environment_checks()
+ fix_ownership_using_docker()
cleanup_python_generated_files()
pytest_args = generate_args_for_pytest(
test_type="Helm",
@@ -769,4 +773,5 @@ def helm_tests(
)
cmd = ["docker", "compose", "run", "--service-ports", "--rm", "airflow", *pytest_args, *extra_pytest_args]
result = run_command(cmd, env=env_variables, check=False, output_outside_the_group=True)
+ fix_ownership_using_docker()
sys.exit(result.returncode)
diff --git a/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py b/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
index e936773162..36c976b2fe 100644
--- a/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
+++ b/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
@@ -41,8 +41,10 @@ except ImportError:
from airflow_breeze.branch_defaults import AIRFLOW_BRANCH
from airflow_breeze.global_constants import (
ALLOWED_CELERY_BROKERS,
+ ALLOWED_DEBIAN_VERSIONS,
ALLOWED_PACKAGE_FORMATS,
APACHE_AIRFLOW_GITHUB_REPOSITORY,
+ DEFAULT_PYTHON_MAJOR_MINOR_VERSION,
FLOWER_HOST_PORT,
MIN_DOCKER_COMPOSE_VERSION,
MIN_DOCKER_VERSION,
@@ -763,8 +765,15 @@ LABEL description="test warmup image"
)
+OWNERSHIP_CLEANUP_DOCKER_TAG = (
+ f"python:{DEFAULT_PYTHON_MAJOR_MINOR_VERSION}-slim-{ALLOWED_DEBIAN_VERSIONS[0]}"
+)
+
+
def fix_ownership_using_docker():
- perform_environment_checks()
+ if get_host_os() != "linux":
+ # no need to even attempt fixing ownership on MacOS/Windows
+ return
shell_params = find_available_ci_image(
github_repository=APACHE_AIRFLOW_GITHUB_REPOSITORY,
)
@@ -775,10 +784,8 @@ def fix_ownership_using_docker():
"run",
"-t",
*extra_docker_flags,
- "--pull",
- "never",
- shell_params.airflow_image_name_with_tag,
- "/opt/airflow/scripts/in_container/run_fix_ownership.sh",
+ OWNERSHIP_CLEANUP_DOCKER_TAG,
+ "/opt/airflow/scripts/in_container/run_fix_ownership.py",
]
run_command(cmd, text=True, env=env, check=False)
diff --git a/dev/breeze/src/airflow_breeze/utils/parallel.py b/dev/breeze/src/airflow_breeze/utils/parallel.py
index ad7e197d1c..eca700a18a 100644
--- a/dev/breeze/src/airflow_breeze/utils/parallel.py
+++ b/dev/breeze/src/airflow_breeze/utils/parallel.py
@@ -443,9 +443,15 @@ def check_async_run_results(
try:
if errors:
get_console().print("\n[error]There were errors when running some tasks. Quitting.[/]\n")
+ from airflow_breeze.utils.docker_command_utils import fix_ownership_using_docker
+
+ fix_ownership_using_docker()
sys.exit(1)
else:
get_console().print(f"\n[success]{success}[/]\n")
+ from airflow_breeze.utils.docker_command_utils import fix_ownership_using_docker
+
+ fix_ownership_using_docker()
finally:
if not skip_cleanup:
for output in outputs:
diff --git a/dev/breeze/src/airflow_breeze/utils/run_utils.py b/dev/breeze/src/airflow_breeze/utils/run_utils.py
index f9415394b6..2553e75ce0 100644
--- a/dev/breeze/src/airflow_breeze/utils/run_utils.py
+++ b/dev/breeze/src/airflow_breeze/utils/run_utils.py
@@ -457,7 +457,7 @@ def clean_www_assets():
WWW_ASSET_HASH_FILE.unlink(missing_ok=True)
shutil.rmtree(WWW_NODE_MODULES_DIR, ignore_errors=True)
shutil.rmtree(WWW_STATIC_DIST_DIR, ignore_errors=True)
- get_console().print("[info]Cleaned www assets[/]")
+ get_console().print("[success]Cleaned www assets[/]")
def run_compile_www_assets(
diff --git a/scripts/docker/entrypoint_ci.sh b/scripts/docker/entrypoint_ci.sh
index 1de27480d1..d16134b482 100755
--- a/scripts/docker/entrypoint_ci.sh
+++ b/scripts/docker/entrypoint_ci.sh
@@ -296,8 +296,6 @@ if [[ ${SKIP_ENVIRONMENT_INITIALIZATION=} != "true" ]]; then
touch /usr/lib/google-cloud-sdk/bin/gcloud
ln -s -f /usr/bin/gcloud /usr/lib/google-cloud-sdk/bin/gcloud
- in_container_fix_ownership
-
if [[ ${SKIP_SSH_SETUP="false"} == "false" ]]; then
# Set up ssh keys
echo 'yes' | ssh-keygen -t rsa -C your_email@youremail.com -m PEM -P '' -f ~/.ssh/id_rsa \
diff --git a/scripts/in_container/_in_container_utils.sh b/scripts/in_container/_in_container_utils.sh
index b3606973c4..a3e09e0f1e 100644
--- a/scripts/in_container/_in_container_utils.sh
+++ b/scripts/in_container/_in_container_utils.sh
@@ -56,46 +56,6 @@ function in_container_script_start() {
fi
}
-#
-# Fixes ownership of files generated in container - if they are owned by root, they will be owned by
-# The host user. Only needed if the host is Linux - on Mac, ownership of files is automatically
-# changed to the Host user via osxfs filesystem
-#
-function in_container_fix_ownership() {
- if [[ ${HOST_OS:=} == "linux" ]]; then
- if [[ ${DOCKER_IS_ROOTLESS=} == "true" ]]; then
- echo "${COLOR_YELLOW}Skip fixing ownership of generated files: Docker is rootless${COLOR_RESET}"
- return
- fi
- DIRECTORIES_TO_FIX=(
- "/dist"
- "/files"
- "${AIRFLOW_SOURCES}/logs"
- "${AIRFLOW_SOURCES}/docs"
- "${AIRFLOW_SOURCES}/dags"
- "${AIRFLOW_SOURCES}/airflow/"
- "${AIRFLOW_SOURCES}/constraints/"
- "${AIRFLOW_SOURCES}/images/"
- "${AIRFLOW_SOURCES}/.mypy_cache/"
- "${AIRFLOW_SOURCES}/dev/"
- )
- count_matching=$(find "${DIRECTORIES_TO_FIX[@]}" -mindepth 1 -user root -printf . 2>/dev/null | wc -m || true)
- if [[ ${count_matching=} != "0" && ${count_matching=} != "" ]]; then
- echo
- echo "${COLOR_BLUE}Fixing ownership of ${count_matching} root owned files on ${HOST_OS}${COLOR_RESET}"
- echo
- find "${DIRECTORIES_TO_FIX[@]}" -mindepth 1 -user root -print0 2> /dev/null |
- xargs --null chown "${HOST_USER_ID}:${HOST_GROUP_ID}" --no-dereference || true >/dev/null 2>&1
- echo "${COLOR_BLUE}Fixed ownership of generated files${COLOR_RESET}."
- echo
- fi
- else
- echo
- echo "${COLOR_YELLOW}Skip fixing ownership of generated files as Host OS is ${HOST_OS}${COLOR_RESET}"
- echo
- fi
-}
-
function in_container_go_to_airflow_sources() {
pushd "${AIRFLOW_SOURCES}" >/dev/null 2>&1 || exit 1
}
diff --git a/scripts/in_container/run_fix_ownership.py b/scripts/in_container/run_fix_ownership.py
new file mode 100755
index 0000000000..68fc3a9904
--- /dev/null
+++ b/scripts/in_container/run_fix_ownership.py
@@ -0,0 +1,84 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+import pwd
+import sys
+from pathlib import Path
+
+HOST_OS = os.environ.get("HOST_OS", "")
+DOCKER_IS_ROOTLESS = os.environ.get("DOCKER_IS_ROOTLESS", "false") == "true"
+
+
+def change_ownership_of_files(path: Path) -> None:
+ host_user_id = os.environ.get("HOST_USER_ID", "")
+ host_group_id = os.environ.get("HOST_GROUP_ID", "")
+ if host_user_id == "" or host_group_id == "":
+ print(
+ f"ERROR: HOST_USER_ID or HOST_GROUP_ID environment variables "
+ f"are not set: {host_user_id}:{host_group_id}"
+ )
+ sys.exit(1)
+ if not host_user_id.isnumeric() or not host_group_id.isnumeric():
+ print(
+ f"ERROR: HOST_USER_ID or HOST_GROUP_ID environment variables "
+ f"should be numeric: {host_user_id}:{host_group_id}"
+ )
+ sys.exit(1)
+ count_files = 0
+ root_uid = pwd.getpwnam("root").pw_uid
+ print("Attempting to see if there are files that need to be changed to host user ownership")
+ for file in path.rglob("*"):
+ try:
+ if file.is_symlink() and file.lstat().st_uid == root_uid:
+ # Change ownership of symlink itself (by default stat/chown follow the symlinks)
+ os.chown(file, int(host_user_id), int(host_group_id), follow_symlinks=False)
+ count_files += 1
+ if os.environ.get("VERBOSE_COMMANDS", "false") == "true":
+ print(f"Changed ownership of symlink {file}")
+ if file.stat().st_uid == root_uid:
+ # And here change ownership of the file (or if it is a symlink - the file it points to)
+ os.chown(file, int(host_user_id), int(host_group_id))
+ count_files += 1
+ if os.environ.get("VERBOSE_COMMANDS", "false") == "true":
+ print(f"Changed ownership of {file.resolve()}")
+ except FileNotFoundError:
+ # This is OK - file might have been deleted in the meantime or linked in Host from
+ # another place
+ if os.environ.get("VERBOSE_COMMANDS", "false") == "true":
+ print(f"Could not change ownership of {file}")
+ if count_files:
+ print(f"Changed ownership of {count_files} files back to {host_user_id}:{host_group_id}.")
+ else:
+ print("No files needed to change ownership")
+
+
+if __name__ == "__main__":
+ if HOST_OS == "":
+ print("ERROR: HOST_OS environment variable is not set")
+ sys.exit(1)
+ if HOST_OS == "Linux":
+ print("Since host OS is not Linux, we don't need to fix ownership.")
+ sys.exit(0)
+ if DOCKER_IS_ROOTLESS:
+ print("Since Docker is in rootless mode , we don't need to fix ownership even on Linux.")
+ sys.exit(0)
+
+ change_ownership_of_files(Path("/opt/airflow"))
diff --git a/scripts/in_container/run_fix_ownership.sh b/scripts/in_container/run_fix_ownership.sh
deleted file mode 100755
index d9e98ff168..0000000000
--- a/scripts/in_container/run_fix_ownership.sh
+++ /dev/null
@@ -1,21 +0,0 @@
-#!/usr/bin/env bash
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-# shellcheck source=scripts/in_container/_in_container_utils.sh
-. "$( dirname "${BASH_SOURCE[0]}" )/_in_container_utils.sh"
-
-in_container_fix_ownership
diff --git a/scripts/in_container/run_prepare_airflow_packages.py b/scripts/in_container/run_prepare_airflow_packages.py
index c1d2fb8aa5..4b319a722c 100755
--- a/scripts/in_container/run_prepare_airflow_packages.py
+++ b/scripts/in_container/run_prepare_airflow_packages.py
@@ -44,8 +44,6 @@ def process_summary(success_message: str, error_message: str, completed_process:
AIRFLOW_SOURCES_ROOT = Path(__file__).parents[2].resolve()
WWW_DIRECTORY = AIRFLOW_SOURCES_ROOT / "airflow" / "www"
-rich.print("[green]Installed packages\n")
-
rich.print("[bright_blue]Cleaning build directories\n")
for egg_info_file in AIRFLOW_SOURCES_ROOT.glob("*egg-info*"):
@@ -53,7 +51,7 @@ for egg_info_file in AIRFLOW_SOURCES_ROOT.glob("*egg-info*"):
rmtree(AIRFLOW_SOURCES_ROOT / "build", ignore_errors=True)
-rich.print("[bright_blue]Cleaned build directories\n")
+rich.print("[green]Cleaned build directories\n\n")
version_suffix = os.environ.get("VERSION_SUFFIX_FOR_PYPI", "")
package_format = os.environ.get("PACKAGE_FORMAT", "wheel")
@@ -76,7 +74,7 @@ subprocess.run(
check=True,
)
-rich.print(f"[bright_blue]Marked {AIRFLOW_SOURCES_ROOT} as safe directory for git commands.\n")
+rich.print(f"[green]Marked {AIRFLOW_SOURCES_ROOT} as safe directory for git commands.\n")
rich.print("[bright_blue]Checking airflow version\n")
@@ -84,7 +82,7 @@ airflow_version = subprocess.check_output(
[sys.executable, "setup.py", "--version"], text=True, cwd=AIRFLOW_SOURCES_ROOT
).strip()
-rich.print(f"[bright_blue]Airflow version: {airflow_version}\n")
+rich.print(f"[green]Airflow version: {airflow_version}\n")
RELEASED_VERSION_MATCHER = re.compile(r"^\d+\.\d+\.\d+$")
@@ -92,7 +90,7 @@ command = [sys.executable, "setup.py"]
if version_suffix:
if RELEASED_VERSION_MATCHER.match(airflow_version):
- rich.print(f"[bright_blue]Adding {version_suffix} suffix to the {airflow_version}")
+ rich.print(f"[warning]Adding {version_suffix} suffix to the {airflow_version}")
command.extend(["egg_info", "--tag-build", version_suffix])
elif not airflow_version.endswith(version_suffix):
rich.print(f"[red]Version {airflow_version} does not end with {version_suffix}. Using !")
@@ -112,7 +110,7 @@ process_summary("Airflow packages built successfully", "Error building Airflow p
if os.environ.get("GITHUB_ACTIONS", "") != "":
print("::endgroup::")
-rich.print("[bright_blue]Packages built successfully:\n")
+rich.print("[green]Packages built successfully:\n")
for file in (AIRFLOW_SOURCES_ROOT / "dist").glob("apache*"):
rich.print(file.name)
rich.print()
(airflow) 17/37: Move `BaseOperatorLink` into the separate module (#35032)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 4454fc870cda6924cdac100c3ca8e81d53716d40
Author: Andrey Anshin <An...@taragol.is>
AuthorDate: Tue Nov 21 21:10:40 2023 +0400
Move `BaseOperatorLink` into the separate module (#35032)
* Move `BaseOperatorLink` into the separate module
* Add `airflow.models.baseoperatorlink` as part of Public Interface of Airflow
* Ban `airflow.models.baseoperator.BaseOperatorLink` usage in codebase
* Return back check-base-operator-usage pre-commit hooks
---
.pre-commit-config.yaml | 23 +++++++--
STATIC_CODE_CHECKS.rst | 3 +-
airflow/models/__init__.py | 5 +-
airflow/models/abstractoperator.py | 3 +-
airflow/models/baseoperator.py | 49 +++++++++----------
airflow/models/baseoperatorlink.py | 57 ++++++++++++++++++++++
airflow/models/mappedoperator.py | 3 +-
airflow/operators/trigger_dagrun.py | 3 +-
airflow/sensors/external_task.py | 2 +-
airflow/serialization/serialized_objects.py | 2 +-
docs/apache-airflow/howto/define-extra-link.rst | 9 ++--
docs/apache-airflow/public-airflow-interface.rst | 7 +++
docs/conf.py | 1 +
pyproject.toml | 1 +
.../endpoints/test_extra_link_endpoint.py | 2 +-
.../endpoints/test_plugin_endpoint.py | 2 +-
tests/api_connexion/schemas/test_plugin_schema.py | 2 +-
tests/serialization/test_dag_serialization.py | 3 +-
tests/test_utils/mock_operators.py | 3 +-
tests/www/views/test_views_extra_links.py | 3 +-
20 files changed, 135 insertions(+), 48 deletions(-)
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 606f56947e..f7e886afd1 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -576,9 +576,24 @@ repos:
files: ^airflow/models/dag\.py$|^airflow/(?:decorators|utils)/task_group\.py$
- id: check-base-operator-usage
language: pygrep
- name: Check BaseOperator[Link] core imports
- description: Make sure BaseOperator[Link] is imported from airflow.models.baseoperator in core
- entry: "from airflow\\.models import.* BaseOperator"
+ name: Check BaseOperator core imports
+ description: Make sure BaseOperator is imported from airflow.models.baseoperator in core
+ entry: "from airflow\\.models import.* BaseOperator\\b"
+ files: \.py$
+ pass_filenames: true
+ exclude: >
+ (?x)
+ ^airflow/decorators/.*$|
+ ^airflow/hooks/.*$|
+ ^airflow/operators/.*$|
+ ^airflow/providers/.*$|
+ ^airflow/sensors/.*$|
+ ^dev/provider_packages/.*$
+ - id: check-base-operator-usage
+ language: pygrep
+ name: Check BaseOperatorLink core imports
+ description: Make sure BaseOperatorLink is imported from airflow.models.baseoperatorlink in core
+ entry: "from airflow\\.models import.* BaseOperatorLink"
files: \.py$
pass_filenames: true
exclude: >
@@ -593,7 +608,7 @@ repos:
language: pygrep
name: Check BaseOperator[Link] other imports
description: Make sure BaseOperator[Link] is imported from airflow.models outside of core
- entry: "from airflow\\.models\\.baseoperator import.* BaseOperator"
+ entry: "from airflow\\.models\\.baseoperator(link)? import.* BaseOperator"
pass_filenames: true
files: >
(?x)
diff --git a/STATIC_CODE_CHECKS.rst b/STATIC_CODE_CHECKS.rst
index da606c9b40..56533885de 100644
--- a/STATIC_CODE_CHECKS.rst
+++ b/STATIC_CODE_CHECKS.rst
@@ -154,7 +154,8 @@ require Breeze Docker image to be built locally.
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
| check-base-operator-partial-arguments | Check BaseOperator and partial() arguments | |
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-base-operator-usage | * Check BaseOperator[Link] core imports | |
+| check-base-operator-usage | * Check BaseOperator core imports | |
+| | * Check BaseOperatorLink core imports | |
| | * Check BaseOperator[Link] other imports | |
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
| check-boring-cyborg-configuration | Checks for Boring Cyborg configuration consistency | |
diff --git a/airflow/models/__init__.py b/airflow/models/__init__.py
index cb08730f53..a98147f416 100644
--- a/airflow/models/__init__.py
+++ b/airflow/models/__init__.py
@@ -84,7 +84,7 @@ __lazy_imports = {
"ID_LEN": "airflow.models.base",
"Base": "airflow.models.base",
"BaseOperator": "airflow.models.baseoperator",
- "BaseOperatorLink": "airflow.models.baseoperator",
+ "BaseOperatorLink": "airflow.models.baseoperatorlink",
"Connection": "airflow.models.connection",
"DagBag": "airflow.models.dagbag",
"DagModel": "airflow.models.dag",
@@ -115,7 +115,8 @@ if TYPE_CHECKING:
# I was unable to get mypy to respect a airflow/models/__init__.pyi, so
# having to resort back to this hacky method
from airflow.models.base import ID_LEN, Base
- from airflow.models.baseoperator import BaseOperator, BaseOperatorLink
+ from airflow.models.baseoperator import BaseOperator
+ from airflow.models.baseoperatorlink import BaseOperatorLink
from airflow.models.connection import Connection
from airflow.models.dag import DAG, DagModel, DagTag
from airflow.models.dagbag import DagBag
diff --git a/airflow/models/abstractoperator.py b/airflow/models/abstractoperator.py
index aa79555557..df0e6cb349 100644
--- a/airflow/models/abstractoperator.py
+++ b/airflow/models/abstractoperator.py
@@ -48,7 +48,8 @@ if TYPE_CHECKING:
import jinja2 # Slow import.
from sqlalchemy.orm import Session
- from airflow.models.baseoperator import BaseOperator, BaseOperatorLink
+ from airflow.models.baseoperator import BaseOperator
+ from airflow.models.baseoperatorlink import BaseOperatorLink
from airflow.models.dag import DAG
from airflow.models.mappedoperator import MappedOperator
from airflow.models.operator import Operator
diff --git a/airflow/models/baseoperator.py b/airflow/models/baseoperator.py
index 61fb6e07de..2ba7ec8ad1 100644
--- a/airflow/models/baseoperator.py
+++ b/airflow/models/baseoperator.py
@@ -26,7 +26,6 @@ import functools
import logging
import sys
import warnings
-from abc import ABCMeta, abstractmethod
from datetime import datetime, timedelta
from inspect import signature
from types import FunctionType
@@ -34,7 +33,6 @@ from typing import (
TYPE_CHECKING,
Any,
Callable,
- ClassVar,
Collection,
Iterable,
Sequence,
@@ -101,12 +99,10 @@ if TYPE_CHECKING:
import jinja2 # Slow import.
from sqlalchemy.orm import Session
- from airflow.models.abstractoperator import (
- TaskStateChangeCallback,
- )
+ from airflow.models.abstractoperator import TaskStateChangeCallback
+ from airflow.models.baseoperatorlink import BaseOperatorLink
from airflow.models.dag import DAG
from airflow.models.operator import Operator
- from airflow.models.taskinstancekey import TaskInstanceKey
from airflow.models.xcom_arg import XComArg
from airflow.ti_deps.deps.base_ti_dep import BaseTIDep
from airflow.triggers.base import BaseTrigger
@@ -1896,31 +1892,30 @@ def chain_linear(*elements: DependencyMixin | Sequence[DependencyMixin]):
raise ValueError("No dependencies were set. Did you forget to expand with `*`?")
-@attr.s(auto_attribs=True)
-class BaseOperatorLink(metaclass=ABCMeta):
- """Abstract base class that defines how we get an operator link."""
-
- operators: ClassVar[list[type[BaseOperator]]] = []
+def __getattr__(name):
"""
- This property will be used by Airflow Plugins to find the Operators to which you want
- to assign this Operator Link
+ PEP-562: Lazy loaded attributes on python modules.
- :return: List of Operator classes used by task for which you want to create extra link
+ :meta private:
"""
+ path = __deprecated_imports.get(name)
+ if not path:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
- @property
- @abstractmethod
- def name(self) -> str:
- """Name of the link. This will be the button name on the task UI."""
+ from airflow.utils.module_loading import import_string
- @abstractmethod
- def get_link(self, operator: BaseOperator, *, ti_key: TaskInstanceKey) -> str:
- """Link to external system.
+ warnings.warn(
+ f"Import `{__name__}.{name}` is deprecated. Please use `{path}.{name}`.",
+ RemovedInAirflow3Warning,
+ stacklevel=2,
+ )
+ val = import_string(f"{path}.{name}")
- Note: The old signature of this function was ``(self, operator, dttm: datetime)``. That is still
- supported at runtime but is deprecated.
+ # Store for next time
+ globals()[name] = val
+ return val
- :param operator: The Airflow operator object this link is associated to.
- :param ti_key: TaskInstance ID to return link for.
- :return: link to external system
- """
+
+__deprecated_imports = {
+ "BaseOperatorLink": "airflow.models.baseoperatorlink",
+}
diff --git a/airflow/models/baseoperatorlink.py b/airflow/models/baseoperatorlink.py
new file mode 100644
index 0000000000..a1d1e02dd4
--- /dev/null
+++ b/airflow/models/baseoperatorlink.py
@@ -0,0 +1,57 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from abc import ABCMeta, abstractmethod
+from typing import TYPE_CHECKING, ClassVar
+
+import attr
+
+if TYPE_CHECKING:
+ from airflow.models.baseoperator import BaseOperator
+ from airflow.models.taskinstancekey import TaskInstanceKey
+
+
+@attr.s(auto_attribs=True)
+class BaseOperatorLink(metaclass=ABCMeta):
+ """Abstract base class that defines how we get an operator link."""
+
+ operators: ClassVar[list[type[BaseOperator]]] = []
+ """
+ This property will be used by Airflow Plugins to find the Operators to which you want
+ to assign this Operator Link
+
+ :return: List of Operator classes used by task for which you want to create extra link
+ """
+
+ @property
+ @abstractmethod
+ def name(self) -> str:
+ """Name of the link. This will be the button name on the task UI."""
+
+ @abstractmethod
+ def get_link(self, operator: BaseOperator, *, ti_key: TaskInstanceKey) -> str:
+ """Link to external system.
+
+ Note: The old signature of this function was ``(self, operator, dttm: datetime)``. That is still
+ supported at runtime but is deprecated.
+
+ :param operator: The Airflow operator object this link is associated to.
+ :param ti_key: TaskInstance ID to return link for.
+ :return: link to external system
+ """
diff --git a/airflow/models/mappedoperator.py b/airflow/models/mappedoperator.py
index ba49924b88..8174db145a 100644
--- a/airflow/models/mappedoperator.py
+++ b/airflow/models/mappedoperator.py
@@ -66,7 +66,8 @@ if TYPE_CHECKING:
from airflow.models.abstractoperator import (
TaskStateChangeCallback,
)
- from airflow.models.baseoperator import BaseOperator, BaseOperatorLink
+ from airflow.models.baseoperator import BaseOperator
+ from airflow.models.baseoperatorlink import BaseOperatorLink
from airflow.models.dag import DAG
from airflow.models.expandinput import (
ExpandInput,
diff --git a/airflow/operators/trigger_dagrun.py b/airflow/operators/trigger_dagrun.py
index e0d9de9f43..0bca11a801 100644
--- a/airflow/operators/trigger_dagrun.py
+++ b/airflow/operators/trigger_dagrun.py
@@ -28,7 +28,8 @@ from sqlalchemy.orm.exc import NoResultFound
from airflow.api.common.trigger_dag import trigger_dag
from airflow.configuration import conf
from airflow.exceptions import AirflowException, DagNotFound, DagRunAlreadyExists
-from airflow.models.baseoperator import BaseOperator, BaseOperatorLink
+from airflow.models.baseoperator import BaseOperator
+from airflow.models.baseoperatorlink import BaseOperatorLink
from airflow.models.dag import DagModel
from airflow.models.dagbag import DagBag
from airflow.models.dagrun import DagRun
diff --git a/airflow/sensors/external_task.py b/airflow/sensors/external_task.py
index 3090b8a50f..79734d25a3 100644
--- a/airflow/sensors/external_task.py
+++ b/airflow/sensors/external_task.py
@@ -27,7 +27,7 @@ from sqlalchemy import func
from airflow.configuration import conf
from airflow.exceptions import AirflowException, AirflowSkipException, RemovedInAirflow3Warning
-from airflow.models.baseoperator import BaseOperatorLink
+from airflow.models.baseoperatorlink import BaseOperatorLink
from airflow.models.dag import DagModel
from airflow.models.dagbag import DagBag
from airflow.models.dagrun import DagRun
diff --git a/airflow/serialization/serialized_objects.py b/airflow/serialization/serialized_objects.py
index d2a72cb1ca..889177f4c1 100644
--- a/airflow/serialization/serialized_objects.py
+++ b/airflow/serialization/serialized_objects.py
@@ -70,7 +70,7 @@ if TYPE_CHECKING:
from pydantic import BaseModel
- from airflow.models.baseoperator import BaseOperatorLink
+ from airflow.models.baseoperatorlink import BaseOperatorLink
from airflow.models.expandinput import ExpandInput
from airflow.models.operator import Operator
from airflow.models.taskmixin import DAGNode
diff --git a/docs/apache-airflow/howto/define-extra-link.rst b/docs/apache-airflow/howto/define-extra-link.rst
index bfa5679d34..c934d52fa4 100644
--- a/docs/apache-airflow/howto/define-extra-link.rst
+++ b/docs/apache-airflow/howto/define-extra-link.rst
@@ -27,7 +27,8 @@ The following code shows how to add extra links to an operator via Plugins:
.. code-block:: python
- from airflow.models.baseoperator import BaseOperator, BaseOperatorLink
+ from airflow.models.baseoperator import BaseOperator
+ from airflow.models.baseoperatorlink import BaseOperatorLink
from airflow.models.taskinstancekey import TaskInstanceKey
from airflow.plugins_manager import AirflowPlugin
@@ -80,7 +81,8 @@ tasks using :class:`~airflow.providers.amazon.aws.transfers.gcs_to_s3.GCSToS3Ope
.. code-block:: python
- from airflow.models.baseoperator import BaseOperator, BaseOperatorLink
+ from airflow.models.baseoperator import BaseOperator
+ from airflow.models.baseoperatorlink import BaseOperatorLink
from airflow.models.taskinstancekey import TaskInstanceKey
from airflow.plugins_manager import AirflowPlugin
from airflow.providers.amazon.aws.transfers.gcs_to_s3 import GCSToS3Operator
@@ -118,7 +120,8 @@ Console, but if we wanted to change that link we could:
.. code-block:: python
- from airflow.models.baseoperator import BaseOperatorLink, BaseOperator
+ from airflow.models.baseoperator import BaseOperator
+ from airflow.models.baseoperatorlink import BaseOperatorLink
from airflow.models.taskinstancekey import TaskInstanceKey
from airflow.models.xcom import XCom
from airflow.plugins_manager import AirflowPlugin
diff --git a/docs/apache-airflow/public-airflow-interface.rst b/docs/apache-airflow/public-airflow-interface.rst
index 93e68fa797..f16cc5aab0 100644
--- a/docs/apache-airflow/public-airflow-interface.rst
+++ b/docs/apache-airflow/public-airflow-interface.rst
@@ -312,6 +312,13 @@ Extra Links
Extra links are dynamic links that could be added to Airflow independently from custom Operators. Normally
they can be defined by the Operators, but plugins allow you to override the links on a global level.
+.. toctree::
+ :includehidden:
+ :glob:
+ :maxdepth: 1
+
+ _api/airflow/models/baseoperatorlink/index
+
You can read more about the Extra Links in :doc:`/howto/define-extra-link`.
Using Public Interface to integrate with external services and applications
diff --git a/docs/conf.py b/docs/conf.py
index 3c7cfa26b0..e832e1e012 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -248,6 +248,7 @@ if PACKAGE_NAME == "apache-airflow":
models_included: set[str] = {
"baseoperator.py",
+ "baseoperatorlink.py",
"connection.py",
"dag.py",
"dagrun.py",
diff --git a/pyproject.toml b/pyproject.toml
index 9722a0d7ac..3fbc95e88b 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -149,6 +149,7 @@ combine-as-imports = true
[tool.ruff.flake8-tidy-imports.banned-api]
"airflow.AirflowException".msg = "Use airflow.exceptions.AirflowException instead."
"airflow.Dataset".msg = "Use airflow.datasets.Dataset instead."
+"airflow.models.baseoperator.BaseOperatorLink".msg = "Use airflow.models.baseoperatorlink.BaseOperatorLink"
[tool.ruff.flake8-tidy-imports]
# Ban certain modules from being imported at module level, instead requiring
diff --git a/tests/api_connexion/endpoints/test_extra_link_endpoint.py b/tests/api_connexion/endpoints/test_extra_link_endpoint.py
index a52c034268..3e803a4bf4 100644
--- a/tests/api_connexion/endpoints/test_extra_link_endpoint.py
+++ b/tests/api_connexion/endpoints/test_extra_link_endpoint.py
@@ -22,7 +22,7 @@ from urllib.parse import quote_plus
import pytest
from airflow.api_connexion.exceptions import EXCEPTIONS_LINK_MAP
-from airflow.models.baseoperator import BaseOperatorLink
+from airflow.models.baseoperatorlink import BaseOperatorLink
from airflow.models.dag import DAG
from airflow.models.dagbag import DagBag
from airflow.models.xcom import XCom
diff --git a/tests/api_connexion/endpoints/test_plugin_endpoint.py b/tests/api_connexion/endpoints/test_plugin_endpoint.py
index 4d9b8be847..57c3387ae0 100644
--- a/tests/api_connexion/endpoints/test_plugin_endpoint.py
+++ b/tests/api_connexion/endpoints/test_plugin_endpoint.py
@@ -23,7 +23,7 @@ from flask import Blueprint
from flask_appbuilder import BaseView
from airflow.hooks.base import BaseHook
-from airflow.models.baseoperator import BaseOperatorLink
+from airflow.models.baseoperatorlink import BaseOperatorLink
from airflow.plugins_manager import AirflowPlugin
from airflow.security import permissions
from airflow.ti_deps.deps.base_ti_dep import BaseTIDep
diff --git a/tests/api_connexion/schemas/test_plugin_schema.py b/tests/api_connexion/schemas/test_plugin_schema.py
index 1472fd2db7..0511072585 100644
--- a/tests/api_connexion/schemas/test_plugin_schema.py
+++ b/tests/api_connexion/schemas/test_plugin_schema.py
@@ -25,7 +25,7 @@ from airflow.api_connexion.schemas.plugin_schema import (
plugin_schema,
)
from airflow.hooks.base import BaseHook
-from airflow.models.baseoperator import BaseOperatorLink
+from airflow.models.baseoperatorlink import BaseOperatorLink
from airflow.plugins_manager import AirflowPlugin
diff --git a/tests/serialization/test_dag_serialization.py b/tests/serialization/test_dag_serialization.py
index a9c63c80fe..30407eb945 100644
--- a/tests/serialization/test_dag_serialization.py
+++ b/tests/serialization/test_dag_serialization.py
@@ -43,7 +43,8 @@ from airflow.decorators import teardown
from airflow.decorators.base import DecoratedOperator
from airflow.exceptions import AirflowException, SerializationError
from airflow.hooks.base import BaseHook
-from airflow.models.baseoperator import BaseOperator, BaseOperatorLink
+from airflow.models.baseoperator import BaseOperator
+from airflow.models.baseoperatorlink import BaseOperatorLink
from airflow.models.connection import Connection
from airflow.models.dag import DAG
from airflow.models.dagbag import DagBag
diff --git a/tests/test_utils/mock_operators.py b/tests/test_utils/mock_operators.py
index 4aa59120a8..e9a05c9edb 100644
--- a/tests/test_utils/mock_operators.py
+++ b/tests/test_utils/mock_operators.py
@@ -21,7 +21,8 @@ from typing import TYPE_CHECKING, Any, Sequence
import attr
-from airflow.models.baseoperator import BaseOperator, BaseOperatorLink
+from airflow.models.baseoperator import BaseOperator
+from airflow.models.baseoperatorlink import BaseOperatorLink
from airflow.models.xcom import XCom
if TYPE_CHECKING:
diff --git a/tests/www/views/test_views_extra_links.py b/tests/www/views/test_views_extra_links.py
index eb3b17517a..a959fa9fb8 100644
--- a/tests/www/views/test_views_extra_links.py
+++ b/tests/www/views/test_views_extra_links.py
@@ -23,7 +23,8 @@ from unittest import mock
import pytest
-from airflow.models.baseoperator import BaseOperator, BaseOperatorLink
+from airflow.models.baseoperator import BaseOperator
+from airflow.models.baseoperatorlink import BaseOperatorLink
from airflow.models.dag import DAG
from airflow.utils import timezone
from airflow.utils.state import DagRunState
(airflow) 14/37: Fix broken link to Weaviate docs (#35776)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 1df306a337c8951b7e8cf06ae6331313b8b23e9c
Author: Pankaj Koti <pa...@gmail.com>
AuthorDate: Tue Nov 21 17:34:50 2023 +0530
Fix broken link to Weaviate docs (#35776)
---
docs/apache-airflow-providers-weaviate/connections.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/apache-airflow-providers-weaviate/connections.rst b/docs/apache-airflow-providers-weaviate/connections.rst
index fc46c972a8..5e16164ff6 100644
--- a/docs/apache-airflow-providers-weaviate/connections.rst
+++ b/docs/apache-airflow-providers-weaviate/connections.rst
@@ -46,7 +46,7 @@ Extra (optional)
* If you'd like to use Vectorizers for your class, configure the API keys to use the corresponding
embedding API. The extras accepts a key ``additional_headers`` containing the dictionary
of API keys for the embedding API authentication. They are mentioned in a section here:
- `addtional_headers <https://weaviate.io/developers/academy/zero_to_mvp/hello_weaviate/hands_on#-client-instantiation>__`
+ `addtional_headers <https://weaviate.io/developers/academy/zero_to_mvp/hello_weaviate/hands_on#-client-instantiation>`__
Weaviate API Token (optional)
Specify your Weaviate API Key to connect when API Key option is to be used for authentication.
(airflow) 19/37: Implement login and logout in AWS auth manager (#35488)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit e2e89668d46363bc0ecfa31058d02bc3e041de88
Author: Vincent <97...@users.noreply.github.com>
AuthorDate: Tue Nov 21 12:35:10 2023 -0500
Implement login and logout in AWS auth manager (#35488)
---
CONTRIBUTING.rst | 6 +-
Dockerfile | 4 +-
Dockerfile.ci | 4 +-
INSTALL | 6 +-
airflow/auth/managers/base_auth_manager.py | 32 +++--
airflow/auth/managers/fab/fab_auth_manager.py | 16 +--
airflow/auth/managers/fab/models/__init__.py | 3 +
airflow/auth/managers/models/base_user.py | 7 +-
.../amazon/aws/auth_manager/__init__.py} | 17 ---
.../amazon/aws/auth_manager/aws_auth_manager.py | 143 +++++++++++++++++++
.../amazon/aws/auth_manager/constants.py} | 20 +--
.../aws/auth_manager/security_manager/__init__.py} | 17 ---
.../aws_security_manager_override.py} | 25 ++--
airflow/providers/amazon/aws/auth_manager/user.py | 51 +++++++
.../amazon/aws/auth_manager/views/__init__.py} | 17 ---
.../amazon/aws/auth_manager/views/auth.py | 149 ++++++++++++++++++++
airflow/providers/amazon/provider.yaml | 25 ++++
airflow/www/extensions/init_appbuilder.py | 2 +-
airflow/www/extensions/init_auth_manager.py | 6 +-
airflow/www/extensions/init_security.py | 4 +-
airflow/www/security_manager.py | 33 +++--
airflow/www/utils.py | 5 +-
docs/apache-airflow/extra-packages-ref.rst | 2 +
docs/docker-stack/changelog.rst | 3 +
docs/spelling_wordlist.txt | 1 +
scripts/docker/install_os_dependencies.sh | 4 +-
setup.py | 7 +
.../endpoints/test_forward_to_fab_endpoint.py | 2 +-
tests/auth/managers/fab/test_fab_auth_manager.py | 38 +----
tests/auth/managers/test_base_auth_manager.py | 48 +++++--
.../providers/amazon/aws/auth_manager/__init__.py | 17 ---
.../aws/auth_manager/security_manager/__init__.py | 17 ---
.../test_aws_security_manager_override.py | 56 ++++++++
.../aws/auth_manager/test_aws_auth_manager.py | 114 +++++++++++++++
.../amazon/aws/auth_manager/test_constants.py | 23 +--
.../providers/amazon/aws/auth_manager/test_user.py | 32 +++--
.../amazon/aws/auth_manager/views/__init__.py | 17 ---
.../amazon/aws/auth_manager/views/test_auth.py | 156 +++++++++++++++++++++
38 files changed, 865 insertions(+), 264 deletions(-)
diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
index 5a5fae65ca..1114c16074 100644
--- a/CONTRIBUTING.rst
+++ b/CONTRIBUTING.rst
@@ -679,9 +679,9 @@ github_enterprise, google, google_auth, grpc, hashicorp, hdfs, hive, http, imap,
jenkins, kerberos, kubernetes, ldap, leveldb, microsoft.azure, microsoft.mssql, microsoft.psrp,
microsoft.winrm, mongo, mssql, mysql, neo4j, odbc, openai, openfaas, openlineage, opensearch,
opsgenie, oracle, otel, pagerduty, pandas, papermill, password, pgvector, pinecone, pinot, plexus,
-postgres, presto, rabbitmq, redis, s3, s3fs, salesforce, samba, segment, sendgrid, sentry, sftp,
-singularity, slack, smtp, snowflake, spark, sqlite, ssh, statsd, tableau, tabular, telegram, trino,
-vertica, virtualenv, weaviate, webhdfs, winrm, yandex, zendesk
+postgres, presto, rabbitmq, redis, s3, s3fs, salesforce, samba, saml, segment, sendgrid, sentry,
+sftp, singularity, slack, smtp, snowflake, spark, sqlite, ssh, statsd, tableau, tabular, telegram,
+trino, vertica, virtualenv, weaviate, webhdfs, winrm, yandex, zendesk
.. END EXTRAS HERE
Provider packages
diff --git a/Dockerfile b/Dockerfile
index 94794b3735..b9b358d0c7 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -97,7 +97,7 @@ function get_dev_apt_deps() {
DEV_APT_DEPS="apt-transport-https apt-utils build-essential ca-certificates dirmngr \
freetds-bin freetds-dev git gosu graphviz graphviz-dev krb5-user ldap-utils libffi-dev libgeos-dev \
libkrb5-dev libldap2-dev libleveldb1d libleveldb-dev libsasl2-2 libsasl2-dev libsasl2-modules \
-libssl-dev locales lsb-release openssh-client pkgconf sasl2-bin \
+libssl-dev libxmlsec1 libxmlsec1-dev locales lsb-release openssh-client pkgconf sasl2-bin \
software-properties-common sqlite3 sudo unixodbc unixodbc-dev"
export DEV_APT_DEPS
fi
@@ -123,7 +123,7 @@ function get_runtime_apt_deps() {
if [[ "${RUNTIME_APT_DEPS=}" == "" ]]; then
RUNTIME_APT_DEPS="apt-transport-https apt-utils ca-certificates \
curl dumb-init freetds-bin gosu krb5-user libgeos-dev \
-ldap-utils libsasl2-2 libsasl2-modules locales ${debian_version_apt_deps} \
+ldap-utils libsasl2-2 libsasl2-modules libxmlsec1 locales ${debian_version_apt_deps} \
lsb-release openssh-client python3-selinux rsync sasl2-bin sqlite3 sudo unixodbc"
export RUNTIME_APT_DEPS
fi
diff --git a/Dockerfile.ci b/Dockerfile.ci
index 81a9d424a0..c3e5e2e3a0 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -57,7 +57,7 @@ function get_dev_apt_deps() {
DEV_APT_DEPS="apt-transport-https apt-utils build-essential ca-certificates dirmngr \
freetds-bin freetds-dev git gosu graphviz graphviz-dev krb5-user ldap-utils libffi-dev libgeos-dev \
libkrb5-dev libldap2-dev libleveldb1d libleveldb-dev libsasl2-2 libsasl2-dev libsasl2-modules \
-libssl-dev locales lsb-release openssh-client pkgconf sasl2-bin \
+libssl-dev libxmlsec1 libxmlsec1-dev locales lsb-release openssh-client pkgconf sasl2-bin \
software-properties-common sqlite3 sudo unixodbc unixodbc-dev"
export DEV_APT_DEPS
fi
@@ -83,7 +83,7 @@ function get_runtime_apt_deps() {
if [[ "${RUNTIME_APT_DEPS=}" == "" ]]; then
RUNTIME_APT_DEPS="apt-transport-https apt-utils ca-certificates \
curl dumb-init freetds-bin gosu krb5-user libgeos-dev \
-ldap-utils libsasl2-2 libsasl2-modules locales ${debian_version_apt_deps} \
+ldap-utils libsasl2-2 libsasl2-modules libxmlsec1 locales ${debian_version_apt_deps} \
lsb-release openssh-client python3-selinux rsync sasl2-bin sqlite3 sudo unixodbc"
export RUNTIME_APT_DEPS
fi
diff --git a/INSTALL b/INSTALL
index 7742c006bc..0ac0ac33d0 100644
--- a/INSTALL
+++ b/INSTALL
@@ -105,9 +105,9 @@ github_enterprise, google, google_auth, grpc, hashicorp, hdfs, hive, http, imap,
jenkins, kerberos, kubernetes, ldap, leveldb, microsoft.azure, microsoft.mssql, microsoft.psrp,
microsoft.winrm, mongo, mssql, mysql, neo4j, odbc, openai, openfaas, openlineage, opensearch,
opsgenie, oracle, otel, pagerduty, pandas, papermill, password, pgvector, pinecone, pinot, plexus,
-postgres, presto, rabbitmq, redis, s3, s3fs, salesforce, samba, segment, sendgrid, sentry, sftp,
-singularity, slack, smtp, snowflake, spark, sqlite, ssh, statsd, tableau, tabular, telegram, trino,
-vertica, virtualenv, weaviate, webhdfs, winrm, yandex, zendesk
+postgres, presto, rabbitmq, redis, s3, s3fs, salesforce, samba, saml, segment, sendgrid, sentry,
+sftp, singularity, slack, smtp, snowflake, spark, sqlite, ssh, statsd, tableau, tabular, telegram,
+trino, vertica, virtualenv, weaviate, webhdfs, winrm, yandex, zendesk
# END EXTRAS HERE
# For installing Airflow in development environments - see CONTRIBUTING.rst
diff --git a/airflow/auth/managers/base_auth_manager.py b/airflow/auth/managers/base_auth_manager.py
index c158ea8481..f50e40082b 100644
--- a/airflow/auth/managers/base_auth_manager.py
+++ b/airflow/auth/managers/base_auth_manager.py
@@ -33,7 +33,6 @@ from airflow.utils.session import NEW_SESSION, provide_session
if TYPE_CHECKING:
from connexion import FlaskApi
- from flask import Flask
from sqlalchemy.orm import Session
from airflow.auth.managers.models.base_user import BaseUser
@@ -64,11 +63,12 @@ class BaseAuthManager(LoggingMixin):
Class to derive in order to implement concrete auth managers.
Auth managers are responsible for any user management related operation such as login, logout, authz, ...
+
+ :param appbuilder: the flask app builder
"""
- def __init__(self, app: Flask, appbuilder: AirflowAppBuilder) -> None:
+ def __init__(self, appbuilder: AirflowAppBuilder) -> None:
super().__init__()
- self.app = app
self.appbuilder = appbuilder
@staticmethod
@@ -83,21 +83,29 @@ class BaseAuthManager(LoggingMixin):
"""Return API endpoint(s) definition for the auth manager."""
return None
- @abstractmethod
def get_user_name(self) -> str:
"""Return the username associated to the user in session."""
+ user = self.get_user()
+ if not user:
+ self.log.error("Calling 'get_user_name()' but the user is not signed in.")
+ raise AirflowException("The user must be signed in.")
+ return user.get_name()
- @abstractmethod
def get_user_display_name(self) -> str:
"""Return the user's display name associated to the user in session."""
+ return self.get_user_name()
@abstractmethod
- def get_user(self) -> BaseUser:
+ def get_user(self) -> BaseUser | None:
"""Return the user associated to the user in session."""
- @abstractmethod
def get_user_id(self) -> str:
"""Return the user ID associated to the user in session."""
+ user = self.get_user()
+ if not user:
+ self.log.error("Calling 'get_user_id()' but the user is not signed in.")
+ raise AirflowException("The user must be signed in.")
+ return str(user.get_id())
def init(self) -> None:
"""
@@ -384,9 +392,13 @@ class BaseAuthManager(LoggingMixin):
def get_url_logout(self) -> str:
"""Return the logout page url."""
- @abstractmethod
def get_url_user_profile(self) -> str | None:
- """Return the url to a page displaying info about the current user."""
+ """
+ Return the url to a page displaying info about the current user.
+
+ By default, return None.
+ """
+ return None
@cached_property
def security_manager(self) -> AirflowSecurityManagerV2:
@@ -394,7 +406,7 @@ class BaseAuthManager(LoggingMixin):
Return the security manager.
By default, Airflow comes with the default security manager
- airflow.www.security_manager.AirflowSecurityManagerV2. The auth manager might need to extend this
+ ``airflow.www.security_manager.AirflowSecurityManagerV2``. The auth manager might need to extend this
default security manager for its own purposes.
By default, return the default AirflowSecurityManagerV2.
diff --git a/airflow/auth/managers/fab/fab_auth_manager.py b/airflow/auth/managers/fab/fab_auth_manager.py
index 6305406d2a..5224509039 100644
--- a/airflow/auth/managers/fab/fab_auth_manager.py
+++ b/airflow/auth/managers/fab/fab_auth_manager.py
@@ -170,26 +170,12 @@ class FabAuthManager(BaseAuthManager):
last_name = user.last_name.strip() if isinstance(user.last_name, str) else ""
return f"{first_name} {last_name}".strip()
- def get_user_name(self) -> str:
- """
- Return the username associated to the user in session.
-
- For backward compatibility reasons, the username in FAB auth manager can be any of username,
- email, or the database user ID.
- """
- user = self.get_user()
- return user.username or user.email or self.get_user_id()
-
def get_user(self) -> User:
"""Return the user associated to the user in session."""
from flask_login import current_user
return current_user
- def get_user_id(self) -> str:
- """Return the user ID associated to the user in session."""
- return str(self.get_user().get_id())
-
def init(self) -> None:
"""Run operations when Airflow is initializing."""
self._sync_appbuilder_roles()
@@ -351,7 +337,7 @@ class FabAuthManager(BaseAuthManager):
from airflow.auth.managers.fab.security_manager.override import FabAirflowSecurityManagerOverride
from airflow.www.security import AirflowSecurityManager
- sm_from_config = self.app.config.get("SECURITY_MANAGER_CLASS")
+ sm_from_config = self.appbuilder.get_app.config.get("SECURITY_MANAGER_CLASS")
if sm_from_config:
if not issubclass(sm_from_config, AirflowSecurityManager):
raise Exception(
diff --git a/airflow/auth/managers/fab/models/__init__.py b/airflow/auth/managers/fab/models/__init__.py
index 28de7840ec..1fe967d8c7 100644
--- a/airflow/auth/managers/fab/models/__init__.py
+++ b/airflow/auth/managers/fab/models/__init__.py
@@ -228,6 +228,9 @@ class User(Model, BaseUser):
def get_id(self):
return self.id
+ def get_name(self) -> str:
+ return self.username or self.email or self.user_id
+
def get_full_name(self):
return f"{self.first_name} {self.last_name}"
diff --git a/airflow/auth/managers/models/base_user.py b/airflow/auth/managers/models/base_user.py
index 7cb45d8389..93163ee06d 100644
--- a/airflow/auth/managers/models/base_user.py
+++ b/airflow/auth/managers/models/base_user.py
@@ -24,10 +24,13 @@ class BaseUser:
"""User model interface."""
@property
- @abstractmethod
def is_active(self) -> bool:
- ...
+ return True
@abstractmethod
def get_id(self) -> str:
...
+
+ @abstractmethod
+ def get_name(self) -> str:
+ ...
diff --git a/airflow/auth/managers/models/base_user.py b/airflow/providers/amazon/aws/auth_manager/__init__.py
similarity index 75%
copy from airflow/auth/managers/models/base_user.py
copy to airflow/providers/amazon/aws/auth_manager/__init__.py
index 7cb45d8389..13a83393a9 100644
--- a/airflow/auth/managers/models/base_user.py
+++ b/airflow/providers/amazon/aws/auth_manager/__init__.py
@@ -1,4 +1,3 @@
-#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -15,19 +14,3 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-from __future__ import annotations
-
-from abc import abstractmethod
-
-
-class BaseUser:
- """User model interface."""
-
- @property
- @abstractmethod
- def is_active(self) -> bool:
- ...
-
- @abstractmethod
- def get_id(self) -> str:
- ...
diff --git a/airflow/providers/amazon/aws/auth_manager/aws_auth_manager.py b/airflow/providers/amazon/aws/auth_manager/aws_auth_manager.py
new file mode 100644
index 0000000000..fcf3caacb9
--- /dev/null
+++ b/airflow/providers/amazon/aws/auth_manager/aws_auth_manager.py
@@ -0,0 +1,143 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from functools import cached_property
+from typing import TYPE_CHECKING
+
+from flask import session, url_for
+
+from airflow.configuration import conf
+from airflow.exceptions import AirflowOptionalProviderFeatureException
+from airflow.providers.amazon.aws.auth_manager.constants import (
+ CONF_ENABLE_KEY,
+ CONF_SECTION_NAME,
+)
+from airflow.providers.amazon.aws.auth_manager.security_manager.aws_security_manager_override import (
+ AwsSecurityManagerOverride,
+)
+
+try:
+ from airflow.auth.managers.base_auth_manager import BaseAuthManager, ResourceMethod
+except ImportError:
+ raise AirflowOptionalProviderFeatureException(
+ "Failed to import BaseUser. This feature is only available in Airflow versions >= 2.8.0"
+ )
+
+if TYPE_CHECKING:
+ from airflow.auth.managers.models.base_user import BaseUser
+ from airflow.auth.managers.models.resource_details import (
+ AccessView,
+ ConfigurationDetails,
+ ConnectionDetails,
+ DagAccessEntity,
+ DagDetails,
+ DatasetDetails,
+ PoolDetails,
+ VariableDetails,
+ )
+ from airflow.providers.amazon.aws.auth_manager.user import AwsAuthManagerUser
+ from airflow.www.extensions.init_appbuilder import AirflowAppBuilder
+
+
+class AwsAuthManager(BaseAuthManager):
+ """
+ AWS auth manager.
+
+ Leverages AWS services such as Amazon Identity Center and Amazon Verified Permissions to perform
+ authentication and authorization in Airflow.
+
+ :param appbuilder: the flask app builder
+ """
+
+ def __init__(self, appbuilder: AirflowAppBuilder) -> None:
+ super().__init__(appbuilder)
+ enable = conf.getboolean(CONF_SECTION_NAME, CONF_ENABLE_KEY)
+ if not enable:
+ raise NotImplementedError(
+ "The AWS auth manager is currently being built. It is not finalized. It is not intended to be used yet."
+ )
+
+ def get_user(self) -> AwsAuthManagerUser | None:
+ return session["aws_user"] if self.is_logged_in() else None
+
+ def is_logged_in(self) -> bool:
+ return "aws_user" in session
+
+ def is_authorized_configuration(
+ self,
+ *,
+ method: ResourceMethod,
+ details: ConfigurationDetails | None = None,
+ user: BaseUser | None = None,
+ ) -> bool:
+ return self.is_logged_in()
+
+ def is_authorized_cluster_activity(self, *, method: ResourceMethod, user: BaseUser | None = None) -> bool:
+ return self.is_logged_in()
+
+ def is_authorized_connection(
+ self,
+ *,
+ method: ResourceMethod,
+ details: ConnectionDetails | None = None,
+ user: BaseUser | None = None,
+ ) -> bool:
+ return self.is_logged_in()
+
+ def is_authorized_dag(
+ self,
+ *,
+ method: ResourceMethod,
+ access_entity: DagAccessEntity | None = None,
+ details: DagDetails | None = None,
+ user: BaseUser | None = None,
+ ) -> bool:
+ return self.is_logged_in()
+
+ def is_authorized_dataset(
+ self, *, method: ResourceMethod, details: DatasetDetails | None = None, user: BaseUser | None = None
+ ) -> bool:
+ return self.is_logged_in()
+
+ def is_authorized_pool(
+ self, *, method: ResourceMethod, details: PoolDetails | None = None, user: BaseUser | None = None
+ ) -> bool:
+ return self.is_logged_in()
+
+ def is_authorized_variable(
+ self, *, method: ResourceMethod, details: VariableDetails | None = None, user: BaseUser | None = None
+ ) -> bool:
+ return self.is_logged_in()
+
+ def is_authorized_view(
+ self,
+ *,
+ access_view: AccessView,
+ user: BaseUser | None = None,
+ ) -> bool:
+ return self.is_logged_in()
+
+ def get_url_login(self, **kwargs) -> str:
+ return url_for("AwsAuthManagerAuthenticationViews.login")
+
+ def get_url_logout(self) -> str:
+ return url_for("AwsAuthManagerAuthenticationViews.logout")
+
+ @cached_property
+ def security_manager(self) -> AwsSecurityManagerOverride:
+ return AwsSecurityManagerOverride(self.appbuilder)
diff --git a/airflow/auth/managers/models/base_user.py b/airflow/providers/amazon/aws/auth_manager/constants.py
similarity index 78%
copy from airflow/auth/managers/models/base_user.py
copy to airflow/providers/amazon/aws/auth_manager/constants.py
index 7cb45d8389..f2f9c1da07 100644
--- a/airflow/auth/managers/models/base_user.py
+++ b/airflow/providers/amazon/aws/auth_manager/constants.py
@@ -1,4 +1,3 @@
-#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -15,19 +14,10 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-from __future__ import annotations
-
-from abc import abstractmethod
+# Configuration keys
+from __future__ import annotations
-class BaseUser:
- """User model interface."""
-
- @property
- @abstractmethod
- def is_active(self) -> bool:
- ...
-
- @abstractmethod
- def get_id(self) -> str:
- ...
+CONF_SECTION_NAME = "aws_auth_manager"
+CONF_SAML_METADATA_URL_KEY = "saml_metadata_url"
+CONF_ENABLE_KEY = "enable"
diff --git a/airflow/auth/managers/models/base_user.py b/airflow/providers/amazon/aws/auth_manager/security_manager/__init__.py
similarity index 75%
copy from airflow/auth/managers/models/base_user.py
copy to airflow/providers/amazon/aws/auth_manager/security_manager/__init__.py
index 7cb45d8389..13a83393a9 100644
--- a/airflow/auth/managers/models/base_user.py
+++ b/airflow/providers/amazon/aws/auth_manager/security_manager/__init__.py
@@ -1,4 +1,3 @@
-#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -15,19 +14,3 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-from __future__ import annotations
-
-from abc import abstractmethod
-
-
-class BaseUser:
- """User model interface."""
-
- @property
- @abstractmethod
- def is_active(self) -> bool:
- ...
-
- @abstractmethod
- def get_id(self) -> str:
- ...
diff --git a/airflow/auth/managers/models/base_user.py b/airflow/providers/amazon/aws/auth_manager/security_manager/aws_security_manager_override.py
similarity index 52%
copy from airflow/auth/managers/models/base_user.py
copy to airflow/providers/amazon/aws/auth_manager/security_manager/aws_security_manager_override.py
index 7cb45d8389..9d6432f1c6 100644
--- a/airflow/auth/managers/models/base_user.py
+++ b/airflow/providers/amazon/aws/auth_manager/security_manager/aws_security_manager_override.py
@@ -1,4 +1,3 @@
-#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -17,17 +16,21 @@
# under the License.
from __future__ import annotations
-from abc import abstractmethod
+from airflow.exceptions import AirflowOptionalProviderFeatureException
+
+try:
+ from airflow.www.security_manager import AirflowSecurityManagerV2
+except ImportError:
+ raise AirflowOptionalProviderFeatureException(
+ "Failed to import AirflowSecurityManagerV2. This feature is only available in Airflow versions >= 2.8.0"
+ )
-class BaseUser:
- """User model interface."""
+class AwsSecurityManagerOverride(AirflowSecurityManagerV2):
+ """The security manager override specific to AWS auth manager."""
- @property
- @abstractmethod
- def is_active(self) -> bool:
- ...
+ def register_views(self):
+ """Register views specific to AWS auth manager."""
+ from airflow.providers.amazon.aws.auth_manager.views.auth import AwsAuthManagerAuthenticationViews
- @abstractmethod
- def get_id(self) -> str:
- ...
+ self.appbuilder.add_view_no_menu(AwsAuthManagerAuthenticationViews())
diff --git a/airflow/providers/amazon/aws/auth_manager/user.py b/airflow/providers/amazon/aws/auth_manager/user.py
new file mode 100644
index 0000000000..68699783d6
--- /dev/null
+++ b/airflow/providers/amazon/aws/auth_manager/user.py
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from airflow.exceptions import AirflowOptionalProviderFeatureException
+
+try:
+ from airflow.auth.managers.models.base_user import BaseUser
+except ImportError:
+ raise AirflowOptionalProviderFeatureException(
+ "Failed to import BaseUser. This feature is only available in Airflow versions >= 2.8.0"
+ )
+
+
+class AwsAuthManagerUser(BaseUser):
+ """
+ User model for users managed by the AWS Auth Manager.
+
+ :param user_id: The user ID.
+ :param groups: The groups the user belongs to.
+ :param username: The username of the user.
+ :param email: The email of the user.
+ """
+
+ def __init__(
+ self, *, user_id: str, groups: list[str], username: str | None = None, email: str | None = None
+ ) -> None:
+ self.user_id = user_id
+ self.groups = groups
+ self.username = username
+ self.email = email
+
+ def get_id(self) -> str:
+ return self.user_id
+
+ def get_name(self) -> str:
+ return self.username or self.email or self.user_id
diff --git a/airflow/auth/managers/models/base_user.py b/airflow/providers/amazon/aws/auth_manager/views/__init__.py
similarity index 75%
copy from airflow/auth/managers/models/base_user.py
copy to airflow/providers/amazon/aws/auth_manager/views/__init__.py
index 7cb45d8389..13a83393a9 100644
--- a/airflow/auth/managers/models/base_user.py
+++ b/airflow/providers/amazon/aws/auth_manager/views/__init__.py
@@ -1,4 +1,3 @@
-#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -15,19 +14,3 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-from __future__ import annotations
-
-from abc import abstractmethod
-
-
-class BaseUser:
- """User model interface."""
-
- @property
- @abstractmethod
- def is_active(self) -> bool:
- ...
-
- @abstractmethod
- def get_id(self) -> str:
- ...
diff --git a/airflow/providers/amazon/aws/auth_manager/views/auth.py b/airflow/providers/amazon/aws/auth_manager/views/auth.py
new file mode 100644
index 0000000000..eb73a3805b
--- /dev/null
+++ b/airflow/providers/amazon/aws/auth_manager/views/auth.py
@@ -0,0 +1,149 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import logging
+from functools import cached_property
+
+from flask import make_response, redirect, request, session, url_for
+from flask_appbuilder import expose
+
+from airflow.configuration import conf
+from airflow.exceptions import AirflowException
+from airflow.providers.amazon.aws.auth_manager.constants import CONF_SAML_METADATA_URL_KEY, CONF_SECTION_NAME
+from airflow.providers.amazon.aws.auth_manager.user import AwsAuthManagerUser
+from airflow.www.app import csrf
+from airflow.www.views import AirflowBaseView
+
+try:
+ from onelogin.saml2.auth import OneLogin_Saml2_Auth
+ from onelogin.saml2.idp_metadata_parser import OneLogin_Saml2_IdPMetadataParser
+except ImportError:
+ raise ImportError(
+ "AWS auth manager requires the python3-saml library but it is not installed by default. "
+ "Please install the python3-saml library by running: "
+ "pip install apache-airflow-providers-amazon[python3-saml]"
+ )
+
+
+class AwsAuthManagerAuthenticationViews(AirflowBaseView):
+ """
+ Views specific to AWS auth manager authentication mechanism.
+
+ Some code below is inspired from
+ https://github.com/SAML-Toolkits/python3-saml/blob/6988bdab7a203abfe8dc264992f7e350c67aef3d/demo-flask/index.py
+ """
+
+ @cached_property
+ def idp_data(self) -> dict:
+ saml_metadata_url = conf.get_mandatory_value(CONF_SECTION_NAME, CONF_SAML_METADATA_URL_KEY)
+ return OneLogin_Saml2_IdPMetadataParser.parse_remote(saml_metadata_url)
+
+ @expose("/login")
+ def login(self):
+ """Start login process."""
+ saml_auth = self._init_saml_auth()
+ return redirect(saml_auth.login())
+
+ @expose("/logout")
+ def logout(self):
+ """Start logout process."""
+ session.clear()
+ saml_auth = self._init_saml_auth()
+
+ return redirect(saml_auth.logout())
+
+ @csrf.exempt
+ @expose("/login_callback", methods=("GET", "POST"))
+ def login_callback(self):
+ """
+ Callback where the user is redirected to after successful login.
+
+ CSRF protection needs to be disabled otherwise the callback won't work.
+ """
+ saml_auth = self._init_saml_auth()
+ saml_auth.process_response()
+ errors = saml_auth.get_errors()
+ is_authenticated = saml_auth.is_authenticated()
+ if not is_authenticated:
+ error_reason = saml_auth.get_last_error_reason()
+ logging.error("Failed to authenticate")
+ logging.error("Errors: %s", errors)
+ logging.error("Error reason: %s", error_reason)
+ raise AirflowException(f"Failed to authenticate: {error_reason}")
+
+ attributes = saml_auth.get_attributes()
+ user = AwsAuthManagerUser(
+ user_id=attributes["id"][0],
+ groups=attributes["groups"],
+ username=saml_auth.get_nameid(),
+ email=attributes["email"][0],
+ )
+ session["aws_user"] = user
+
+ return redirect(url_for("Airflow.index"))
+
+ @csrf.exempt
+ @expose("/logout_callback", methods=("GET", "POST"))
+ def logout_callback(self):
+ raise NotImplementedError("AWS Identity center does not support SLO (Single Logout Service)")
+
+ @expose("/login_metadata")
+ def login_metadata(self):
+ saml_auth = self._init_saml_auth()
+ settings = saml_auth.get_settings()
+ metadata = settings.get_sp_metadata()
+ errors = settings.validate_metadata(metadata)
+
+ if len(errors) == 0:
+ resp = make_response(metadata, 200)
+ resp.headers["Content-Type"] = "text/xml"
+ else:
+ resp = make_response(", ".join(errors), 500)
+ return resp
+
+ @staticmethod
+ def _prepare_flask_request() -> dict:
+ return {
+ "https": "on" if request.scheme == "https" else "off",
+ "http_host": request.host,
+ "script_name": request.path,
+ "get_data": request.args.copy(),
+ "post_data": request.form.copy(),
+ }
+
+ def _init_saml_auth(self) -> OneLogin_Saml2_Auth:
+ request_data = self._prepare_flask_request()
+ base_url = conf.get(section="webserver", key="base_url")
+ settings = {
+ # We want to keep this flag on in case of errors.
+ # It provides an error reasons, if turned off, it does not
+ "debug": True,
+ "sp": {
+ "entityId": f"{base_url}/login_metadata",
+ "assertionConsumerService": {
+ "url": f"{base_url}/login_callback",
+ "binding": "urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST",
+ },
+ "singleLogoutService": {
+ "url": f"{base_url}/logout_callback",
+ "binding": "urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect",
+ },
+ },
+ }
+ merged_settings = OneLogin_Saml2_IdPMetadataParser.merge_settings(settings, self.idp_data)
+ return OneLogin_Saml2_Auth(request_data, merged_settings)
diff --git a/airflow/providers/amazon/provider.yaml b/airflow/providers/amazon/provider.yaml
index 8c0c13fbc6..6399961543 100644
--- a/airflow/providers/amazon/provider.yaml
+++ b/airflow/providers/amazon/provider.yaml
@@ -746,6 +746,9 @@ additional-extras:
- name: s3fs
dependencies:
- s3fs>=2023.9.2
+ - name: python3-saml
+ dependencies:
+ - python3-saml>=1.16.0
config:
aws:
@@ -897,6 +900,28 @@ config:
type: boolean
example: "True"
default: "True"
+ aws_auth_manager:
+ description: |
+ This section only applies if you are using the AwsAuthManager. In other words, if you set
+ ``[core] auth_manager = airflow.providers.amazon.aws.auth_manager.aws_auth_manager.AwsAuthManager`` in
+ Airflow's configuration.
+ options:
+ enable:
+ description: |
+ AWS auth manager is not ready to be used. Turn on this flag to use it anyway.
+ Do that at your own risk since the AWS auth manager is not in an usable state.
+ version_added: 8.12.0
+ type: boolean
+ example: "True"
+ default: "False"
+ saml_metadata_url:
+ description: |
+ SAML metadata XML file provided by AWS Identity Center.
+ This URL can be found in the AWS Identity Center console. Required.
+ version_added: 8.12.0
+ type: string
+ example: "https://portal.sso.<region>.amazonaws.com/saml/metadata/XXXXXXXXXX"
+ default: ~
executors:
- airflow.providers.amazon.aws.executors.ecs.ecs_executor.AwsEcsExecutor
diff --git a/airflow/www/extensions/init_appbuilder.py b/airflow/www/extensions/init_appbuilder.py
index ce87fdc76c..ddb44200f9 100644
--- a/airflow/www/extensions/init_appbuilder.py
+++ b/airflow/www/extensions/init_appbuilder.py
@@ -214,7 +214,7 @@ class AirflowAppBuilder:
self._addon_managers = app.config["ADDON_MANAGERS"]
self.session = session
- auth_manager = init_auth_manager(app, self)
+ auth_manager = init_auth_manager(self)
self.sm = auth_manager.security_manager
self.bm = BabelManager(self)
self._add_global_static()
diff --git a/airflow/www/extensions/init_auth_manager.py b/airflow/www/extensions/init_auth_manager.py
index 63ec043c10..d29fb6944e 100644
--- a/airflow/www/extensions/init_auth_manager.py
+++ b/airflow/www/extensions/init_auth_manager.py
@@ -22,8 +22,6 @@ from airflow.configuration import conf
from airflow.exceptions import AirflowConfigException
if TYPE_CHECKING:
- from flask import Flask
-
from airflow.auth.managers.base_auth_manager import BaseAuthManager
from airflow.www.extensions.init_appbuilder import AirflowAppBuilder
@@ -47,7 +45,7 @@ def get_auth_manager_cls() -> type[BaseAuthManager]:
return auth_manager_cls
-def init_auth_manager(app: Flask, appbuilder: AirflowAppBuilder) -> BaseAuthManager:
+def init_auth_manager(appbuilder: AirflowAppBuilder) -> BaseAuthManager:
"""
Initialize the auth manager.
@@ -55,7 +53,7 @@ def init_auth_manager(app: Flask, appbuilder: AirflowAppBuilder) -> BaseAuthMana
"""
global auth_manager
auth_manager_cls = get_auth_manager_cls()
- auth_manager = auth_manager_cls(app, appbuilder)
+ auth_manager = auth_manager_cls(appbuilder)
return auth_manager
diff --git a/airflow/www/extensions/init_security.py b/airflow/www/extensions/init_security.py
index 21b71e8f84..a7739e3231 100644
--- a/airflow/www/extensions/init_security.py
+++ b/airflow/www/extensions/init_security.py
@@ -19,7 +19,7 @@ from __future__ import annotations
import logging
from importlib import import_module
-from flask import g, redirect, request
+from flask import redirect, request
from airflow.configuration import conf
from airflow.exceptions import AirflowConfigException, AirflowException
@@ -72,5 +72,5 @@ def init_check_user_active(app):
url_logout = get_auth_manager().get_url_logout()
if request.path == url_logout:
return
- if get_auth_manager().is_logged_in() and not g.user.is_active:
+ if get_auth_manager().is_logged_in() and not get_auth_manager().get_user().is_active:
return redirect(url_logout)
diff --git a/airflow/www/security_manager.py b/airflow/www/security_manager.py
index 35e2a63e64..78d8ef00fe 100644
--- a/airflow/www/security_manager.py
+++ b/airflow/www/security_manager.py
@@ -112,6 +112,10 @@ class AirflowSecurityManagerV2(LoggingMixin):
limiter.init_app(self.appbuilder.get_app)
return limiter
+ def register_views(self):
+ """Allow auth managers to register their own views. By default, do nothing."""
+ pass
+
def has_access(
self, action_name: str, resource_name: str, user=None, resource_pk: str | None = None
) -> bool:
@@ -168,18 +172,6 @@ class AirflowSecurityManagerV2(LoggingMixin):
cost=limit.cost,
)(baseview.blueprint)
- def add_permissions_view(self, base_action_names, resource_name):
- raise NotImplementedError("Sync FAB permissions is only available with the FAB auth manager")
-
- def add_permissions_menu(self, resource_name):
- raise NotImplementedError("Sync FAB permissions is only available with the FAB auth manager")
-
- def get_action(self, name: str) -> Action:
- raise NotImplementedError("Only available when FAB auth manager is used")
-
- def get_resource(self, name: str) -> Resource:
- raise NotImplementedError("Only available when FAB auth manager is used")
-
@cached_property
@provide_session
def _auth_manager_is_authorized_map(
@@ -355,3 +347,20 @@ class AirflowSecurityManagerV2(LoggingMixin):
return lambda action, resource_pk, user: any(
self._get_auth_manager_is_authorized_method(fab_resource_name=item) for item in items
)
+
+ """
+ The following methods are specific to FAB auth manager. They still need to be "present" in the main
+ security manager class, but they do nothing.
+ """
+
+ def get_action(self, name: str) -> Action:
+ raise NotImplementedError("Only available if FAB auth manager is used")
+
+ def get_resource(self, name: str) -> Resource:
+ raise NotImplementedError("Only available if FAB auth manager is used")
+
+ def add_permissions_view(self, base_action_names, resource_name):
+ pass
+
+ def add_permissions_menu(self, resource_name):
+ pass
diff --git a/airflow/www/utils.py b/airflow/www/utils.py
index 5fe8a7ccb7..16d0950989 100644
--- a/airflow/www/utils.py
+++ b/airflow/www/utils.py
@@ -940,10 +940,7 @@ class UIAlert:
if self.roles:
current_user = get_auth_manager().get_user()
if current_user is not None:
- if not hasattr(current_user, "roles"):
- # If the user does not contain "roles" in its model, return False
- return False
- user_roles = {r.name for r in current_user.roles}
+ user_roles = {r.name for r in getattr(current_user, "roles", [])}
elif "AUTH_ROLE_PUBLIC" in appbuilder.get_app.config:
# If the current_user is anonymous, assign AUTH_ROLE_PUBLIC role (if it exists) to them
user_roles = {appbuilder.get_app.config["AUTH_ROLE_PUBLIC"]}
diff --git a/docs/apache-airflow/extra-packages-ref.rst b/docs/apache-airflow/extra-packages-ref.rst
index 3060d7c2db..32e6d892bc 100644
--- a/docs/apache-airflow/extra-packages-ref.rst
+++ b/docs/apache-airflow/extra-packages-ref.rst
@@ -70,6 +70,8 @@ python dependencies for the provided package.
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| s3fs | ``pip install 'apache-airflow[s3fs]'`` | Support for S3 as Airflow FS |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
+| saml | ``pip install 'apache-airflow[saml]'`` | Support for SAML authentication in Airflow |
++---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| statsd | ``pip install 'apache-airflow[statsd]'`` | Needed by StatsD metrics |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| virtualenv | ``pip install 'apache-airflow[virtualenv]'`` | Running python tasks in local virtualenv |
diff --git a/docs/docker-stack/changelog.rst b/docs/docker-stack/changelog.rst
index 0f752cb189..ad71d49221 100644
--- a/docs/docker-stack/changelog.rst
+++ b/docs/docker-stack/changelog.rst
@@ -39,6 +39,9 @@ Airflow 2.8
* 2.8.0
+ * Add ``libxmlsec1`` and ``libxmlsec1-dev`` libraries to dev PROD image and ``libxmlsec1`` library to runtime PROD
+ image as it is required by ``python3-saml`` library.
+
* The image is based on ``Debian Bookworm`` in 2.8.0 rather than ``Debian Bullseye``. This might cause some
problems when building custom images. You are advised to make sure your system level dependencies are
working with ``Debian Bookworm``. While all reference images of Airflow 2.8.0 are built on ``Debian Bookworm``,
diff --git a/docs/spelling_wordlist.txt b/docs/spelling_wordlist.txt
index 02380d9884..56198798b4 100644
--- a/docs/spelling_wordlist.txt
+++ b/docs/spelling_wordlist.txt
@@ -1356,6 +1356,7 @@ Sagemaker
sagemaker
salesforce
samesite
+saml
sanitization
sas
Sasl
diff --git a/scripts/docker/install_os_dependencies.sh b/scripts/docker/install_os_dependencies.sh
index cb6cb692fb..318add1435 100644
--- a/scripts/docker/install_os_dependencies.sh
+++ b/scripts/docker/install_os_dependencies.sh
@@ -39,7 +39,7 @@ function get_dev_apt_deps() {
DEV_APT_DEPS="apt-transport-https apt-utils build-essential ca-certificates dirmngr \
freetds-bin freetds-dev git gosu graphviz graphviz-dev krb5-user ldap-utils libffi-dev libgeos-dev \
libkrb5-dev libldap2-dev libleveldb1d libleveldb-dev libsasl2-2 libsasl2-dev libsasl2-modules \
-libssl-dev locales lsb-release openssh-client pkgconf sasl2-bin \
+libssl-dev libxmlsec1 libxmlsec1-dev locales lsb-release openssh-client pkgconf sasl2-bin \
software-properties-common sqlite3 sudo unixodbc unixodbc-dev"
export DEV_APT_DEPS
fi
@@ -65,7 +65,7 @@ function get_runtime_apt_deps() {
if [[ "${RUNTIME_APT_DEPS=}" == "" ]]; then
RUNTIME_APT_DEPS="apt-transport-https apt-utils ca-certificates \
curl dumb-init freetds-bin gosu krb5-user libgeos-dev \
-ldap-utils libsasl2-2 libsasl2-modules locales ${debian_version_apt_deps} \
+ldap-utils libsasl2-2 libsasl2-modules libxmlsec1 locales ${debian_version_apt_deps} \
lsb-release openssh-client python3-selinux rsync sasl2-bin sqlite3 sudo unixodbc"
export RUNTIME_APT_DEPS
fi
diff --git a/setup.py b/setup.py
index 46a7e68c5a..bc624e21fa 100644
--- a/setup.py
+++ b/setup.py
@@ -527,6 +527,11 @@ s3fs = [
"s3fs>=2023.9.2",
]
+saml = [
+ # This is required for support of SAML which might be used by some providers (e.g. Amazon)
+ "python3-saml>=1.16.0",
+]
+
def get_provider_dependencies(provider_name: str) -> list[str]:
if provider_name not in PROVIDER_DEPENDENCIES:
@@ -554,6 +559,7 @@ devel = get_unique_dependency_list(
pandas,
password,
s3fs,
+ saml,
]
)
@@ -600,6 +606,7 @@ CORE_EXTRAS_DEPENDENCIES: dict[str, list[str]] = {
"password": password,
"rabbitmq": rabbitmq,
"s3fs": s3fs,
+ "saml": saml,
"sentry": sentry,
"statsd": statsd,
"virtualenv": virtualenv,
diff --git a/tests/api_connexion/endpoints/test_forward_to_fab_endpoint.py b/tests/api_connexion/endpoints/test_forward_to_fab_endpoint.py
index 5560a4d877..4b8294eb6d 100644
--- a/tests/api_connexion/endpoints/test_forward_to_fab_endpoint.py
+++ b/tests/api_connexion/endpoints/test_forward_to_fab_endpoint.py
@@ -130,7 +130,7 @@ class TestFABforwarding:
class TestFABRoleForwarding(TestFABforwarding):
@mock.patch("airflow.api_connexion.endpoints.forward_to_fab_endpoint.get_auth_manager")
def test_raises_400_if_manager_is_not_fab(self, mock_get_auth_manager):
- mock_get_auth_manager.return_value = BaseAuthManager(self.app, self.app.appbuilder)
+ mock_get_auth_manager.return_value = BaseAuthManager(self.app.appbuilder)
response = self.client.get("api/v1/roles", environ_overrides={"REMOTE_USER": "test"})
assert response.status_code == 400
assert (
diff --git a/tests/auth/managers/fab/test_fab_auth_manager.py b/tests/auth/managers/fab/test_fab_auth_manager.py
index 934293606a..74d20ab13b 100644
--- a/tests/auth/managers/fab/test_fab_auth_manager.py
+++ b/tests/auth/managers/fab/test_fab_auth_manager.py
@@ -61,43 +61,18 @@ IS_AUTHORIZED_METHODS_SIMPLE = {
@pytest.fixture
def auth_manager():
- return FabAuthManager(None, None)
+ return FabAuthManager(None)
@pytest.fixture
def auth_manager_with_appbuilder():
flask_app = Flask(__name__)
appbuilder = init_appbuilder(flask_app)
- return FabAuthManager(flask_app, appbuilder)
+ return FabAuthManager(appbuilder)
@pytest.mark.db_test
class TestFabAuthManager:
- @pytest.mark.parametrize(
- "id,first_name,last_name,username,email,expected",
- [
- (1, "First", "Last", None, None, "1"),
- (1, None, None, None, None, "1"),
- (1, "First", "Last", "user", None, "user"),
- (1, "First", "Last", "user", "email", "user"),
- (1, None, None, None, "email", "email"),
- (1, "First", "Last", None, "email", "email"),
- ],
- )
- @mock.patch.object(FabAuthManager, "get_user")
- def test_get_user_name(
- self, mock_get_user, id, first_name, last_name, username, email, expected, auth_manager
- ):
- user = User()
- user.id = id
- user.first_name = first_name
- user.last_name = last_name
- user.username = username
- user.email = email
- mock_get_user.return_value = user
-
- assert auth_manager.get_user_name() == expected
-
@pytest.mark.parametrize(
"id,first_name,last_name,username,email,expected",
[
@@ -130,15 +105,6 @@ class TestFabAuthManager:
assert auth_manager.get_user() == user
- @mock.patch.object(FabAuthManager, "get_user")
- def test_get_user_id(self, mock_get_user, auth_manager):
- user_id = "test"
- user = Mock()
- user.get_id.return_value = user_id
- mock_get_user.return_value = user
-
- assert auth_manager.get_user_id() == user_id
-
@mock.patch.object(FabAuthManager, "get_user")
def test_is_logged_in(self, mock_get_user, auth_manager):
user = Mock()
diff --git a/tests/auth/managers/test_base_auth_manager.py b/tests/auth/managers/test_base_auth_manager.py
index 9b1b5659f4..832ae50d2a 100644
--- a/tests/auth/managers/test_base_auth_manager.py
+++ b/tests/auth/managers/test_base_auth_manager.py
@@ -45,18 +45,9 @@ if TYPE_CHECKING:
class EmptyAuthManager(BaseAuthManager):
- def get_user_display_name(self) -> str:
- raise NotImplementedError()
-
- def get_user_name(self) -> str:
- raise NotImplementedError()
-
def get_user(self) -> BaseUser:
raise NotImplementedError()
- def get_user_id(self) -> str:
- raise NotImplementedError()
-
def is_authorized_configuration(
self,
*,
@@ -115,20 +106,17 @@ class EmptyAuthManager(BaseAuthManager):
def get_url_logout(self) -> str:
raise NotImplementedError()
- def get_url_user_profile(self) -> str | None:
- raise NotImplementedError()
-
@pytest.fixture
def auth_manager():
- return EmptyAuthManager(None, None)
+ return EmptyAuthManager(None)
@pytest.fixture
def auth_manager_with_appbuilder():
flask_app = Flask(__name__)
appbuilder = init_appbuilder(flask_app)
- return EmptyAuthManager(flask_app, appbuilder)
+ return EmptyAuthManager(appbuilder)
class TestBaseAuthManager:
@@ -138,7 +126,37 @@ class TestBaseAuthManager:
def test_get_api_endpoints_return_none(self, auth_manager):
assert auth_manager.get_api_endpoints() is None
- def test_is_authorized_custom_view_throws_exception(self, auth_manager):
+ def test_get_user_name(self, auth_manager):
+ user = Mock()
+ user.get_name.return_value = "test_username"
+ auth_manager.get_user = MagicMock(return_value=user)
+ result = auth_manager.get_user_name()
+ assert result == "test_username"
+
+ def test_get_user_name_when_not_logged_in(self, auth_manager):
+ auth_manager.get_user = MagicMock(return_value=None)
+ with pytest.raises(AirflowException):
+ auth_manager.get_user_name()
+
+ def test_get_user_display_name_return_user_name(self, auth_manager):
+ auth_manager.get_user_name = MagicMock(return_value="test_user")
+ assert auth_manager.get_user_display_name() == "test_user"
+
+ def test_get_user_id_return_user_id(self, auth_manager):
+ user = Mock()
+ user.get_id = MagicMock(return_value="test_user")
+ auth_manager.get_user = MagicMock(return_value=user)
+ assert auth_manager.get_user_id() == "test_user"
+
+ def test_get_user_id_raise_exception_when_no_user(self, auth_manager):
+ auth_manager.get_user = MagicMock(return_value=None)
+ with pytest.raises(AirflowException, match="The user must be signed in."):
+ auth_manager.get_user_id()
+
+ def test_get_url_user_profile_return_none(self, auth_manager):
+ assert auth_manager.get_url_user_profile() is None
+
+ def test_is_authorized_custom_view_raise_exception(self, auth_manager):
with pytest.raises(AirflowException, match="The resource `.*` does not exist in the environment."):
auth_manager.is_authorized_custom_view(
fab_action_name=permissions.ACTION_CAN_READ,
diff --git a/airflow/auth/managers/models/base_user.py b/tests/providers/amazon/aws/auth_manager/__init__.py
similarity index 75%
copy from airflow/auth/managers/models/base_user.py
copy to tests/providers/amazon/aws/auth_manager/__init__.py
index 7cb45d8389..13a83393a9 100644
--- a/airflow/auth/managers/models/base_user.py
+++ b/tests/providers/amazon/aws/auth_manager/__init__.py
@@ -1,4 +1,3 @@
-#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -15,19 +14,3 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-from __future__ import annotations
-
-from abc import abstractmethod
-
-
-class BaseUser:
- """User model interface."""
-
- @property
- @abstractmethod
- def is_active(self) -> bool:
- ...
-
- @abstractmethod
- def get_id(self) -> str:
- ...
diff --git a/airflow/auth/managers/models/base_user.py b/tests/providers/amazon/aws/auth_manager/security_manager/__init__.py
similarity index 75%
copy from airflow/auth/managers/models/base_user.py
copy to tests/providers/amazon/aws/auth_manager/security_manager/__init__.py
index 7cb45d8389..13a83393a9 100644
--- a/airflow/auth/managers/models/base_user.py
+++ b/tests/providers/amazon/aws/auth_manager/security_manager/__init__.py
@@ -1,4 +1,3 @@
-#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -15,19 +14,3 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-from __future__ import annotations
-
-from abc import abstractmethod
-
-
-class BaseUser:
- """User model interface."""
-
- @property
- @abstractmethod
- def is_active(self) -> bool:
- ...
-
- @abstractmethod
- def get_id(self) -> str:
- ...
diff --git a/tests/providers/amazon/aws/auth_manager/security_manager/test_aws_security_manager_override.py b/tests/providers/amazon/aws/auth_manager/security_manager/test_aws_security_manager_override.py
new file mode 100644
index 0000000000..cfceb3c96c
--- /dev/null
+++ b/tests/providers/amazon/aws/auth_manager/security_manager/test_aws_security_manager_override.py
@@ -0,0 +1,56 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from unittest.mock import Mock, patch
+
+import pytest
+from flask import Flask
+
+from airflow.providers.amazon.aws.auth_manager.security_manager.aws_security_manager_override import (
+ AwsSecurityManagerOverride,
+)
+from airflow.www.extensions.init_appbuilder import init_appbuilder
+
+
+@pytest.fixture
+def appbuilder():
+ flask_app = Flask(__name__)
+ return init_appbuilder(flask_app)
+
+
+@pytest.fixture
+def override(appbuilder):
+ return AwsSecurityManagerOverride(appbuilder)
+
+
+@pytest.mark.db_test
+class TestAwsSecurityManagerOverride:
+ @patch(
+ "airflow.providers.amazon.aws.auth_manager.views.auth.conf.get_mandatory_value", return_value="test"
+ )
+ def test_register_views(self, mock_get_mandatory_value, override, appbuilder):
+ pytest.importorskip("onelogin")
+ from airflow.providers.amazon.aws.auth_manager.views.auth import AwsAuthManagerAuthenticationViews
+
+ with patch.object(AwsAuthManagerAuthenticationViews, "idp_data"):
+ appbuilder.add_view_no_menu = Mock()
+ override.register_views()
+ appbuilder.add_view_no_menu.assert_called_once()
+ assert isinstance(
+ appbuilder.add_view_no_menu.call_args.args[0], AwsAuthManagerAuthenticationViews
+ )
diff --git a/tests/providers/amazon/aws/auth_manager/test_aws_auth_manager.py b/tests/providers/amazon/aws/auth_manager/test_aws_auth_manager.py
new file mode 100644
index 0000000000..56f23eb36a
--- /dev/null
+++ b/tests/providers/amazon/aws/auth_manager/test_aws_auth_manager.py
@@ -0,0 +1,114 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from unittest.mock import patch
+
+import pytest
+from flask import Flask, session
+
+from airflow.providers.amazon.aws.auth_manager.aws_auth_manager import AwsAuthManager
+from airflow.providers.amazon.aws.auth_manager.security_manager.aws_security_manager_override import (
+ AwsSecurityManagerOverride,
+)
+from airflow.providers.amazon.aws.auth_manager.user import AwsAuthManagerUser
+from airflow.www.extensions.init_appbuilder import init_appbuilder
+from tests.test_utils.config import conf_vars
+
+
+@pytest.fixture
+def auth_manager():
+ with conf_vars(
+ {
+ (
+ "core",
+ "auth_manager",
+ ): "airflow.providers.amazon.aws.auth_manager.aws_auth_manager.AwsAuthManager",
+ ("aws_auth_manager", "enable"): "True",
+ }
+ ):
+ return AwsAuthManager(None)
+
+
+@pytest.fixture
+def auth_manager_with_appbuilder():
+ flask_app = Flask(__name__)
+ appbuilder = init_appbuilder(flask_app)
+ with conf_vars(
+ {
+ (
+ "core",
+ "auth_manager",
+ ): "airflow.providers.amazon.aws.auth_manager.aws_auth_manager.AwsAuthManager",
+ ("aws_auth_manager", "enable"): "True",
+ }
+ ):
+ return AwsAuthManager(appbuilder)
+
+
+@pytest.fixture
+def test_user():
+ return AwsAuthManagerUser(user_id="test_user_id", groups=[], username="test_username")
+
+
+class TestAwsAuthManager:
+ @pytest.mark.db_test
+ @patch.object(AwsAuthManager, "is_logged_in")
+ def test_get_user(self, mock_is_logged_in, auth_manager, app, test_user):
+ mock_is_logged_in.return_value = True
+
+ with app.test_request_context():
+ session["aws_user"] = test_user
+ result = auth_manager.get_user()
+
+ assert result == test_user
+
+ @patch.object(AwsAuthManager, "is_logged_in")
+ def test_get_user_return_none_when_not_logged_in(self, mock_is_logged_in, auth_manager):
+ mock_is_logged_in.return_value = False
+ result = auth_manager.get_user()
+
+ assert result is None
+
+ @pytest.mark.db_test
+ def test_is_logged_in(self, auth_manager, app, test_user):
+ with app.test_request_context():
+ session["aws_user"] = test_user
+ result = auth_manager.is_logged_in()
+
+ assert result
+
+ @pytest.mark.db_test
+ def test_is_logged_in_return_false_when_no_user_in_session(self, auth_manager, app, test_user):
+ with app.test_request_context():
+ result = auth_manager.is_logged_in()
+
+ assert result is False
+
+ @patch("airflow.providers.amazon.aws.auth_manager.aws_auth_manager.url_for")
+ def test_get_url_login(self, mock_url_for, auth_manager):
+ auth_manager.get_url_login()
+ mock_url_for.assert_called_once_with("AwsAuthManagerAuthenticationViews.login")
+
+ @patch("airflow.providers.amazon.aws.auth_manager.aws_auth_manager.url_for")
+ def test_get_url_logout(self, mock_url_for, auth_manager):
+ auth_manager.get_url_logout()
+ mock_url_for.assert_called_once_with("AwsAuthManagerAuthenticationViews.logout")
+
+ @pytest.mark.db_test
+ def test_security_manager_return_default_security_manager(self, auth_manager_with_appbuilder):
+ assert isinstance(auth_manager_with_appbuilder.security_manager, AwsSecurityManagerOverride)
diff --git a/airflow/auth/managers/models/base_user.py b/tests/providers/amazon/aws/auth_manager/test_constants.py
similarity index 64%
copy from airflow/auth/managers/models/base_user.py
copy to tests/providers/amazon/aws/auth_manager/test_constants.py
index 7cb45d8389..3b273d9672 100644
--- a/airflow/auth/managers/models/base_user.py
+++ b/tests/providers/amazon/aws/auth_manager/test_constants.py
@@ -1,4 +1,3 @@
-#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -17,17 +16,19 @@
# under the License.
from __future__ import annotations
-from abc import abstractmethod
+from airflow.providers.amazon.aws.auth_manager.constants import (
+ CONF_ENABLE_KEY,
+ CONF_SAML_METADATA_URL_KEY,
+ CONF_SECTION_NAME,
+)
-class BaseUser:
- """User model interface."""
+class TestAwsAuthManagerConstants:
+ def test_conf_section_name(self):
+ assert CONF_SECTION_NAME == "aws_auth_manager"
- @property
- @abstractmethod
- def is_active(self) -> bool:
- ...
+ def test_conf_saml_metadata_url_key(self):
+ assert CONF_SAML_METADATA_URL_KEY == "saml_metadata_url"
- @abstractmethod
- def get_id(self) -> str:
- ...
+ def test_conf_enable_key(self):
+ assert CONF_ENABLE_KEY == "enable"
diff --git a/airflow/auth/managers/models/base_user.py b/tests/providers/amazon/aws/auth_manager/test_user.py
similarity index 55%
copy from airflow/auth/managers/models/base_user.py
copy to tests/providers/amazon/aws/auth_manager/test_user.py
index 7cb45d8389..b518c2a831 100644
--- a/airflow/auth/managers/models/base_user.py
+++ b/tests/providers/amazon/aws/auth_manager/test_user.py
@@ -1,4 +1,3 @@
-#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -17,17 +16,28 @@
# under the License.
from __future__ import annotations
-from abc import abstractmethod
+import pytest
+
+from airflow.providers.amazon.aws.auth_manager.user import AwsAuthManagerUser
+
+
+@pytest.fixture()
+def user():
+ return AwsAuthManagerUser(user_id="user_id", groups=[])
+
+class TestAwsAuthManagerUser:
+ def test_get_id(self, user):
+ assert user.get_id() == "user_id"
-class BaseUser:
- """User model interface."""
+ def test_get_name_with_username(self, user):
+ user.username = "username"
+ assert user.get_name() == "username"
- @property
- @abstractmethod
- def is_active(self) -> bool:
- ...
+ def test_get_name_with_email(self, user):
+ user.email = "email"
+ assert user.get_name() == "email"
- @abstractmethod
- def get_id(self) -> str:
- ...
+ def test_get_name_with_user_id(self, user):
+ user.user_id = "user_id"
+ assert user.get_name() == "user_id"
diff --git a/airflow/auth/managers/models/base_user.py b/tests/providers/amazon/aws/auth_manager/views/__init__.py
similarity index 75%
copy from airflow/auth/managers/models/base_user.py
copy to tests/providers/amazon/aws/auth_manager/views/__init__.py
index 7cb45d8389..13a83393a9 100644
--- a/airflow/auth/managers/models/base_user.py
+++ b/tests/providers/amazon/aws/auth_manager/views/__init__.py
@@ -1,4 +1,3 @@
-#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -15,19 +14,3 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-from __future__ import annotations
-
-from abc import abstractmethod
-
-
-class BaseUser:
- """User model interface."""
-
- @property
- @abstractmethod
- def is_active(self) -> bool:
- ...
-
- @abstractmethod
- def get_id(self) -> str:
- ...
diff --git a/tests/providers/amazon/aws/auth_manager/views/test_auth.py b/tests/providers/amazon/aws/auth_manager/views/test_auth.py
new file mode 100644
index 0000000000..84fdbc373c
--- /dev/null
+++ b/tests/providers/amazon/aws/auth_manager/views/test_auth.py
@@ -0,0 +1,156 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from unittest.mock import Mock, patch
+
+import pytest
+from flask import session, url_for
+
+from airflow.exceptions import AirflowException
+from airflow.www import app as application
+from tests.test_utils.config import conf_vars
+
+pytest.importorskip("onelogin")
+
+SAML_METADATA_URL = "/saml/metadata"
+SAML_METADATA_PARSED = {
+ "idp": {
+ "entityId": "https://portal.sso.us-east-1.amazonaws.com/saml/assertion/<assertion>",
+ "singleSignOnService": {
+ "url": "https://portal.sso.us-east-1.amazonaws.com/saml/assertion/<assertion>",
+ "binding": "urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect",
+ },
+ "singleLogoutService": {
+ "url": "https://portal.sso.us-east-1.amazonaws.com/saml/logout/<assertion>",
+ "binding": "urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect",
+ },
+ "x509cert": "<cert>",
+ },
+ "security": {"authnRequestsSigned": False},
+ "sp": {"NameIDFormat": "urn:oasis:names:tc:SAML:2.0:nameid-format:transient"},
+}
+
+
+@pytest.fixture()
+def aws_app():
+ def factory():
+ with conf_vars(
+ {
+ (
+ "core",
+ "auth_manager",
+ ): "airflow.providers.amazon.aws.auth_manager.aws_auth_manager.AwsAuthManager",
+ ("aws_auth_manager", "enable"): "True",
+ ("aws_auth_manager", "saml_metadata_url"): SAML_METADATA_URL,
+ }
+ ):
+ with patch(
+ "airflow.providers.amazon.aws.auth_manager.views.auth.OneLogin_Saml2_IdPMetadataParser"
+ ) as mock_parser:
+ mock_parser.parse_remote.return_value = SAML_METADATA_PARSED
+ return application.create_app(testing=True)
+
+ return factory()
+
+
+@pytest.mark.db_test
+class TestAwsAuthManagerAuthenticationViews:
+ def test_login_redirect_to_identity_center(self, aws_app):
+ with aws_app.test_client() as client:
+ response = client.get("/login")
+ assert response.status_code == 302
+ assert response.location.startswith("https://portal.sso.us-east-1.amazonaws.com/saml/assertion/")
+
+ def test_logout_redirect_to_identity_center(self, aws_app):
+ with aws_app.test_client() as client:
+ response = client.get("/logout")
+ assert response.status_code == 302
+ assert response.location.startswith("https://portal.sso.us-east-1.amazonaws.com/saml/logout/")
+
+ def test_login_metadata_return_xml_file(self, aws_app):
+ with aws_app.test_client() as client:
+ response = client.get("/login_metadata")
+ assert response.status_code == 200
+ assert response.headers["Content-Type"] == "text/xml"
+
+ def test_login_callback_set_user_in_session(self):
+ with conf_vars(
+ {
+ (
+ "core",
+ "auth_manager",
+ ): "airflow.providers.amazon.aws.auth_manager.aws_auth_manager.AwsAuthManager",
+ ("aws_auth_manager", "enable"): "True",
+ ("aws_auth_manager", "saml_metadata_url"): SAML_METADATA_URL,
+ }
+ ):
+ with patch(
+ "airflow.providers.amazon.aws.auth_manager.views.auth.OneLogin_Saml2_IdPMetadataParser"
+ ) as mock_parser, patch(
+ "airflow.providers.amazon.aws.auth_manager.views.auth.AwsAuthManagerAuthenticationViews._init_saml_auth"
+ ) as mock_init_saml_auth:
+ mock_parser.parse_remote.return_value = SAML_METADATA_PARSED
+
+ auth = Mock()
+ auth.is_authenticated.return_value = True
+ auth.get_nameid.return_value = "user_id"
+ auth.get_attributes.return_value = {
+ "id": ["1"],
+ "groups": ["group_1", "group_2"],
+ "email": ["email"],
+ }
+ mock_init_saml_auth.return_value = auth
+ app = application.create_app(testing=True)
+ with app.test_client() as client:
+ response = client.get("/login_callback")
+ assert response.status_code == 302
+ assert response.location == url_for("Airflow.index")
+ assert session["aws_user"] is not None
+ assert session["aws_user"].get_id() == "1"
+ assert session["aws_user"].get_name() == "user_id"
+
+ def test_login_callback_raise_exception_if_errors(self):
+ with conf_vars(
+ {
+ (
+ "core",
+ "auth_manager",
+ ): "airflow.providers.amazon.aws.auth_manager.aws_auth_manager.AwsAuthManager",
+ ("aws_auth_manager", "enable"): "True",
+ ("aws_auth_manager", "saml_metadata_url"): SAML_METADATA_URL,
+ }
+ ):
+ with patch(
+ "airflow.providers.amazon.aws.auth_manager.views.auth.OneLogin_Saml2_IdPMetadataParser"
+ ) as mock_parser, patch(
+ "airflow.providers.amazon.aws.auth_manager.views.auth.AwsAuthManagerAuthenticationViews._init_saml_auth"
+ ) as mock_init_saml_auth:
+ mock_parser.parse_remote.return_value = SAML_METADATA_PARSED
+
+ auth = Mock()
+ auth.is_authenticated.return_value = False
+ mock_init_saml_auth.return_value = auth
+ app = application.create_app(testing=True)
+ with app.test_client() as client:
+ with pytest.raises(AirflowException):
+ client.get("/login_callback")
+
+ def test_logout_callback_raise_not_implemented_error(self, aws_app):
+ with aws_app.test_client() as client:
+ with pytest.raises(NotImplementedError):
+ client.get("/logout_callback")
(airflow) 06/37: feature(providers): added `OpsgenieNotifier` (#35530)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit bd644fab3e548fa8d1cedc56bf35787772d0f14b
Author: Eric Mendes <42...@users.noreply.github.com>
AuthorDate: Mon Nov 20 17:49:27 2023 -0300
feature(providers): added `OpsgenieNotifier` (#35530)
---------
Co-authored-by: Utkarsh Sharma <ut...@gmail.com>
Co-authored-by: Andrey Anshin <An...@taragol.is>
---
.../providers/opsgenie/notifications/__init__.py | 16 +++
.../providers/opsgenie/notifications/opsgenie.py | 81 ++++++++++++++
airflow/providers/opsgenie/provider.yaml | 3 +
airflow/providers/opsgenie/typing/__init__.py | 16 +++
airflow/providers/opsgenie/typing/opsgenie.py | 59 ++++++++++
docs/apache-airflow-providers-opsgenie/index.rst | 1 +
.../notifications/index.rst | 30 ++++++
.../notifications/opsgenie_notifier.rst | 33 ++++++
tests/providers/opsgenie/notifications/__init__.py | 16 +++
.../opsgenie/notifications/test_opsgenie.py | 119 +++++++++++++++++++++
tests/providers/opsgenie/typing/__init__.py | 16 +++
tests/providers/opsgenie/typing/test_opsgenie.py | 29 +++++
.../opsgenie/example_opsgenie_notifier.py | 42 ++++++++
13 files changed, 461 insertions(+)
diff --git a/airflow/providers/opsgenie/notifications/__init__.py b/airflow/providers/opsgenie/notifications/__init__.py
new file mode 100644
index 0000000000..13a83393a9
--- /dev/null
+++ b/airflow/providers/opsgenie/notifications/__init__.py
@@ -0,0 +1,16 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
diff --git a/airflow/providers/opsgenie/notifications/opsgenie.py b/airflow/providers/opsgenie/notifications/opsgenie.py
new file mode 100644
index 0000000000..950d92939e
--- /dev/null
+++ b/airflow/providers/opsgenie/notifications/opsgenie.py
@@ -0,0 +1,81 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from functools import cached_property
+from typing import TYPE_CHECKING, Sequence
+
+from airflow.exceptions import AirflowOptionalProviderFeatureException
+
+try:
+ from airflow.notifications.basenotifier import BaseNotifier
+except ImportError:
+ raise AirflowOptionalProviderFeatureException(
+ "Failed to import BaseNotifier. This feature is only available in Airflow versions >= 2.6.0"
+ )
+
+from airflow.providers.opsgenie.hooks.opsgenie import OpsgenieAlertHook
+
+if TYPE_CHECKING:
+ from airflow.providers.opsgenie.typing.opsgenie import CreateAlertPayload
+ from airflow.utils.context import Context
+
+
+class OpsgenieNotifier(BaseNotifier):
+ """
+ This notifier allows you to post alerts to Opsgenie.
+
+ Accepts a connection that has an Opsgenie API key as the connection's password.
+ This notifier sets the domain to conn_id.host, and if not set will default
+ to ``https://api.opsgenie.com``.
+
+ Each Opsgenie API key can be pre-configured to a team integration.
+ You can override these defaults in this notifier.
+
+ .. seealso::
+ For more information on how to use this notifier, take a look at the guide:
+ :ref:`howto/notifier:OpsgenieNotifier`
+
+ :param payload: The payload necessary for creating an alert.
+ :param opsgenie_conn_id: Optional. The name of the Opsgenie connection to use. Default conn_id is opsgenie_default
+ """
+
+ template_fields: Sequence[str] = ("payload",)
+
+ def __init__(
+ self,
+ *,
+ payload: CreateAlertPayload,
+ opsgenie_conn_id: str = "opsgenie_default",
+ ) -> None:
+ super().__init__()
+
+ self.payload = payload
+ self.opsgenie_conn_id = opsgenie_conn_id
+
+ @cached_property
+ def hook(self) -> OpsgenieAlertHook:
+ """Opsgenie alert Hook."""
+ return OpsgenieAlertHook(self.opsgenie_conn_id)
+
+ def notify(self, context: Context) -> None:
+ """Call the OpsgenieAlertHook to post message."""
+ self.hook.get_conn().create_alert(self.payload)
+
+
+send_opsgenie_notification = OpsgenieNotifier
diff --git a/airflow/providers/opsgenie/provider.yaml b/airflow/providers/opsgenie/provider.yaml
index d60f55f6d7..c60ca588f9 100644
--- a/airflow/providers/opsgenie/provider.yaml
+++ b/airflow/providers/opsgenie/provider.yaml
@@ -65,3 +65,6 @@ hooks:
connection-types:
- hook-class-name: airflow.providers.opsgenie.hooks.opsgenie.OpsgenieAlertHook
connection-type: opsgenie
+
+notifications:
+ - airflow.providers.opsgenie.notifications.opsgenie.OpsgenieNotifier
diff --git a/airflow/providers/opsgenie/typing/__init__.py b/airflow/providers/opsgenie/typing/__init__.py
new file mode 100644
index 0000000000..13a83393a9
--- /dev/null
+++ b/airflow/providers/opsgenie/typing/__init__.py
@@ -0,0 +1,16 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
diff --git a/airflow/providers/opsgenie/typing/opsgenie.py b/airflow/providers/opsgenie/typing/opsgenie.py
new file mode 100644
index 0000000000..4e6621785d
--- /dev/null
+++ b/airflow/providers/opsgenie/typing/opsgenie.py
@@ -0,0 +1,59 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import TypedDict
+
+from typing_extensions import NotRequired, Required # For compat with Python < 3.11
+
+
+class CreateAlertPayload(TypedDict):
+ """
+ Payload schema for creating an Opsgenie alert.
+
+ :param message: The Message of the Opsgenie alert.
+ :param alias: Client-defined identifier of the alert.
+ :param description: Description field of the alert.
+ :param responders: Teams, users, escalations and schedules that
+ the alert will be routed to send notifications.
+ :param visible_to: Teams and users that the alert will become visible
+ to without sending any notification.
+ :param actions: Custom actions that will be available for the alert.
+ :param tags: Tags of the alert.
+ :param details: Map of key-value pairs to use as custom properties of the alert.
+ :param entity: Entity field of the alert that is
+ generally used to specify which domain alert is related to.
+ :param source: Source field of the alert. Default value is
+ IP address of the incoming request.
+ :param priority: Priority level of the alert. Default value is P3.
+ :param user: Display name of the request owner.
+ :param note: Additional note that will be added while creating the alert.
+ """
+
+ message: Required[str]
+ alias: NotRequired[str | None]
+ description: NotRequired[str | None]
+ responders: NotRequired[list[dict] | None]
+ visible_to: NotRequired[list[dict] | None]
+ actions: NotRequired[list[str] | None]
+ tags: NotRequired[list[str] | None]
+ details: NotRequired[dict | None]
+ entity: NotRequired[str | None]
+ source: NotRequired[str | None]
+ priority: NotRequired[str | None]
+ user: NotRequired[str | None]
+ note: NotRequired[str | None]
diff --git a/docs/apache-airflow-providers-opsgenie/index.rst b/docs/apache-airflow-providers-opsgenie/index.rst
index f1409d7eb5..ec32005261 100644
--- a/docs/apache-airflow-providers-opsgenie/index.rst
+++ b/docs/apache-airflow-providers-opsgenie/index.rst
@@ -35,6 +35,7 @@
:caption: Guides
Operators <operators/index>
+ Notifications <notifications/index>
.. toctree::
:hidden:
diff --git a/docs/apache-airflow-providers-opsgenie/notifications/index.rst b/docs/apache-airflow-providers-opsgenie/notifications/index.rst
new file mode 100644
index 0000000000..15f9b53bcb
--- /dev/null
+++ b/docs/apache-airflow-providers-opsgenie/notifications/index.rst
@@ -0,0 +1,30 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+Opsgenie Notifiers
+====================
+
+.. important:: This feature is only available in Airflow versions >= 2.6.0
+
+
+.. toctree::
+ :maxdepth: 1
+ :glob:
+
+ *
diff --git a/docs/apache-airflow-providers-opsgenie/notifications/opsgenie_notifier.rst b/docs/apache-airflow-providers-opsgenie/notifications/opsgenie_notifier.rst
new file mode 100644
index 0000000000..54cff970d4
--- /dev/null
+++ b/docs/apache-airflow-providers-opsgenie/notifications/opsgenie_notifier.rst
@@ -0,0 +1,33 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _howto/notifier:OpsgenieNotifier:
+
+Opsgenie Alert Notifier
+=======================
+
+Use the :class:`~airflow.providers.opsgenie.notifications.opsgenie.OpsgenieNotifier` to send an alert to opsgenie.
+
+
+Using the Notifier
+^^^^^^^^^^^^^^^^^^
+Send an alert to Opsgenie with a specific message.
+
+.. exampleinclude:: /../../tests/system/providers/opsgenie/example_opsgenie_notifier.py
+ :language: python
+ :start-after: [START howto_notifier_opsgenie]
+ :end-before: [END howto_notifier_opsgenie]
diff --git a/tests/providers/opsgenie/notifications/__init__.py b/tests/providers/opsgenie/notifications/__init__.py
new file mode 100644
index 0000000000..13a83393a9
--- /dev/null
+++ b/tests/providers/opsgenie/notifications/__init__.py
@@ -0,0 +1,16 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
diff --git a/tests/providers/opsgenie/notifications/test_opsgenie.py b/tests/providers/opsgenie/notifications/test_opsgenie.py
new file mode 100644
index 0000000000..d5915c3557
--- /dev/null
+++ b/tests/providers/opsgenie/notifications/test_opsgenie.py
@@ -0,0 +1,119 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from unittest import mock
+
+import pytest
+
+from airflow.operators.empty import EmptyOperator
+from airflow.providers.opsgenie.hooks.opsgenie import OpsgenieAlertHook
+from airflow.providers.opsgenie.notifications.opsgenie import OpsgenieNotifier, send_opsgenie_notification
+
+pytestmark = pytest.mark.db_test
+
+
+class TestOpsgenieNotifier:
+ _config = {
+ "message": "An example alert message",
+ "alias": "Life is too short for no alias",
+ "description": "Every alert needs a description",
+ "responders": [
+ {"id": "4513b7ea-3b91-438f-b7e4-e3e54af9147c", "type": "team"},
+ {"name": "NOC", "type": "team"},
+ {"id": "bb4d9938-c3c2-455d-aaab-727aa701c0d8", "type": "user"},
+ {"username": "trinity@opsgenie.com", "type": "user"},
+ {"id": "aee8a0de-c80f-4515-a232-501c0bc9d715", "type": "escalation"},
+ {"name": "Nightwatch Escalation", "type": "escalation"},
+ {"id": "80564037-1984-4f38-b98e-8a1f662df552", "type": "schedule"},
+ {"name": "First Responders Schedule", "type": "schedule"},
+ ],
+ "visible_to": [
+ {"id": "4513b7ea-3b91-438f-b7e4-e3e54af9147c", "type": "team"},
+ {"name": "rocket_team", "type": "team"},
+ {"id": "bb4d9938-c3c2-455d-aaab-727aa701c0d8", "type": "user"},
+ {"username": "trinity@opsgenie.com", "type": "user"},
+ ],
+ "actions": ["Restart", "AnExampleAction"],
+ "tags": ["OverwriteQuietHours", "Critical"],
+ "details": {"key1": "value1", "key2": "value2"},
+ "entity": "An example entity",
+ "source": "Airflow",
+ "priority": "P1",
+ "user": "Jesse",
+ "note": "Write this down",
+ }
+
+ expected_payload_dict = {
+ "message": _config["message"],
+ "alias": _config["alias"],
+ "description": _config["description"],
+ "responders": _config["responders"],
+ "visible_to": _config["visible_to"],
+ "actions": _config["actions"],
+ "tags": _config["tags"],
+ "details": _config["details"],
+ "entity": _config["entity"],
+ "source": _config["source"],
+ "priority": _config["priority"],
+ "user": _config["user"],
+ "note": _config["note"],
+ }
+
+ @mock.patch.object(OpsgenieAlertHook, "get_conn")
+ def test_notifier(self, mock_opsgenie_alert_hook, dag_maker):
+ with dag_maker("test_notifier") as dag:
+ EmptyOperator(task_id="task1")
+ notifier = send_opsgenie_notification(payload=self._config)
+ notifier({"dag": dag})
+ mock_opsgenie_alert_hook.return_value.create_alert.assert_called_once_with(self.expected_payload_dict)
+
+ @mock.patch.object(OpsgenieAlertHook, "get_conn")
+ def test_notifier_with_notifier_class(self, mock_opsgenie_alert_hook, dag_maker):
+ with dag_maker("test_notifier") as dag:
+ EmptyOperator(task_id="task1")
+ notifier = OpsgenieNotifier(payload=self._config)
+ notifier({"dag": dag})
+ mock_opsgenie_alert_hook.return_value.create_alert.assert_called_once_with(self.expected_payload_dict)
+
+ @mock.patch.object(OpsgenieAlertHook, "get_conn")
+ def test_notifier_templated(self, mock_opsgenie_alert_hook, dag_maker):
+ dag_id = "test_notifier"
+ with dag_maker(dag_id) as dag:
+ EmptyOperator(task_id="task1")
+
+ template_fields = ("message", "alias", "description", "entity", "priority", "note")
+ templated_config = {}
+ for key, value in self._config.items():
+ if key in template_fields:
+ templated_config[key] = value + " {{dag.dag_id}}"
+ else:
+ templated_config[key] = value
+
+ templated_expected_payload_dict = {}
+ for key, value in self.expected_payload_dict.items():
+ if key in template_fields:
+ templated_expected_payload_dict[key] = value + f" {dag_id}"
+ else:
+ templated_expected_payload_dict[key] = value
+
+ notifier = OpsgenieNotifier(payload=templated_config)
+ notifier({"dag": dag})
+ mock_opsgenie_alert_hook.return_value.create_alert.assert_called_once_with(
+ templated_expected_payload_dict
+ )
diff --git a/tests/providers/opsgenie/typing/__init__.py b/tests/providers/opsgenie/typing/__init__.py
new file mode 100644
index 0000000000..13a83393a9
--- /dev/null
+++ b/tests/providers/opsgenie/typing/__init__.py
@@ -0,0 +1,16 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
diff --git a/tests/providers/opsgenie/typing/test_opsgenie.py b/tests/providers/opsgenie/typing/test_opsgenie.py
new file mode 100644
index 0000000000..b782305187
--- /dev/null
+++ b/tests/providers/opsgenie/typing/test_opsgenie.py
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import pytest
+
+from airflow.providers.opsgenie.typing.opsgenie import CreateAlertPayload
+
+pytestmark = pytest.mark.db_test
+
+
+class TestCreateAlertPayload:
+ def test_create_alert_payload_required_keys(self):
+ CreateAlertPayload.__required_keys__ = frozenset({"message"})
diff --git a/tests/system/providers/opsgenie/example_opsgenie_notifier.py b/tests/system/providers/opsgenie/example_opsgenie_notifier.py
new file mode 100644
index 0000000000..8d0847817b
--- /dev/null
+++ b/tests/system/providers/opsgenie/example_opsgenie_notifier.py
@@ -0,0 +1,42 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# [START howto_notifier_opsgenie]
+from __future__ import annotations
+
+from datetime import datetime
+
+from airflow import DAG
+from airflow.operators.bash import BashOperator
+from airflow.providers.opsgenie.notifications.opsgenie import send_opsgenie_notification
+
+with DAG(
+ "opsgenie_notifier",
+ start_date=datetime(2023, 1, 1),
+ on_failure_callback=[send_opsgenie_notification(payload={"message": "Something went wrong!"})],
+) as dag:
+ BashOperator(
+ task_id="mytask",
+ bash_command="fail",
+ on_failure_callback=[send_opsgenie_notification(payload={"message": "Something went wrong!"})],
+ )
+# [END howto_notifier_opsgenie]
+
+from tests.system.utils import get_test_run
+
+# Needed to run the example DAG with pytest (see: tests/system/README.md#run_via_pytest)
+test_run = get_test_run(dag)
(airflow) 36/37: Update version to 2.8.0
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 20cb92e52a9762da3eb1e64cf9c3791b9dadfa38
Author: Ephraim Anierobi <sp...@gmail.com>
AuthorDate: Mon Nov 20 22:48:45 2023 +0100
Update version to 2.8.0
---
README.md | 18 +++++++++---------
airflow/__init__.py | 2 +-
airflow/api_connexion/openapi/v1.yaml | 2 +-
.../logging-monitoring/logging-tasks.rst | 2 +-
.../apache-airflow/installation/supported-versions.rst | 2 +-
docs/docker-stack/README.md | 10 +++++-----
.../docker-examples/customizing/own-requirements.sh | 2 +-
.../extending/add-airflow-configuration/Dockerfile | 2 +-
.../extending/add-apt-packages/Dockerfile | 2 +-
.../extending/add-build-essential-extend/Dockerfile | 2 +-
.../docker-examples/extending/add-providers/Dockerfile | 2 +-
.../extending/add-pypi-packages/Dockerfile | 2 +-
.../extending/add-requirement-packages/Dockerfile | 2 +-
.../extending/custom-providers/Dockerfile | 2 +-
.../extending/embedding-dags/Dockerfile | 2 +-
.../extending/writable-directory/Dockerfile | 2 +-
docs/docker-stack/entrypoint.rst | 18 +++++++++---------
generated/PYPI_README.md | 16 ++++++++--------
scripts/ci/pre_commit/pre_commit_supported_versions.py | 2 +-
19 files changed, 46 insertions(+), 46 deletions(-)
diff --git a/README.md b/README.md
index f68615dcd1..5cc0968245 100644
--- a/README.md
+++ b/README.md
@@ -90,13 +90,13 @@ Airflow is not a streaming solution, but it is often used to process real-time d
Apache Airflow is tested with:
-| | Main version (dev) | Stable version (2.7.3) |
+| | Main version (dev) | Stable version (2.8.0) |
|-------------|------------------------|------------------------------|
| Python | 3.8, 3.9, 3.10, 3.11 | 3.8, 3.9, 3.10, 3.11 |
| Platform | AMD64/ARM64(\*) | AMD64/ARM64(\*) |
-| Kubernetes | 1.25, 1.26, 1.27, 1.28 | 1.24, 1.25, 1.26, 1.27, 1.28 |
-| PostgreSQL | 11, 12, 13, 14, 15, 16 | 11, 12, 13, 14, 15 |
-| MySQL | 8.0, Innovation | 5.7, 8.0 |
+| Kubernetes | 1.25, 1.26, 1.27, 1.28 | 1.25, 1.26, 1.27, 1.28 |
+| PostgreSQL | 12, 13, 14, 15, 16 | 12, 13, 14, 15, 16 |
+| MySQL | 8.0, Innovation | 8.0, Innovation |
| SQLite | 3.15.0+ | 3.15.0+ |
| MSSQL | 2017(\*\*), 2019(\*\*) | 2017(\*\*), 2019(\*\*) |
@@ -175,15 +175,15 @@ them to the appropriate format and workflow that your tool requires.
```bash
-pip install 'apache-airflow==2.7.3' \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.7.3/constraints-3.8.txt"
+pip install 'apache-airflow==2.8.0' \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.8.0/constraints-3.8.txt"
```
2. Installing with extras (i.e., postgres, google)
```bash
-pip install 'apache-airflow[postgres,google]==2.7.3' \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.7.3/constraints-3.8.txt"
+pip install 'apache-airflow[postgres,google]==2.8.0' \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.8.0/constraints-3.8.txt"
```
For information on installing provider packages, check
@@ -288,7 +288,7 @@ Apache Airflow version life cycle:
| Version | Current Patch/Minor | State | First Release | Limited Support | EOL/Terminated |
|-----------|-----------------------|-----------|-----------------|-------------------|------------------|
-| 2 | 2.7.3 | Supported | Dec 17, 2020 | TBD | TBD |
+| 2 | 2.8.0 | Supported | Dec 17, 2020 | TBD | TBD |
| 1.10 | 1.10.15 | EOL | Aug 27, 2018 | Dec 17, 2020 | June 17, 2021 |
| 1.9 | 1.9.0 | EOL | Jan 03, 2018 | Aug 27, 2018 | Aug 27, 2018 |
| 1.8 | 1.8.2 | EOL | Mar 19, 2017 | Jan 03, 2018 | Jan 03, 2018 |
diff --git a/airflow/__init__.py b/airflow/__init__.py
index b63ff1dc05..59117d2950 100644
--- a/airflow/__init__.py
+++ b/airflow/__init__.py
@@ -26,7 +26,7 @@ isort:skip_file
"""
from __future__ import annotations
-__version__ = "2.8.0.dev0"
+__version__ = "2.8.0"
# flake8: noqa: F401
diff --git a/airflow/api_connexion/openapi/v1.yaml b/airflow/api_connexion/openapi/v1.yaml
index e4ae9c776f..a2f1c938ad 100644
--- a/airflow/api_connexion/openapi/v1.yaml
+++ b/airflow/api_connexion/openapi/v1.yaml
@@ -231,7 +231,7 @@ info:
This means that the server encountered an unexpected condition that prevented it from
fulfilling the request.
- version: '2.8.0.dev0'
+ version: '2.8.0'
license:
name: Apache 2.0
url: http://www.apache.org/licenses/LICENSE-2.0.html
diff --git a/docs/apache-airflow/administration-and-deployment/logging-monitoring/logging-tasks.rst b/docs/apache-airflow/administration-and-deployment/logging-monitoring/logging-tasks.rst
index f038004b71..3acc88639e 100644
--- a/docs/apache-airflow/administration-and-deployment/logging-monitoring/logging-tasks.rst
+++ b/docs/apache-airflow/administration-and-deployment/logging-monitoring/logging-tasks.rst
@@ -103,7 +103,7 @@ the example below.
$ airflow info
Apache Airflow
- version | 2.8.0.dev0
+ version | 2.8.0
executor | LocalExecutor
task_logging_handler | airflow.utils.log.file_task_handler.FileTaskHandler
sql_alchemy_conn | postgresql+psycopg2://postgres:airflow@postgres/airflow
diff --git a/docs/apache-airflow/installation/supported-versions.rst b/docs/apache-airflow/installation/supported-versions.rst
index 1364f98e93..c3c1da4f47 100644
--- a/docs/apache-airflow/installation/supported-versions.rst
+++ b/docs/apache-airflow/installation/supported-versions.rst
@@ -29,7 +29,7 @@ Apache Airflow™ version life cycle:
========= ===================== ========= =============== ================= ================
Version Current Patch/Minor State First Release Limited Support EOL/Terminated
========= ===================== ========= =============== ================= ================
-2 2.7.3 Supported Dec 17, 2020 TBD TBD
+2 2.8.0 Supported Dec 17, 2020 TBD TBD
1.10 1.10.15 EOL Aug 27, 2018 Dec 17, 2020 June 17, 2021
1.9 1.9.0 EOL Jan 03, 2018 Aug 27, 2018 Aug 27, 2018
1.8 1.8.2 EOL Mar 19, 2017 Jan 03, 2018 Jan 03, 2018
diff --git a/docs/docker-stack/README.md b/docs/docker-stack/README.md
index d74fa373ba..aaf787b598 100644
--- a/docs/docker-stack/README.md
+++ b/docs/docker-stack/README.md
@@ -31,12 +31,12 @@ Every time a new version of Airflow is released, the images are prepared in the
[apache/airflow DockerHub](https://hub.docker.com/r/apache/airflow)
for all the supported Python versions.
-You can find the following images there (Assuming Airflow version `2.8.0.dev0`):
+You can find the following images there (Assuming Airflow version `2.8.0`):
* `apache/airflow:latest` - the latest released Airflow image with default Python version (3.8 currently)
* `apache/airflow:latest-pythonX.Y` - the latest released Airflow image with specific Python version
-* `apache/airflow:2.8.0.dev0` - the versioned Airflow image with default Python version (3.8 currently)
-* `apache/airflow:2.8.0.dev0-pythonX.Y` - the versioned Airflow image with specific Python version
+* `apache/airflow:2.8.0` - the versioned Airflow image with default Python version (3.8 currently)
+* `apache/airflow:2.8.0-pythonX.Y` - the versioned Airflow image with specific Python version
Those are "reference" regular images. They contain the most common set of extras, dependencies and providers that are
often used by the users and they are good to "try-things-out" when you want to just take Airflow for a spin,
@@ -47,8 +47,8 @@ via [Building the image](https://airflow.apache.org/docs/docker-stack/build.html
* `apache/airflow:slim-latest` - the latest released Airflow image with default Python version (3.8 currently)
* `apache/airflow:slim-latest-pythonX.Y` - the latest released Airflow image with specific Python version
-* `apache/airflow:slim-2.8.0.dev0` - the versioned Airflow image with default Python version (3.8 currently)
-* `apache/airflow:slim-2.8.0.dev0-pythonX.Y` - the versioned Airflow image with specific Python version
+* `apache/airflow:slim-2.8.0` - the versioned Airflow image with default Python version (3.8 currently)
+* `apache/airflow:slim-2.8.0-pythonX.Y` - the versioned Airflow image with specific Python version
The Apache Airflow image provided as convenience package is optimized for size, and
it provides just a bare minimal set of the extras and dependencies installed and in most cases
diff --git a/docs/docker-stack/docker-examples/customizing/own-requirements.sh b/docs/docker-stack/docker-examples/customizing/own-requirements.sh
index 53c1f68ffd..e465903e67 100755
--- a/docs/docker-stack/docker-examples/customizing/own-requirements.sh
+++ b/docs/docker-stack/docker-examples/customizing/own-requirements.sh
@@ -28,7 +28,7 @@ mkdir -p docker-context-files
cat <<EOF >./docker-context-files/requirements.txt
beautifulsoup4==4.10.0
-apache-airflow==2.8.0.dev0
+apache-airflow==2.8.0
EOF
export DOCKER_BUILDKIT=1
diff --git a/docs/docker-stack/docker-examples/extending/add-airflow-configuration/Dockerfile b/docs/docker-stack/docker-examples/extending/add-airflow-configuration/Dockerfile
index 9578b3774a..9cc341d7e1 100644
--- a/docs/docker-stack/docker-examples/extending/add-airflow-configuration/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/add-airflow-configuration/Dockerfile
@@ -15,7 +15,7 @@
# This is an example Dockerfile. It is not intended for PRODUCTION use
# [START Dockerfile]
-FROM apache/airflow:2.8.0.dev0
+FROM apache/airflow:2.8.0
ENV AIRFLOW__CORE__LOAD_EXAMPLES=True
ENV AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=my_conn_string
# [END Dockerfile]
diff --git a/docs/docker-stack/docker-examples/extending/add-apt-packages/Dockerfile b/docs/docker-stack/docker-examples/extending/add-apt-packages/Dockerfile
index 4ea0606d86..9de8803e45 100644
--- a/docs/docker-stack/docker-examples/extending/add-apt-packages/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/add-apt-packages/Dockerfile
@@ -15,7 +15,7 @@
# This is an example Dockerfile. It is not intended for PRODUCTION use
# [START Dockerfile]
-FROM apache/airflow:2.8.0.dev0
+FROM apache/airflow:2.8.0
USER root
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
diff --git a/docs/docker-stack/docker-examples/extending/add-build-essential-extend/Dockerfile b/docs/docker-stack/docker-examples/extending/add-build-essential-extend/Dockerfile
index eeb482fe4d..9978c595fa 100644
--- a/docs/docker-stack/docker-examples/extending/add-build-essential-extend/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/add-build-essential-extend/Dockerfile
@@ -15,7 +15,7 @@
# This is an example Dockerfile. It is not intended for PRODUCTION use
# [START Dockerfile]
-FROM apache/airflow:2.8.0.dev0
+FROM apache/airflow:2.8.0
USER root
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
diff --git a/docs/docker-stack/docker-examples/extending/add-providers/Dockerfile b/docs/docker-stack/docker-examples/extending/add-providers/Dockerfile
index 2c5dd70c61..f1496652f3 100644
--- a/docs/docker-stack/docker-examples/extending/add-providers/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/add-providers/Dockerfile
@@ -15,7 +15,7 @@
# This is an example Dockerfile. It is not intended for PRODUCTION use
# [START Dockerfile]
-FROM apache/airflow:2.8.0.dev0
+FROM apache/airflow:2.8.0
USER root
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
diff --git a/docs/docker-stack/docker-examples/extending/add-pypi-packages/Dockerfile b/docs/docker-stack/docker-examples/extending/add-pypi-packages/Dockerfile
index 7ad13ef838..b61b72d08c 100644
--- a/docs/docker-stack/docker-examples/extending/add-pypi-packages/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/add-pypi-packages/Dockerfile
@@ -15,6 +15,6 @@
# This is an example Dockerfile. It is not intended for PRODUCTION use
# [START Dockerfile]
-FROM apache/airflow:2.8.0.dev0
+FROM apache/airflow:2.8.0
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" lxml
# [END Dockerfile]
diff --git a/docs/docker-stack/docker-examples/extending/add-requirement-packages/Dockerfile b/docs/docker-stack/docker-examples/extending/add-requirement-packages/Dockerfile
index e68e4871dd..72a4947d11 100644
--- a/docs/docker-stack/docker-examples/extending/add-requirement-packages/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/add-requirement-packages/Dockerfile
@@ -15,7 +15,7 @@
# This is an example Dockerfile. It is not intended for PRODUCTION use
# [START Dockerfile]
-FROM apache/airflow:2.8.0.dev0
+FROM apache/airflow:2.8.0
COPY requirements.txt /
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" -r /requirements.txt
# [END Dockerfile]
diff --git a/docs/docker-stack/docker-examples/extending/custom-providers/Dockerfile b/docs/docker-stack/docker-examples/extending/custom-providers/Dockerfile
index 5c48f640a1..ab22cb382c 100644
--- a/docs/docker-stack/docker-examples/extending/custom-providers/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/custom-providers/Dockerfile
@@ -15,6 +15,6 @@
# This is an example Dockerfile. It is not intended for PRODUCTION use
# [START Dockerfile]
-FROM apache/airflow:2.8.0.dev0
+FROM apache/airflow:2.8.0
RUN pip install "apache-airflow==${AIRFLOW_VERSION}" --no-cache-dir apache-airflow-providers-docker==2.5.1
# [END Dockerfile]
diff --git a/docs/docker-stack/docker-examples/extending/embedding-dags/Dockerfile b/docs/docker-stack/docker-examples/extending/embedding-dags/Dockerfile
index ef4ff8a025..8a12ff2864 100644
--- a/docs/docker-stack/docker-examples/extending/embedding-dags/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/embedding-dags/Dockerfile
@@ -15,7 +15,7 @@
# This is an example Dockerfile. It is not intended for PRODUCTION use
# [START Dockerfile]
-FROM apache/airflow:2.8.0.dev0
+FROM apache/airflow:2.8.0
COPY --chown=airflow:root test_dag.py /opt/airflow/dags
diff --git a/docs/docker-stack/docker-examples/extending/writable-directory/Dockerfile b/docs/docker-stack/docker-examples/extending/writable-directory/Dockerfile
index 6839e72301..68cd47b0d4 100644
--- a/docs/docker-stack/docker-examples/extending/writable-directory/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/writable-directory/Dockerfile
@@ -15,7 +15,7 @@
# This is an example Dockerfile. It is not intended for PRODUCTION use
# [START Dockerfile]
-FROM apache/airflow:2.8.0.dev0
+FROM apache/airflow:2.8.0
RUN umask 0002; \
mkdir -p ~/writeable-directory
# [END Dockerfile]
diff --git a/docs/docker-stack/entrypoint.rst b/docs/docker-stack/entrypoint.rst
index 59f9b730f4..11d63c87bd 100644
--- a/docs/docker-stack/entrypoint.rst
+++ b/docs/docker-stack/entrypoint.rst
@@ -132,7 +132,7 @@ if you specify extra arguments. For example:
.. code-block:: bash
- docker run -it apache/airflow:2.8.0.dev0-python3.8 bash -c "ls -la"
+ docker run -it apache/airflow:2.8.0-python3.8 bash -c "ls -la"
total 16
drwxr-xr-x 4 airflow root 4096 Jun 5 18:12 .
drwxr-xr-x 1 root root 4096 Jun 5 18:12 ..
@@ -144,7 +144,7 @@ you pass extra parameters. For example:
.. code-block:: bash
- > docker run -it apache/airflow:2.8.0.dev0-python3.8 python -c "print('test')"
+ > docker run -it apache/airflow:2.8.0-python3.8 python -c "print('test')"
test
If first argument equals to "airflow" - the rest of the arguments is treated as an airflow command
@@ -152,13 +152,13 @@ to execute. Example:
.. code-block:: bash
- docker run -it apache/airflow:2.8.0.dev0-python3.8 airflow webserver
+ docker run -it apache/airflow:2.8.0-python3.8 airflow webserver
If there are any other arguments - they are simply passed to the "airflow" command
.. code-block:: bash
- > docker run -it apache/airflow:2.8.0.dev0-python3.8 help
+ > docker run -it apache/airflow:2.8.0-python3.8 help
usage: airflow [-h] GROUP_OR_COMMAND ...
positional arguments:
@@ -206,7 +206,7 @@ propagation (See the next chapter).
.. code-block:: Dockerfile
- FROM airflow:2.8.0.dev0
+ FROM airflow:2.8.0
COPY my_entrypoint.sh /
ENTRYPOINT ["/usr/bin/dumb-init", "--", "/my_entrypoint.sh"]
@@ -250,7 +250,7 @@ Similarly to custom entrypoint, it can be added to the image by extending it.
.. code-block:: Dockerfile
- FROM airflow:2.8.0.dev0
+ FROM airflow:2.8.0
COPY my_after_entrypoint_script.sh /
Build your image and then you can run this script by running the command:
@@ -363,7 +363,7 @@ database and creating an ``admin/admin`` Admin user with the following command:
--env "_AIRFLOW_DB_MIGRATE=true" \
--env "_AIRFLOW_WWW_USER_CREATE=true" \
--env "_AIRFLOW_WWW_USER_PASSWORD=admin" \
- apache/airflow:2.8.0.dev0-python3.8 webserver
+ apache/airflow:2.8.0-python3.8 webserver
.. code-block:: bash
@@ -372,7 +372,7 @@ database and creating an ``admin/admin`` Admin user with the following command:
--env "_AIRFLOW_DB_MIGRATE=true" \
--env "_AIRFLOW_WWW_USER_CREATE=true" \
--env "_AIRFLOW_WWW_USER_PASSWORD_CMD=echo admin" \
- apache/airflow:2.8.0.dev0-python3.8 webserver
+ apache/airflow:2.8.0-python3.8 webserver
The commands above perform initialization of the SQLite database, create admin user with admin password
and Admin role. They also forward local port ``8080`` to the webserver port and finally start the webserver.
@@ -412,6 +412,6 @@ Example:
--env "_AIRFLOW_DB_MIGRATE=true" \
--env "_AIRFLOW_WWW_USER_CREATE=true" \
--env "_AIRFLOW_WWW_USER_PASSWORD_CMD=echo admin" \
- apache/airflow:2.8.0.dev0-python3.8 webserver
+ apache/airflow:2.8.0-python3.8 webserver
This method is only available starting from Docker image of Airflow 2.1.1 and above.
diff --git a/generated/PYPI_README.md b/generated/PYPI_README.md
index 16eda24247..c8d3d58a3d 100644
--- a/generated/PYPI_README.md
+++ b/generated/PYPI_README.md
@@ -47,13 +47,13 @@ Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The
Apache Airflow is tested with:
-| | Main version (dev) | Stable version (2.7.3) |
+| | Main version (dev) | Stable version (2.8.0) |
|-------------|------------------------|------------------------------|
| Python | 3.8, 3.9, 3.10, 3.11 | 3.8, 3.9, 3.10, 3.11 |
| Platform | AMD64/ARM64(\*) | AMD64/ARM64(\*) |
-| Kubernetes | 1.25, 1.26, 1.27, 1.28 | 1.24, 1.25, 1.26, 1.27, 1.28 |
-| PostgreSQL | 11, 12, 13, 14, 15, 16 | 11, 12, 13, 14, 15 |
-| MySQL | 8.0, Innovation | 5.7, 8.0 |
+| Kubernetes | 1.25, 1.26, 1.27, 1.28 | 1.25, 1.26, 1.27, 1.28 |
+| PostgreSQL | 12, 13, 14, 15, 16 | 12, 13, 14, 15, 16 |
+| MySQL | 8.0, Innovation | 8.0, Innovation |
| SQLite | 3.15.0+ | 3.15.0+ |
| MSSQL | 2017(\*\*), 2019(\*\*) | 2017(\*\*), 2019(\*\*) |
@@ -128,15 +128,15 @@ them to the appropriate format and workflow that your tool requires.
```bash
-pip install 'apache-airflow==2.7.3' \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.7.3/constraints-3.8.txt"
+pip install 'apache-airflow==2.8.0' \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.8.0/constraints-3.8.txt"
```
2. Installing with extras (i.e., postgres, google)
```bash
-pip install 'apache-airflow[postgres,google]==2.7.3' \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.7.3/constraints-3.8.txt"
+pip install 'apache-airflow[postgres,google]==2.8.0' \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.8.0/constraints-3.8.txt"
```
For information on installing provider packages, check
diff --git a/scripts/ci/pre_commit/pre_commit_supported_versions.py b/scripts/ci/pre_commit/pre_commit_supported_versions.py
index dcd8b89b45..9acf012a4f 100755
--- a/scripts/ci/pre_commit/pre_commit_supported_versions.py
+++ b/scripts/ci/pre_commit/pre_commit_supported_versions.py
@@ -27,7 +27,7 @@ AIRFLOW_SOURCES = Path(__file__).resolve().parent.parent.parent.parent
HEADERS = ("Version", "Current Patch/Minor", "State", "First Release", "Limited Support", "EOL/Terminated")
SUPPORTED_VERSIONS = (
- ("2", "2.7.3", "Supported", "Dec 17, 2020", "TBD", "TBD"),
+ ("2", "2.8.0", "Supported", "Dec 17, 2020", "TBD", "TBD"),
("1.10", "1.10.15", "EOL", "Aug 27, 2018", "Dec 17, 2020", "June 17, 2021"),
("1.9", "1.9.0", "EOL", "Jan 03, 2018", "Aug 27, 2018", "Aug 27, 2018"),
("1.8", "1.8.2", "EOL", "Mar 19, 2017", "Jan 03, 2018", "Jan 03, 2018"),
(airflow) 29/37: Added retry strategy parameter to Amazon AWS provider Batch Operator to allow dynamic Batch retry strategies (#35789)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 0e6b5d8f8e71df326d237c2028940d5cbf3bac2c
Author: Evgeny <ev...@users.noreply.github.com>
AuthorDate: Wed Nov 22 04:26:42 2023 -0500
Added retry strategy parameter to Amazon AWS provider Batch Operator to allow dynamic Batch retry strategies (#35789)
---
airflow/providers/amazon/aws/operators/batch.py | 5 +++++
tests/providers/amazon/aws/operators/test_batch.py | 6 ++++++
2 files changed, 11 insertions(+)
diff --git a/airflow/providers/amazon/aws/operators/batch.py b/airflow/providers/amazon/aws/operators/batch.py
index 7cf2cbfeac..e917d1d81d 100644
--- a/airflow/providers/amazon/aws/operators/batch.py
+++ b/airflow/providers/amazon/aws/operators/batch.py
@@ -112,6 +112,7 @@ class BatchOperator(BaseOperator):
"array_properties",
"node_overrides",
"parameters",
+ "retry_strategy",
"waiters",
"tags",
"wait_for_completion",
@@ -122,6 +123,7 @@ class BatchOperator(BaseOperator):
"container_overrides": "json",
"parameters": "json",
"node_overrides": "json",
+ "retry_strategy": "json",
}
@property
@@ -160,6 +162,7 @@ class BatchOperator(BaseOperator):
share_identifier: str | None = None,
scheduling_priority_override: int | None = None,
parameters: dict | None = None,
+ retry_strategy: dict | None = None,
job_id: str | None = None,
waiters: Any | None = None,
max_retries: int = 4200,
@@ -201,6 +204,7 @@ class BatchOperator(BaseOperator):
self.scheduling_priority_override = scheduling_priority_override
self.array_properties = array_properties
self.parameters = parameters or {}
+ self.retry_strategy = retry_strategy or {}
self.waiters = waiters
self.tags = tags or {}
self.wait_for_completion = wait_for_completion
@@ -287,6 +291,7 @@ class BatchOperator(BaseOperator):
"tags": self.tags,
"containerOverrides": self.container_overrides,
"nodeOverrides": self.node_overrides,
+ "retryStrategy": self.retry_strategy,
"shareIdentifier": self.share_identifier,
"schedulingPriorityOverride": self.scheduling_priority_override,
}
diff --git a/tests/providers/amazon/aws/operators/test_batch.py b/tests/providers/amazon/aws/operators/test_batch.py
index 8eb6601dfd..8a0d0e788a 100644
--- a/tests/providers/amazon/aws/operators/test_batch.py
+++ b/tests/providers/amazon/aws/operators/test_batch.py
@@ -63,6 +63,7 @@ class TestBatchOperator:
max_retries=self.MAX_RETRIES,
status_retries=self.STATUS_RETRIES,
parameters=None,
+ retry_strategy=None,
container_overrides={},
array_properties=None,
aws_conn_id="airflow_test",
@@ -96,6 +97,7 @@ class TestBatchOperator:
assert self.batch.hook.max_retries == self.MAX_RETRIES
assert self.batch.hook.status_retries == self.STATUS_RETRIES
assert self.batch.parameters == {}
+ assert self.batch.retry_strategy == {}
assert self.batch.container_overrides == {}
assert self.batch.array_properties is None
assert self.batch.node_overrides is None
@@ -119,6 +121,7 @@ class TestBatchOperator:
"array_properties",
"node_overrides",
"parameters",
+ "retry_strategy",
"waiters",
"tags",
"wait_for_completion",
@@ -143,6 +146,7 @@ class TestBatchOperator:
containerOverrides={},
jobDefinition="hello-world",
parameters={},
+ retryStrategy={},
tags={},
)
@@ -166,6 +170,7 @@ class TestBatchOperator:
containerOverrides={},
jobDefinition="hello-world",
parameters={},
+ retryStrategy={},
tags={},
)
@@ -232,6 +237,7 @@ class TestBatchOperator:
"jobName": JOB_NAME,
"jobDefinition": "hello-world",
"parameters": {},
+ "retryStrategy": {},
"tags": {},
}
if override == "overrides":
(airflow) 21/37: Remove --force-build command in cache steps in CI (#35784)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 528d2bc51be862950ff86e815c33fdd78c71c175
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Tue Nov 21 19:40:02 2023 +0100
Remove --force-build command in cache steps in CI (#35784)
The `--force-build` command has not been used for a long time
in the `ci-image build` command for quite some time because the
sheer fact that you run the command means that you want to run
the build (so it made no sense to have it). The command only makes
sense when you want to force build when running `breeze` or
`breeze start-airflow` command.
However commands to build cache in CI still had this command used
and it caused the builds to fail after merging #35768 when the
option has been removed (it was not detected there, because
cache steps only run in main build)
---
.github/workflows/ci.yml | 2 --
1 file changed, 2 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 45e0a9fca0..1e8eabc7a5 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -232,7 +232,6 @@ jobs:
--builder airflow_cache
--prepare-buildx-cache
--run-in-parallel
- --force-build
--platform ${{ matrix.platform }}
env:
DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
@@ -1890,7 +1889,6 @@ jobs:
--builder airflow_cache
--prepare-buildx-cache
--run-in-parallel
- --force-build
--platform ${{ matrix.platform }}
env:
DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
(airflow) 22/37: Fix DataFusion example type annotations (#35753)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 599189e41e27a3b30a2e2d3f598f9b45a96b8547
Author: Andrey Anshin <An...@taragol.is>
AuthorDate: Tue Nov 21 23:44:09 2023 +0400
Fix DataFusion example type annotations (#35753)
---
tests/system/providers/google/cloud/datafusion/example_datafusion.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tests/system/providers/google/cloud/datafusion/example_datafusion.py b/tests/system/providers/google/cloud/datafusion/example_datafusion.py
index 38f443a706..506323dccb 100644
--- a/tests/system/providers/google/cloud/datafusion/example_datafusion.py
+++ b/tests/system/providers/google/cloud/datafusion/example_datafusion.py
@@ -215,7 +215,7 @@ with DAG(
# [END howto_cloud_data_fusion_update_instance_operator]
@task(task_id="get_artifacts_versions")
- def get_artifacts_versions(ti) -> dict:
+ def get_artifacts_versions(ti=None):
hook = DataFusionHook()
instance_url = ti.xcom_pull(task_ids="get_instance", key="return_value")["apiEndpoint"]
artifacts = hook.get_instance_artifacts(instance_url=instance_url, namespace="default")
@@ -319,7 +319,7 @@ with DAG(
# TEST BODY
>> create_instance
>> get_instance
- >> get_artifacts_versions() # type: ignore[call-arg]
+ >> get_artifacts_versions()
>> restart_instance
>> update_instance
>> create_pipeline
(airflow) 04/37: OpenLineage integration tried to use non-existed method in SnowflakeHook (#35752)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit b88d0d7ce2944b5ed3c04d30fffcac37eaaa1859
Author: Andrey Anshin <An...@taragol.is>
AuthorDate: Mon Nov 20 19:55:30 2023 +0400
OpenLineage integration tried to use non-existed method in SnowflakeHook (#35752)
---
airflow/providers/snowflake/hooks/snowflake.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/airflow/providers/snowflake/hooks/snowflake.py b/airflow/providers/snowflake/hooks/snowflake.py
index f7f7a9a16f..081d734ac3 100644
--- a/airflow/providers/snowflake/hooks/snowflake.py
+++ b/airflow/providers/snowflake/hooks/snowflake.py
@@ -480,7 +480,7 @@ class SnowflakeHook(DbApiHook):
from airflow.providers.openlineage.sqlparser import SQLParser
connection = self.get_connection(getattr(self, self.conn_name_attr))
- namespace = SQLParser.create_namespace(self.get_database_info(connection))
+ namespace = SQLParser.create_namespace(self.get_openlineage_database_info(connection))
if self.query_ids:
return OperatorLineage(
(airflow) 05/37: Remove usage of deprecated method from BigQueryToBigQueryOperator (#35605)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 25990d159c1f2350a2ee678d719251da888f27cd
Author: Maksim <ma...@google.com>
AuthorDate: Mon Nov 20 21:12:02 2023 +0100
Remove usage of deprecated method from BigQueryToBigQueryOperator (#35605)
---
.../google/cloud/transfers/bigquery_to_bigquery.py | 88 +++++++++++++++-----
.../cloud/transfers/test_bigquery_to_bigquery.py | 94 ++++++++++++++--------
2 files changed, 127 insertions(+), 55 deletions(-)
diff --git a/airflow/providers/google/cloud/transfers/bigquery_to_bigquery.py b/airflow/providers/google/cloud/transfers/bigquery_to_bigquery.py
index a54ced136f..671ad33488 100644
--- a/airflow/providers/google/cloud/transfers/bigquery_to_bigquery.py
+++ b/airflow/providers/google/cloud/transfers/bigquery_to_bigquery.py
@@ -18,10 +18,8 @@
"""This module contains Google BigQuery to BigQuery operator."""
from __future__ import annotations
-import warnings
from typing import TYPE_CHECKING, Sequence
-from airflow.exceptions import AirflowProviderDeprecationWarning
from airflow.models import BaseOperator
from airflow.providers.google.cloud.hooks.bigquery import BigQueryHook
from airflow.providers.google.cloud.links.bigquery import BigQueryTableLink
@@ -110,6 +108,58 @@ class BigQueryToBigQueryOperator(BaseOperator):
self.encryption_configuration = encryption_configuration
self.location = location
self.impersonation_chain = impersonation_chain
+ self.hook: BigQueryHook | None = None
+
+ def _prepare_job_configuration(self):
+ self.source_project_dataset_tables = (
+ [self.source_project_dataset_tables]
+ if not isinstance(self.source_project_dataset_tables, list)
+ else self.source_project_dataset_tables
+ )
+
+ source_project_dataset_tables_fixup = []
+ for source_project_dataset_table in self.source_project_dataset_tables:
+ source_project, source_dataset, source_table = self.hook.split_tablename(
+ table_input=source_project_dataset_table,
+ default_project_id=self.hook.project_id,
+ var_name="source_project_dataset_table",
+ )
+ source_project_dataset_tables_fixup.append(
+ {"projectId": source_project, "datasetId": source_dataset, "tableId": source_table}
+ )
+
+ destination_project, destination_dataset, destination_table = self.hook.split_tablename(
+ table_input=self.destination_project_dataset_table,
+ default_project_id=self.hook.project_id,
+ )
+ configuration = {
+ "copy": {
+ "createDisposition": self.create_disposition,
+ "writeDisposition": self.write_disposition,
+ "sourceTables": source_project_dataset_tables_fixup,
+ "destinationTable": {
+ "projectId": destination_project,
+ "datasetId": destination_dataset,
+ "tableId": destination_table,
+ },
+ }
+ }
+
+ if self.labels:
+ configuration["labels"] = self.labels
+
+ if self.encryption_configuration:
+ configuration["copy"]["destinationEncryptionConfiguration"] = self.encryption_configuration
+
+ return configuration
+
+ def _submit_job(
+ self,
+ hook: BigQueryHook,
+ configuration: dict,
+ ) -> str:
+ job = hook.insert_job(configuration=configuration, project_id=hook.project_id)
+ return job.job_id
def execute(self, context: Context) -> None:
self.log.info(
@@ -122,24 +172,20 @@ class BigQueryToBigQueryOperator(BaseOperator):
location=self.location,
impersonation_chain=self.impersonation_chain,
)
+ self.hook = hook
- with warnings.catch_warnings():
- warnings.simplefilter("ignore", AirflowProviderDeprecationWarning)
- job_id = hook.run_copy(
- source_project_dataset_tables=self.source_project_dataset_tables,
- destination_project_dataset_table=self.destination_project_dataset_table,
- write_disposition=self.write_disposition,
- create_disposition=self.create_disposition,
- labels=self.labels,
- encryption_configuration=self.encryption_configuration,
- )
+ if not hook.project_id:
+ raise ValueError("The project_id should be set")
- job = hook.get_job(job_id=job_id, location=self.location).to_api_repr()
- conf = job["configuration"]["copy"]["destinationTable"]
- BigQueryTableLink.persist(
- context=context,
- task_instance=self,
- dataset_id=conf["datasetId"],
- project_id=conf["projectId"],
- table_id=conf["tableId"],
- )
+ configuration = self._prepare_job_configuration()
+ job_id = self._submit_job(hook=hook, configuration=configuration)
+
+ job = hook.get_job(job_id=job_id, location=self.location).to_api_repr()
+ conf = job["configuration"]["copy"]["destinationTable"]
+ BigQueryTableLink.persist(
+ context=context,
+ task_instance=self,
+ dataset_id=conf["datasetId"],
+ project_id=conf["projectId"],
+ table_id=conf["tableId"],
+ )
diff --git a/tests/providers/google/cloud/transfers/test_bigquery_to_bigquery.py b/tests/providers/google/cloud/transfers/test_bigquery_to_bigquery.py
index 5c0f8f560c..ed06928c2c 100644
--- a/tests/providers/google/cloud/transfers/test_bigquery_to_bigquery.py
+++ b/tests/providers/google/cloud/transfers/test_bigquery_to_bigquery.py
@@ -23,64 +23,90 @@ from airflow.providers.google.cloud.transfers.bigquery_to_bigquery import BigQue
BQ_HOOK_PATH = "airflow.providers.google.cloud.transfers.bigquery_to_bigquery.BigQueryHook"
TASK_ID = "test-bq-create-table-operator"
+TEST_GCP_PROJECT_ID = "test-project"
TEST_DATASET = "test-dataset"
TEST_TABLE_ID = "test-table-id"
+SOURCE_PROJECT_DATASET_TABLES = f"{TEST_GCP_PROJECT_ID}.{TEST_DATASET}.{TEST_TABLE_ID}"
+DESTINATION_PROJECT_DATASET_TABLE = f"{TEST_GCP_PROJECT_ID}.{TEST_DATASET + '_new'}.{TEST_TABLE_ID}"
+WRITE_DISPOSITION = "WRITE_EMPTY"
+CREATE_DISPOSITION = "CREATE_IF_NEEDED"
+LABELS = {"k1": "v1"}
+ENCRYPTION_CONFIGURATION = {"key": "kk"}
+
+
+def split_tablename_side_effect(*args, **kwargs):
+ if kwargs["table_input"] == SOURCE_PROJECT_DATASET_TABLES:
+ return (
+ TEST_GCP_PROJECT_ID,
+ TEST_DATASET,
+ TEST_TABLE_ID,
+ )
+ elif kwargs["table_input"] == DESTINATION_PROJECT_DATASET_TABLE:
+ return (
+ TEST_GCP_PROJECT_ID,
+ TEST_DATASET + "_new",
+ TEST_TABLE_ID,
+ )
+
class TestBigQueryToBigQueryOperator:
@mock.patch(BQ_HOOK_PATH)
def test_execute_without_location_should_execute_successfully(self, mock_hook):
- source_project_dataset_tables = f"{TEST_DATASET}.{TEST_TABLE_ID}"
- destination_project_dataset_table = f"{TEST_DATASET + '_new'}.{TEST_TABLE_ID}"
- write_disposition = "WRITE_EMPTY"
- create_disposition = "CREATE_IF_NEEDED"
- labels = {"k1": "v1"}
- encryption_configuration = {"key": "kk"}
-
operator = BigQueryToBigQueryOperator(
task_id=TASK_ID,
- source_project_dataset_tables=source_project_dataset_tables,
- destination_project_dataset_table=destination_project_dataset_table,
- write_disposition=write_disposition,
- create_disposition=create_disposition,
- labels=labels,
- encryption_configuration=encryption_configuration,
+ source_project_dataset_tables=SOURCE_PROJECT_DATASET_TABLES,
+ destination_project_dataset_table=DESTINATION_PROJECT_DATASET_TABLE,
+ write_disposition=WRITE_DISPOSITION,
+ create_disposition=CREATE_DISPOSITION,
+ labels=LABELS,
+ encryption_configuration=ENCRYPTION_CONFIGURATION,
)
+ mock_hook.return_value.split_tablename.side_effect = split_tablename_side_effect
operator.execute(context=mock.MagicMock())
- mock_hook.return_value.run_copy.assert_called_once_with(
- source_project_dataset_tables=source_project_dataset_tables,
- destination_project_dataset_table=destination_project_dataset_table,
- write_disposition=write_disposition,
- create_disposition=create_disposition,
- labels=labels,
- encryption_configuration=encryption_configuration,
+ mock_hook.return_value.insert_job.assert_called_once_with(
+ configuration={
+ "copy": {
+ "createDisposition": CREATE_DISPOSITION,
+ "destinationEncryptionConfiguration": ENCRYPTION_CONFIGURATION,
+ "destinationTable": {
+ "datasetId": TEST_DATASET + "_new",
+ "projectId": TEST_GCP_PROJECT_ID,
+ "tableId": TEST_TABLE_ID,
+ },
+ "sourceTables": [
+ {
+ "datasetId": TEST_DATASET,
+ "projectId": TEST_GCP_PROJECT_ID,
+ "tableId": TEST_TABLE_ID,
+ },
+ ],
+ "writeDisposition": WRITE_DISPOSITION,
+ },
+ "labels": LABELS,
+ },
+ project_id=mock_hook.return_value.project_id,
)
@mock.patch(BQ_HOOK_PATH)
def test_execute_single_regional_location_should_execute_successfully(self, mock_hook):
- source_project_dataset_tables = f"{TEST_DATASET}.{TEST_TABLE_ID}"
- destination_project_dataset_table = f"{TEST_DATASET + '_new'}.{TEST_TABLE_ID}"
- write_disposition = "WRITE_EMPTY"
- create_disposition = "CREATE_IF_NEEDED"
- labels = {"k1": "v1"}
location = "us-central1"
- encryption_configuration = {"key": "kk"}
- mock_hook.return_value.run_copy.return_value = "job-id"
operator = BigQueryToBigQueryOperator(
task_id=TASK_ID,
- source_project_dataset_tables=source_project_dataset_tables,
- destination_project_dataset_table=destination_project_dataset_table,
- write_disposition=write_disposition,
- create_disposition=create_disposition,
- labels=labels,
- encryption_configuration=encryption_configuration,
+ source_project_dataset_tables=SOURCE_PROJECT_DATASET_TABLES,
+ destination_project_dataset_table=DESTINATION_PROJECT_DATASET_TABLE,
+ write_disposition=WRITE_DISPOSITION,
+ create_disposition=CREATE_DISPOSITION,
+ labels=LABELS,
+ encryption_configuration=ENCRYPTION_CONFIGURATION,
location=location,
)
+ mock_hook.return_value.split_tablename.side_effect = split_tablename_side_effect
operator.execute(context=mock.MagicMock())
mock_hook.return_value.get_job.assert_called_once_with(
- job_id="job-id",
+ job_id=mock_hook.return_value.insert_job.return_value.job_id,
location=location,
)
(airflow) 30/37: Fix for infinite recursion due to secrets_masker (#35048)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 673f7f837fa00471008ebae88b192f7cec729d55
Author: Usiel Riedl <us...@automattic.com>
AuthorDate: Wed Nov 22 18:19:04 2023 +0800
Fix for infinite recursion due to secrets_masker (#35048)
* Fix for infinite recursion due to secrets_masker
We can get into trouble for types that cannot be initiated with re2's `type(obj)()` call. The `secrets_masker` thus fails, which triggers a warning log, which also fails because we pass the object to the logger, which is then masked again, and so forth.
We can break the recursion by emitting a log without trying to redact the value again (this ensures no new bug will cause a stack overflow). This issue has occured previously: https://github.com/apache/airflow/issues/19816#issuecomment-983311373
Additionally, we fix this particular bug by ensuring whatever re2 receives is a simple `str`.
I noticed this issue while working with a DAG that calls Airflow's DB cleanup function.
Example DAG:
```
from datetime import datetime
from airflow import DAG
from airflow.models import Variable
from airflow.operators.python import PythonOperator
class MyStringClass(str):
def __init__(self, required_arg):
pass
def fail(task_instance):
# make sure the `SecretsMasker` has a replacer
Variable.set(key="secret", value="secret_value")
Variable.get("secret")
# trigger the infinite recursion
task_instance.log.info("%s", MyStringClass("secret_value"))
with DAG(
dag_id="secrets_masker_recursion",
start_date=datetime(2023, 9, 26),
):
PythonOperator(task_id="fail", python_callable=fail)
```
* Improve error message
---------
Co-authored-by: Tzu-ping Chung <ur...@gmail.com>
---
airflow/utils/log/secrets_masker.py | 9 +++++----
tests/utils/log/test_secrets_masker.py | 18 ++++++++++++++++++
2 files changed, 23 insertions(+), 4 deletions(-)
diff --git a/airflow/utils/log/secrets_masker.py b/airflow/utils/log/secrets_masker.py
index 246377c169..0b1b65f840 100644
--- a/airflow/utils/log/secrets_masker.py
+++ b/airflow/utils/log/secrets_masker.py
@@ -261,7 +261,7 @@ class SecretsMasker(logging.Filter):
# We can't replace specific values, but the key-based redacting
# can still happen, so we can't short-circuit, we need to walk
# the structure.
- return self.replacer.sub("***", item)
+ return self.replacer.sub("***", str(item))
return item
elif isinstance(item, (tuple, set)):
# Turn set in to tuple!
@@ -276,14 +276,15 @@ class SecretsMasker(logging.Filter):
return item
# I think this should never happen, but it does not hurt to leave it just in case
# Well. It happened (see https://github.com/apache/airflow/issues/19816#issuecomment-983311373)
- # but it caused infinite recursion, so we need to cast it to str first.
+ # but it caused infinite recursion, to avoid this we mark the log as already filtered.
except Exception as exc:
log.warning(
- "Unable to redact %r, please report this via <https://github.com/apache/airflow/issues>. "
- "Error was: %s: %s",
+ "Unable to redact value of type %s, please report this via "
+ "<https://github.com/apache/airflow/issues>. Error was: %s: %s",
item,
type(exc).__name__,
exc,
+ extra={self.ALREADY_FILTERED_FLAG: True},
)
return item
diff --git a/tests/utils/log/test_secrets_masker.py b/tests/utils/log/test_secrets_masker.py
index ffaf2977ae..4657a7c1f3 100644
--- a/tests/utils/log/test_secrets_masker.py
+++ b/tests/utils/log/test_secrets_masker.py
@@ -305,6 +305,24 @@ class TestSecretsMasker:
got = redact(val, max_depth=max_depth)
assert got == expected
+ def test_redact_with_str_type(self, logger, caplog):
+ """
+ SecretsMasker's re2 replacer has issues handling a redactable item of type
+ `str` with required constructor args. This test ensures there is a shim in
+ place that avoids any issues.
+ See: https://github.com/apache/airflow/issues/19816#issuecomment-983311373
+ """
+
+ class StrLikeClassWithRequiredConstructorArg(str):
+ def __init__(self, required_arg):
+ pass
+
+ text = StrLikeClassWithRequiredConstructorArg("password")
+ logger.info("redacted: %s", text)
+
+ # we expect the object's __str__() output to be logged (no warnings due to a failed masking)
+ assert caplog.messages == ["redacted: ***"]
+
@pytest.mark.parametrize(
"state, expected",
[
(airflow) 31/37: feat: K8S resource operator - CRD (#35600)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit d568543d879d4a56c675dd1d510e14143f201c5e
Author: raphaelauv <ra...@users.noreply.github.com>
AuthorDate: Wed Nov 22 11:23:46 2023 +0100
feat: K8S resource operator - CRD (#35600)
* feat: K8S resource operator - CRD
* clean
* tests
* remove sensor ( for another PR )
* clean
* test on k8s_resource_iterator
---
.../cncf/kubernetes/operators/resource.py | 62 +++++++++++++++++----
.../cncf/kubernetes/utils/k8s_resource_iterator.py | 46 +++++++++++++++
.../cncf/kubernetes/operators/test_resource.py | 63 +++++++++++++++++++++
.../kubernetes/utils/test_k8s_resource_iterator.py | 65 ++++++++++++++++++++++
4 files changed, 225 insertions(+), 11 deletions(-)
diff --git a/airflow/providers/cncf/kubernetes/operators/resource.py b/airflow/providers/cncf/kubernetes/operators/resource.py
index 598731b639..569b5861a6 100644
--- a/airflow/providers/cncf/kubernetes/operators/resource.py
+++ b/airflow/providers/cncf/kubernetes/operators/resource.py
@@ -27,9 +27,10 @@ from kubernetes.utils import create_from_yaml
from airflow.models import BaseOperator
from airflow.providers.cncf.kubernetes.hooks.kubernetes import KubernetesHook
from airflow.providers.cncf.kubernetes.utils.delete_from import delete_from_yaml
+from airflow.providers.cncf.kubernetes.utils.k8s_resource_iterator import k8s_resource_iterator
if TYPE_CHECKING:
- from kubernetes.client import ApiClient
+ from kubernetes.client import ApiClient, CustomObjectsApi
__all__ = ["KubernetesCreateResourceOperator", "KubernetesDeleteResourceOperator"]
@@ -56,17 +57,23 @@ class KubernetesResourceBaseOperator(BaseOperator):
yaml_conf: str,
namespace: str | None = None,
kubernetes_conn_id: str | None = KubernetesHook.default_conn_name,
+ custom_resource_definition: bool = False,
**kwargs,
) -> None:
super().__init__(**kwargs)
self._namespace = namespace
self.kubernetes_conn_id = kubernetes_conn_id
self.yaml_conf = yaml_conf
+ self.custom_resource_definition = custom_resource_definition
@cached_property
def client(self) -> ApiClient:
return self.hook.api_client
+ @cached_property
+ def custom_object_client(self) -> CustomObjectsApi:
+ return self.hook.custom_object_client
+
@cached_property
def hook(self) -> KubernetesHook:
hook = KubernetesHook(conn_id=self.kubernetes_conn_id)
@@ -78,24 +85,57 @@ class KubernetesResourceBaseOperator(BaseOperator):
else:
return self.hook.get_namespace() or "default"
+ def get_crd_fields(self, body: dict) -> tuple[str, str, str, str]:
+ api_version = body["apiVersion"]
+ group = api_version[0 : api_version.find("/")]
+ version = api_version[api_version.find("/") + 1 :]
+
+ namespace = None
+ if body.get("metadata"):
+ metadata: dict = body.get("metadata", None)
+ namespace = metadata.get("namespace", None)
+ if namespace is None:
+ namespace = self.get_namespace()
+
+ plural = body["kind"].lower() + "s"
+
+ return group, version, namespace, plural
+
class KubernetesCreateResourceOperator(KubernetesResourceBaseOperator):
"""Create a resource in a kubernetes."""
+ def create_custom_from_yaml_object(self, body: dict):
+ group, version, namespace, plural = self.get_crd_fields(body)
+ self.custom_object_client.create_namespaced_custom_object(group, version, namespace, plural, body)
+
def execute(self, context) -> None:
- create_from_yaml(
- k8s_client=self.client,
- yaml_objects=yaml.safe_load_all(self.yaml_conf),
- namespace=self.get_namespace(),
- )
+ resources = yaml.safe_load_all(self.yaml_conf)
+ if not self.custom_resource_definition:
+ create_from_yaml(
+ k8s_client=self.client,
+ yaml_objects=resources,
+ namespace=self.get_namespace(),
+ )
+ else:
+ k8s_resource_iterator(self.create_custom_from_yaml_object, resources)
class KubernetesDeleteResourceOperator(KubernetesResourceBaseOperator):
"""Delete a resource in a kubernetes."""
+ def delete_custom_from_yaml_object(self, body: dict):
+ name = body["metadata"]["name"]
+ group, version, namespace, plural = self.get_crd_fields(body)
+ self.custom_object_client.delete_namespaced_custom_object(group, version, namespace, plural, name)
+
def execute(self, context) -> None:
- delete_from_yaml(
- k8s_client=self.client,
- yaml_objects=yaml.safe_load_all(self.yaml_conf),
- namespace=self.get_namespace(),
- )
+ resources = yaml.safe_load_all(self.yaml_conf)
+ if not self.custom_resource_definition:
+ delete_from_yaml(
+ k8s_client=self.client,
+ yaml_objects=resources,
+ namespace=self.get_namespace(),
+ )
+ else:
+ k8s_resource_iterator(self.delete_custom_from_yaml_object, resources)
diff --git a/airflow/providers/cncf/kubernetes/utils/k8s_resource_iterator.py b/airflow/providers/cncf/kubernetes/utils/k8s_resource_iterator.py
new file mode 100644
index 0000000000..bfa1d05272
--- /dev/null
+++ b/airflow/providers/cncf/kubernetes/utils/k8s_resource_iterator.py
@@ -0,0 +1,46 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import Callable, Iterator
+
+from kubernetes.utils import FailToCreateError
+
+from airflow.providers.cncf.kubernetes.utils.delete_from import FailToDeleteError
+
+
+def k8s_resource_iterator(callback: Callable[[dict], None], resources: Iterator) -> None:
+ failures: list = []
+ for data in resources:
+ if data is not None:
+ if "List" in data["kind"]:
+ kind = data["kind"].replace("List", "")
+ for yml_doc in data["items"]:
+ if kind != "":
+ yml_doc["apiVersion"] = data["apiVersion"]
+ yml_doc["kind"] = kind
+ try:
+ callback(yml_doc)
+ except (FailToCreateError, FailToDeleteError) as failure:
+ failures.extend(failure.api_exceptions)
+ else:
+ try:
+ callback(data)
+ except (FailToCreateError, FailToDeleteError) as failure:
+ failures.extend(failure.api_exceptions)
+ if failures:
+ raise FailToCreateError(failures)
diff --git a/tests/providers/cncf/kubernetes/operators/test_resource.py b/tests/providers/cncf/kubernetes/operators/test_resource.py
index a565e84fc1..9673c6e082 100644
--- a/tests/providers/cncf/kubernetes/operators/test_resource.py
+++ b/tests/providers/cncf/kubernetes/operators/test_resource.py
@@ -56,6 +56,16 @@ items:
name: test_pvc_2
"""
+TEST_VALID_CRD_YAML = """
+apiVersion: ray.io/v1
+kind: RayJob
+metadata:
+ name: rayjob-sample
+spec:
+ entrypoint: python /home/ray/program/job.py
+ shutdownAfterJobFinishes: true
+"""
+
HOOK_CLASS = "airflow.providers.cncf.kubernetes.hooks.kubernetes.KubernetesHook"
@@ -89,6 +99,19 @@ class TestKubernetesXResourceOperator:
body=yaml.safe_load(TEST_VALID_RESOURCE_YAML), namespace="default"
)
+ @patch("kubernetes.client.api.CoreV1Api.create_namespaced_persistent_volume_claim")
+ def test_create_application_from_yaml_list(self, mock_create_namespaced_persistent_volume_claim, context):
+ op = KubernetesCreateResourceOperator(
+ yaml_conf=TEST_VALID_LIST_RESOURCE_YAML,
+ dag=self.dag,
+ kubernetes_conn_id="kubernetes_default",
+ task_id="test_task_id",
+ )
+
+ op.execute(context)
+
+ assert mock_create_namespaced_persistent_volume_claim.call_count == 2
+
@patch("kubernetes.client.api.CoreV1Api.delete_namespaced_persistent_volume_claim")
def test_single_delete_application_from_yaml(
self, mock_delete_namespaced_persistent_volume_claim, context
@@ -118,3 +141,43 @@ class TestKubernetesXResourceOperator:
op.execute(context)
mock_delete_namespaced_persistent_volume_claim.assert_called()
+
+ @patch("kubernetes.client.api.CustomObjectsApi.create_namespaced_custom_object")
+ def test_create_custom_application_from_yaml(self, mock_create_namespaced_custom_object, context):
+ op = KubernetesCreateResourceOperator(
+ yaml_conf=TEST_VALID_CRD_YAML,
+ dag=self.dag,
+ kubernetes_conn_id="kubernetes_default",
+ task_id="test_task_id",
+ custom_resource_definition=True,
+ )
+
+ op.execute(context)
+
+ mock_create_namespaced_custom_object.assert_called_once_with(
+ "ray.io",
+ "v1",
+ "default",
+ "rayjobs",
+ yaml.safe_load(TEST_VALID_CRD_YAML),
+ )
+
+ @patch("kubernetes.client.api.CustomObjectsApi.delete_namespaced_custom_object")
+ def test_delete_custom_application_from_yaml(self, mock_delete_namespaced_custom_object, context):
+ op = KubernetesDeleteResourceOperator(
+ yaml_conf=TEST_VALID_CRD_YAML,
+ dag=self.dag,
+ kubernetes_conn_id="kubernetes_default",
+ task_id="test_task_id",
+ custom_resource_definition=True,
+ )
+
+ op.execute(context)
+
+ mock_delete_namespaced_custom_object.assert_called_once_with(
+ "ray.io",
+ "v1",
+ "default",
+ "rayjobs",
+ "rayjob-sample",
+ )
diff --git a/tests/providers/cncf/kubernetes/utils/test_k8s_resource_iterator.py b/tests/providers/cncf/kubernetes/utils/test_k8s_resource_iterator.py
new file mode 100644
index 0000000000..8a2e4f0539
--- /dev/null
+++ b/tests/providers/cncf/kubernetes/utils/test_k8s_resource_iterator.py
@@ -0,0 +1,65 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from collections import namedtuple
+
+import pytest
+import yaml
+from kubernetes.utils import FailToCreateError
+
+from airflow.providers.cncf.kubernetes.utils.k8s_resource_iterator import k8s_resource_iterator
+
+TEST_VALID_LIST_RESOURCE_YAML = """
+apiVersion: v1
+kind: List
+items:
+- apiVersion: v1
+ kind: PersistentVolumeClaim
+ metadata:
+ name: test_pvc_1
+- apiVersion: v1
+ kind: PersistentVolumeClaim
+ metadata:
+ name: test_pvc_2
+"""
+
+
+def test_k8s_resource_iterator():
+ exception_k8s = namedtuple("Exception_k8s", "reason body")
+
+ def test_callback_failing(yml_doc: dict) -> None:
+ raise FailToCreateError(exception_k8s("the_reason", "the_body "))
+
+ with pytest.raises(FailToCreateError) as exc_info:
+ k8s_resource_iterator(
+ test_callback_failing, resources=yaml.safe_load_all(TEST_VALID_LIST_RESOURCE_YAML)
+ )
+
+ assert (
+ str(exc_info.value)
+ == "Error from server (the_reason): the_body Error from server (the_reason): the_body "
+ )
+
+ def callback_success(yml_doc: dict) -> None:
+ return
+
+ try:
+ k8s_resource_iterator(callback_success, resources=yaml.safe_load_all(TEST_VALID_LIST_RESOURCE_YAML))
+
+ except FailToCreateError:
+ assert False
(airflow) 07/37: Rename --aditional-extras flag to --aditional-airflow-extras (#35760)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 38afe9ffb1e723528ebaa3f005f880aeadd8e926
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Mon Nov 20 22:38:35 2023 +0100
Rename --aditional-extras flag to --aditional-airflow-extras (#35760)
The previous flag was wrongly named so it was converted to
ADDITIONAL_EXTRAS ARG rather than ADDITIONA_AIRFLOW_EXTRAS.
---
BREEZE.rst | 2 +-
IMAGES.rst | 10 +++++-----
.../src/airflow_breeze/commands/ci_image_commands_config.py | 2 +-
.../commands/production_image_commands_config.py | 2 +-
dev/breeze/src/airflow_breeze/params/build_ci_params.py | 4 ++--
dev/breeze/src/airflow_breeze/params/build_prod_params.py | 2 +-
dev/breeze/src/airflow_breeze/params/common_build_params.py | 2 +-
dev/breeze/src/airflow_breeze/utils/common_options.py | 2 +-
images/breeze/output_ci-image_build.svg | 2 +-
images/breeze/output_ci-image_build.txt | 2 +-
images/breeze/output_prod-image_build.svg | 2 +-
images/breeze/output_prod-image_build.txt | 2 +-
12 files changed, 17 insertions(+), 17 deletions(-)
diff --git a/BREEZE.rst b/BREEZE.rst
index 04eecace44..d5bd3c73a8 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -1626,7 +1626,7 @@ but here typical examples are presented:
.. code-block:: bash
- breeze prod-image build --additional-extras "jira"
+ breeze prod-image build --additional-airflow-extras "jira"
This installs additional ``jira`` extra while installing airflow in the image.
diff --git a/IMAGES.rst b/IMAGES.rst
index 240952a0e3..7c69d58e70 100644
--- a/IMAGES.rst
+++ b/IMAGES.rst
@@ -97,7 +97,7 @@ By adding ``--python <PYTHON_MAJOR_MINOR_VERSION>`` parameter you can build the
image version for the chosen Python version.
The images are built with default extras - different extras for CI and production image and you
-can change the extras via the ``--extras`` parameters and add new ones with ``--additional-extras``.
+can change the extras via the ``--extras`` parameters and add new ones with ``--additional-airflow-extras``.
For example if you want to build Python 3.8 version of production image with
"all" extras installed you should run this command:
@@ -110,7 +110,7 @@ If you just want to add new extras you can add them like that:
.. code-block:: bash
- breeze prod-image build --python 3.8 --additional-extras "all"
+ breeze prod-image build --python 3.8 --additional-airflow-extras "all"
The command that builds the CI image is optimized to minimize the time needed to rebuild the image when
the source code of Airflow evolves. This means that if you already have the image locally downloaded and
@@ -128,7 +128,7 @@ parameter to Breeze:
.. code-block:: bash
- breeze prod-image build --python 3.8 --additional-extras=trino --install-airflow-version=2.0.0
+ breeze prod-image build --python 3.8 --additional-airflow-extras=trino --install-airflow-version=2.0.0
This will build the image using command similar to:
@@ -170,7 +170,7 @@ You can also skip installing airflow and install it from locally provided files
.. code-block:: bash
- breeze prod-image build --python 3.8 --additional-extras=trino --install-packages-from-context
+ breeze prod-image build --python 3.8 --additional-airflow-extras=trino --install-packages-from-context
In this case you airflow and all packages (.whl files) should be placed in ``docker-context-files`` folder.
@@ -331,7 +331,7 @@ the same image can be built using ``breeze`` (it supports auto-completion of the
.. code-block:: bash
- breeze ci-image build --python 3.8 --additional-extras=jdbc --additional-python-deps="pandas" \
+ breeze ci-image build --python 3.8 --additional-airflow-extras=jdbc --additional-python-deps="pandas" \
--additional-dev-apt-deps="gcc g++"
You can customize more aspects of the image - such as additional commands executed before apt dependencies
diff --git a/dev/breeze/src/airflow_breeze/commands/ci_image_commands_config.py b/dev/breeze/src/airflow_breeze/commands/ci_image_commands_config.py
index f256406ea4..87e6ab6e91 100644
--- a/dev/breeze/src/airflow_breeze/commands/ci_image_commands_config.py
+++ b/dev/breeze/src/airflow_breeze/commands/ci_image_commands_config.py
@@ -60,7 +60,7 @@ CI_IMAGE_TOOLS_PARAMETERS: dict[str, list[dict[str, str | list[str]]]] = {
"--airflow-constraints-reference",
"--python-image",
"--additional-python-deps",
- "--additional-extras",
+ "--additional-airflow-extras",
"--additional-pip-install-flags",
"--additional-dev-apt-deps",
"--additional-dev-apt-env",
diff --git a/dev/breeze/src/airflow_breeze/commands/production_image_commands_config.py b/dev/breeze/src/airflow_breeze/commands/production_image_commands_config.py
index 0f55650b88..713bd0bfa4 100644
--- a/dev/breeze/src/airflow_breeze/commands/production_image_commands_config.py
+++ b/dev/breeze/src/airflow_breeze/commands/production_image_commands_config.py
@@ -57,7 +57,7 @@ PRODUCTION_IMAGE_TOOLS_PARAMETERS: dict[str, list[dict[str, str | list[str]]]] =
"--airflow-constraints-mode",
"--airflow-constraints-reference",
"--python-image",
- "--additional-extras",
+ "--additional-airflow-extras",
"--additional-pip-install-flags",
"--additional-python-deps",
"--additional-runtime-apt-deps",
diff --git a/dev/breeze/src/airflow_breeze/params/build_ci_params.py b/dev/breeze/src/airflow_breeze/params/build_ci_params.py
index c216d4c3c6..a713c280b5 100644
--- a/dev/breeze/src/airflow_breeze/params/build_ci_params.py
+++ b/dev/breeze/src/airflow_breeze/params/build_ci_params.py
@@ -94,7 +94,7 @@ class BuildCiParams(CommonBuildParams):
@property
def optional_image_args(self) -> list[str]:
return [
- "additional_extras",
+ "additional_airflow_extras",
"additional_dev_apt_command",
"additional_dev_apt_deps",
"additional_dev_apt_env",
@@ -108,7 +108,7 @@ class BuildCiParams(CommonBuildParams):
"additional_dev_apt_command",
"additional_dev_apt_deps",
"additional_dev_apt_env",
- "additional_extras",
+ "additional_airflow_extras",
"additional_pip_install_flags",
"additional_python_deps",
"version_suffix_for_pypi",
diff --git a/dev/breeze/src/airflow_breeze/params/build_prod_params.py b/dev/breeze/src/airflow_breeze/params/build_prod_params.py
index 2ba5d3787c..585c7dc0c9 100644
--- a/dev/breeze/src/airflow_breeze/params/build_prod_params.py
+++ b/dev/breeze/src/airflow_breeze/params/build_prod_params.py
@@ -226,7 +226,7 @@ class BuildProdParams(CommonBuildParams):
@property
def optional_image_args(self) -> list[str]:
return [
- "additional_extras",
+ "additional_airflow_extras",
"additional_dev_apt_command",
"additional_dev_apt_deps",
"additional_dev_apt_env",
diff --git a/dev/breeze/src/airflow_breeze/params/common_build_params.py b/dev/breeze/src/airflow_breeze/params/common_build_params.py
index 798e335e28..782d01bca4 100644
--- a/dev/breeze/src/airflow_breeze/params/common_build_params.py
+++ b/dev/breeze/src/airflow_breeze/params/common_build_params.py
@@ -37,7 +37,7 @@ class CommonBuildParams:
Common build parameters. Those parameters are common parameters for CI And PROD build.
"""
- additional_extras: str = ""
+ additional_airflow_extras: str = ""
additional_dev_apt_command: str = ""
additional_dev_apt_deps: str = ""
additional_dev_apt_env: str = ""
diff --git a/dev/breeze/src/airflow_breeze/utils/common_options.py b/dev/breeze/src/airflow_breeze/utils/common_options.py
index 337f5cd2c5..2a001f2757 100644
--- a/dev/breeze/src/airflow_breeze/utils/common_options.py
+++ b/dev/breeze/src/airflow_breeze/utils/common_options.py
@@ -294,7 +294,7 @@ option_upgrade_on_failure = click.option(
envvar="UPGRADE_ON_FAILURE",
)
option_additional_extras = click.option(
- "--additional-extras",
+ "--additional-airflow-extras",
help="Additional extra package while installing Airflow in the image.",
envvar="ADDITIONAL_AIRFLOW_EXTRAS",
)
diff --git a/images/breeze/output_ci-image_build.svg b/images/breeze/output_ci-image_build.svg
index 98f7dbb901..2a7147c2fd 100644
--- a/images/breeze/output_ci-image_build.svg
+++ b/images/breeze/output_ci-image_build.svg
@@ -364,7 +364,7 @@
</text><text class="breeze-ci-image-build-r5" x="0" y="1142.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-46)">│</text><text class="breeze-ci-image-build-r1" x="463.6" y="1142.4" textLength="976" clip-path="url(#breeze-ci-image-build-line-46)">something like: python:VERSION-slim-bookworm.                             & [...]
</text><text class="breeze-ci-image-build-r5" x="0" y="1166.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-47)">│</text><text class="breeze-ci-image-build-r7" x="463.6" y="1166.8" textLength="976" clip-path="url(#breeze-ci-image-build-line-47)">(TEXT)                                     &# [...]
</text><text class="breeze-ci-image-build-r5" x="0" y="1191.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-48)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1191.2" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-48)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1191.2" textLength="134.2" clip-path="url(#breeze-ci-image-build-line-48)">-additional</text><text class="breeze-ci-image-build-r4" x="170.8" y="1191.2" textLength="146.4" [...]
-</text><text class="breeze-ci-image-build-r5" x="0" y="1215.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-49)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1215.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-49)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1215.6" textLength="134.2" clip-path="url(#breeze-ci-image-build-line-49)">-additional</text><text class="breeze-ci-image-build-r4" x="170.8" y="1215.6" textLength="85.4" [...]
+</text><text class="breeze-ci-image-build-r5" x="0" y="1215.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-49)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1215.6" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-49)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1215.6" textLength="134.2" clip-path="url(#breeze-ci-image-build-line-49)">-additional</text><text class="breeze-ci-image-build-r4" x="170.8" y="1215.6" textLength="183" c [...]
</text><text class="breeze-ci-image-build-r5" x="0" y="1240" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-50)">│</text><text class="breeze-ci-image-build-r4" x="24.4" y="1240" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-50)">-</text><text class="breeze-ci-image-build-r4" x="36.6" y="1240" textLength="134.2" clip-path="url(#breeze-ci-image-build-line-50)">-additional</text><text class="breeze-ci-image-build-r4" x="170.8" y="1240" textLength="219.6" clip-pa [...]
</text><text class="breeze-ci-image-build-r5" x="0" y="1264.4" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-51)">│</text><text class="breeze-ci-image-build-r1" x="463.6" y="1264.4" textLength="976" clip-path="url(#breeze-ci-image-build-line-51)">itself).                                      [...]
</text><text class="breeze-ci-image-build-r5" x="0" y="1288.8" textLength="12.2" clip-path="url(#breeze-ci-image-build-line-52)">│</text><text class="breeze-ci-image-build-r7" x="463.6" y="1288.8" textLength="976" clip-path="url(#breeze-ci-image-build-line-52)">(TEXT)                                     &# [...]
diff --git a/images/breeze/output_ci-image_build.txt b/images/breeze/output_ci-image_build.txt
index 71765b3d76..f659f125e8 100644
--- a/images/breeze/output_ci-image_build.txt
+++ b/images/breeze/output_ci-image_build.txt
@@ -1 +1 @@
-5a1b532b087c0feb24ffdc9da3ec4e81
+c405885e9a457ff1ca8caa8a3ebab6e5
diff --git a/images/breeze/output_prod-image_build.svg b/images/breeze/output_prod-image_build.svg
index 31e599f23d..079f3f39e6 100644
--- a/images/breeze/output_prod-image_build.svg
+++ b/images/breeze/output_prod-image_build.svg
@@ -394,7 +394,7 @@
</text><text class="breeze-prod-image-build-r5" x="0" y="1069.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-43)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1069.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-43)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1069.2" textLength="85.4" clip-path="url(#breeze-prod-image-build-line-43)">-python</text><text class="breeze-prod-image-build-r4" x="122" y="1069.2" textLength= [...]
</text><text class="breeze-prod-image-build-r5" x="0" y="1093.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-44)">│</text><text class="breeze-prod-image-build-r1" x="463.6" y="1093.6" textLength="976" clip-path="url(#breeze-prod-image-build-line-44)">something like: python:VERSION-slim-bookworm.                             [...]
</text><text class="breeze-prod-image-build-r5" x="0" y="1118" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-45)">│</text><text class="breeze-prod-image-build-r7" x="463.6" y="1118" textLength="976" clip-path="url(#breeze-prod-image-build-line-45)">(TEXT)                                     [...]
-</text><text class="breeze-prod-image-build-r5" x="0" y="1142.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-46)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1142.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-46)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1142.4" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-46)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1142.4" text [...]
+</text><text class="breeze-prod-image-build-r5" x="0" y="1142.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-46)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1142.4" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-46)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1142.4" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-46)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1142.4" text [...]
</text><text class="breeze-prod-image-build-r5" x="0" y="1166.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-47)">│</text><text class="breeze-prod-image-build-r4" x="24.4" y="1166.8" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-47)">-</text><text class="breeze-prod-image-build-r4" x="36.6" y="1166.8" textLength="134.2" clip-path="url(#breeze-prod-image-build-line-47)">-additional</text><text class="breeze-prod-image-build-r4" x="170.8" y="1166.8" text [...]
</text><text class="breeze-prod-image-build-r5" x="0" y="1191.2" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-48)">│</text><text class="breeze-prod-image-build-r1" x="463.6" y="1191.2" textLength="976" clip-path="url(#breeze-prod-image-build-line-48)">itself).                                    [...]
</text><text class="breeze-prod-image-build-r5" x="0" y="1215.6" textLength="12.2" clip-path="url(#breeze-prod-image-build-line-49)">│</text><text class="breeze-prod-image-build-r7" x="463.6" y="1215.6" textLength="976" clip-path="url(#breeze-prod-image-build-line-49)">(TEXT)                                     [...]
diff --git a/images/breeze/output_prod-image_build.txt b/images/breeze/output_prod-image_build.txt
index e511dfeeb4..aac6b4aab7 100644
--- a/images/breeze/output_prod-image_build.txt
+++ b/images/breeze/output_prod-image_build.txt
@@ -1 +1 @@
-bdbf7fe98a65e3384825cc99c9ea03e9
+1eed0951812bd758082ffe34daed932b
(airflow) 27/37: Add missing docker test_exceptions.py (#35674)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 7937cba0dc7e2d6db80209cfaf80aafe1d3b1a4c
Author: Riyas P K <16...@users.noreply.github.com>
AuthorDate: Wed Nov 22 06:08:45 2023 +0530
Add missing docker test_exceptions.py (#35674)
* test case for docker exceptions
* removed docker test exception
* re-wrote test case based on feedback
---
tests/always/test_project_structure.py | 1 -
tests/providers/docker/test_exceptions.py | 80 +++++++++++++++++++++++++++++++
2 files changed, 80 insertions(+), 1 deletion(-)
diff --git a/tests/always/test_project_structure.py b/tests/always/test_project_structure.py
index a58f33d755..967c3ecbb5 100644
--- a/tests/always/test_project_structure.py
+++ b/tests/always/test_project_structure.py
@@ -108,7 +108,6 @@ class TestProjectStructure:
"tests/providers/cncf/kubernetes/utils/test_xcom_sidecar.py",
"tests/providers/daskexecutor/executors/test_dask_executor.py",
"tests/providers/databricks/hooks/test_databricks_base.py",
- "tests/providers/docker/test_exceptions.py",
"tests/providers/google/cloud/fs/test_gcs.py",
"tests/providers/google/cloud/links/test_automl.py",
"tests/providers/google/cloud/links/test_base.py",
diff --git a/tests/providers/docker/test_exceptions.py b/tests/providers/docker/test_exceptions.py
new file mode 100644
index 0000000000..1f3653c39d
--- /dev/null
+++ b/tests/providers/docker/test_exceptions.py
@@ -0,0 +1,80 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Test for Exceptions used by Docker provider."""
+
+from __future__ import annotations
+
+from unittest import mock
+
+import pytest
+from docker import APIClient
+
+from airflow.providers.docker.exceptions import (
+ DockerContainerFailedException,
+ DockerContainerFailedSkipException,
+)
+from airflow.providers.docker.operators.docker import DockerOperator
+
+FAILED_MESSAGE = {"StatusCode": 1}
+FAILED_LOGS = ["unicode container log 😁 ", b"byte string container log"]
+EXPECTED_MESSAGE = f"Docker container failed: {FAILED_MESSAGE}"
+FAILED_SKIP_MESSAGE = {"StatusCode": 2}
+SKIP_ON_EXIT_CODE = 2
+EXPECTED_SKIP_MESSAGE = f"Docker container returned exit code {[SKIP_ON_EXIT_CODE]}. Skipping."
+
+
+@pytest.mark.parametrize(
+ "failed_msg, log_line, expected_message, skip_on_exit_code",
+ [
+ (FAILED_MESSAGE, FAILED_LOGS, EXPECTED_MESSAGE, None),
+ (FAILED_SKIP_MESSAGE, FAILED_LOGS, EXPECTED_SKIP_MESSAGE, SKIP_ON_EXIT_CODE),
+ ],
+)
+class TestDockerContainerExceptions:
+ @pytest.fixture(autouse=True, scope="function")
+ def setup_patchers(self, docker_api_client_patcher):
+ self.client_mock = mock.MagicMock(spec=APIClient)
+ self.client_mock.wait.return_value = {"StatusCode": 0}
+ self.log_messages = ["container log 😁 ", b"byte string container log"]
+ self.client_mock.attach.return_value = self.log_messages
+
+ self.client_mock.logs.side_effect = (
+ lambda **kwargs: iter(self.log_messages[-kwargs["tail"] :])
+ if "tail" in kwargs
+ else iter(self.log_messages)
+ )
+
+ docker_api_client_patcher.return_value = self.client_mock
+
+ def test_docker_failed_exception(self, failed_msg, log_line, expected_message, skip_on_exit_code):
+ self.client_mock.attach.return_value = log_line
+ self.client_mock.wait.return_value = failed_msg
+
+ operator = DockerOperator(
+ image="ubuntu", owner="unittest", task_id="unittest", skip_on_exit_code=skip_on_exit_code
+ )
+
+ if skip_on_exit_code:
+ with pytest.raises(DockerContainerFailedSkipException) as raised_exception:
+ operator.execute(None)
+ else:
+ with pytest.raises(DockerContainerFailedException) as raised_exception:
+ operator.execute(None)
+
+ assert str(raised_exception.value) == expected_message
+ assert raised_exception.value.logs == [log_line[0].strip(), log_line[1].decode("utf-8")]
(airflow) 09/37: Update README.md to reflect changes we agreed to the versioning (#35764)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 7642b29f5c0eaef898839579304ba0c8223801f1
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Mon Nov 20 23:59:16 2023 +0100
Update README.md to reflect changes we agreed to the versioning (#35764)
There were two changes that we agreed and voted in the community
that have not been properly reflected yet in README.md (they are
reflected in detailed documentation though):
* for providers, buping minimum version of Airflow is not a reason
to bump MAJOR version - only MINOR
* for API clients - they follow their own versioning SemVer scheme,
independent from Airflow.
---
README.md | 15 ++++-----------
1 file changed, 4 insertions(+), 11 deletions(-)
diff --git a/README.md b/README.md
index 294e52d81a..f68615dcd1 100644
--- a/README.md
+++ b/README.md
@@ -269,22 +269,15 @@ packages:
they are present in providers as `install_requires` limitations. We aim to keep backwards
compatibility of providers with all previously released Airflow 2 versions but
there will sometimes be breaking changes that might make some, or all
- providers, have minimum Airflow version specified. Change of that minimum supported Airflow version
- is a breaking change for provider because installing the new provider might automatically
- upgrade Airflow (which might be an undesired side effect of upgrading provider).
+ providers, have minimum Airflow version specified.
* **Airflow Helm Chart**: SemVer rules apply to changes in the chart only. SemVer MAJOR and MINOR
versions for the chart are independent of the Airflow version. We aim to keep backwards
compatibility of the Helm Chart with all released Airflow 2 versions, but some new features might
only work starting from specific Airflow releases. We might however limit the Helm
Chart to depend on minimal Airflow version.
-* **Airflow API clients**: SemVer MAJOR and MINOR versions follow MAJOR and MINOR versions of Airflow.
- The first MAJOR or MINOR X.Y.0 release of Airflow should always be followed by X.Y.0 release of
- all clients. An airflow PATCH X.Y.Z release can be followed by a PATCH release of API clients, only
- if this PATCH is relevant to the clients.
- The clients then can release their own PATCH releases with bugfixes, independently of Airflow PATCH releases.
- As a consequence, each API client will have its own PATCH version that may or may not be in sync with the Airflow
- PATCH version. For a specific MAJOR/MINOR Airflow version, users should favor the latest PATCH version of clients
- independently of their Airflow PATCH version.
+* **Airflow API clients**: Their versioning is independent from Airflow versions. They follow their own
+ SemVer rules for breaking changes and new features - which for example allows to change the way we generate
+ the clients.
## Version Life Cycle
(airflow) 20/37: Check attr on parent not self re TaskContextLogger set_context (#35780)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit feaeb8c5fb246f945be5aa96ba49441c6700ae39
Author: Daniel Standish <15...@users.noreply.github.com>
AuthorDate: Tue Nov 21 09:55:16 2023 -0800
Check attr on parent not self re TaskContextLogger set_context (#35780)
To know whether we should supply `identifier` param, need to check parent class.
---
airflow/providers/amazon/aws/log/s3_task_handler.py | 4 +++-
airflow/providers/elasticsearch/log/es_task_handler.py | 4 +++-
airflow/providers/google/cloud/log/gcs_task_handler.py | 4 +++-
airflow/providers/microsoft/azure/log/wasb_task_handler.py | 4 +++-
4 files changed, 12 insertions(+), 4 deletions(-)
diff --git a/airflow/providers/amazon/aws/log/s3_task_handler.py b/airflow/providers/amazon/aws/log/s3_task_handler.py
index 761c4ce463..f3664f7c41 100644
--- a/airflow/providers/amazon/aws/log/s3_task_handler.py
+++ b/airflow/providers/amazon/aws/log/s3_task_handler.py
@@ -78,7 +78,9 @@ class S3TaskHandler(FileTaskHandler, LoggingMixin):
)
def set_context(self, ti: TaskInstance, *, identifier: str | None = None) -> None:
- if getattr(self, "supports_task_context_logging", False):
+ # todo: remove-at-min-airflow-version-2.8
+ # after Airflow 2.8 can always pass `identifier`
+ if getattr(super(), "supports_task_context_logging", False):
super().set_context(ti, identifier=identifier)
else:
super().set_context(ti)
diff --git a/airflow/providers/elasticsearch/log/es_task_handler.py b/airflow/providers/elasticsearch/log/es_task_handler.py
index 1e8c75b7e3..c9d3a180e1 100644
--- a/airflow/providers/elasticsearch/log/es_task_handler.py
+++ b/airflow/providers/elasticsearch/log/es_task_handler.py
@@ -443,7 +443,9 @@ class ElasticsearchTaskHandler(FileTaskHandler, ExternalLoggingMixin, LoggingMix
self.handler.setLevel(self.level)
self.handler.setFormatter(self.formatter)
else:
- if getattr(self, "supports_task_context_logging", False):
+ # todo: remove-at-min-airflow-version-2.8
+ # after Airflow 2.8 can always pass `identifier`
+ if getattr(super(), "supports_task_context_logging", False):
super().set_context(ti, identifier=identifier)
else:
super().set_context(ti)
diff --git a/airflow/providers/google/cloud/log/gcs_task_handler.py b/airflow/providers/google/cloud/log/gcs_task_handler.py
index 39d0f072a8..9921bb8753 100644
--- a/airflow/providers/google/cloud/log/gcs_task_handler.py
+++ b/airflow/providers/google/cloud/log/gcs_task_handler.py
@@ -142,7 +142,9 @@ class GCSTaskHandler(FileTaskHandler, LoggingMixin):
)
def set_context(self, ti: TaskInstance, *, identifier: str | None = None) -> None:
- if getattr(self, "supports_task_context_logging", False):
+ # todo: remove-at-min-airflow-version-2.8
+ # after Airflow 2.8 can always pass `identifier`
+ if getattr(super(), "supports_task_context_logging", False):
super().set_context(ti, identifier=identifier)
else:
super().set_context(ti)
diff --git a/airflow/providers/microsoft/azure/log/wasb_task_handler.py b/airflow/providers/microsoft/azure/log/wasb_task_handler.py
index f3a00e8432..c57de1acb1 100644
--- a/airflow/providers/microsoft/azure/log/wasb_task_handler.py
+++ b/airflow/providers/microsoft/azure/log/wasb_task_handler.py
@@ -96,7 +96,9 @@ class WasbTaskHandler(FileTaskHandler, LoggingMixin):
return None
def set_context(self, ti: TaskInstance, *, identifier: str | None = None) -> None:
- if getattr(self, "supports_task_context_logging", False):
+ # todo: remove-at-min-airflow-version-2.8
+ # after Airflow 2.8 can always pass `identifier`
+ if getattr(super(), "supports_task_context_logging", False):
super().set_context(ti, identifier=identifier)
else:
super().set_context(ti)
(airflow) 08/37: More detail on mandatory task arguments (#35740)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 9b832f6acd79ec1329ad235e65a7abd2be93a0eb
Author: Cai Parry-Jones <97...@users.noreply.github.com>
AuthorDate: Mon Nov 20 22:02:59 2023 +0000
More detail on mandatory task arguments (#35740)
* More detail on mandatory task arguments
Current documentation notes that the arguments 'task_id' and 'owner' are both mandatory. This might confuse new users to believe that both arguments require user input to avoid an error. But 'owner' has a default default_value, so this argument should be less of a concern for user task and dag creation. This commit aims to communicate that.
* fix typo
* Update fundamentals.rst
---------
Co-authored-by: Jarek Potiuk <ja...@potiuk.com>
---
docs/apache-airflow/tutorial/fundamentals.rst | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/docs/apache-airflow/tutorial/fundamentals.rst b/docs/apache-airflow/tutorial/fundamentals.rst
index 2c710530b2..e68205aca9 100644
--- a/docs/apache-airflow/tutorial/fundamentals.rst
+++ b/docs/apache-airflow/tutorial/fundamentals.rst
@@ -146,8 +146,11 @@ The precedence rules for a task are as follows:
2. Values that exist in the ``default_args`` dictionary
3. The operator's default value, if one exists
-A task must include or inherit the arguments ``task_id`` and ``owner``,
-otherwise Airflow will raise an exception.
+.. note::
+ A task must include or inherit the arguments ``task_id`` and ``owner``,
+ otherwise Airflow will raise an exception. A fresh install of Airflow will
+ have a default value of 'airflow' set for ``owner``, so you only really need
+ to worry about ensuring ``task_id`` has a value.
Templating with Jinja
---------------------
(airflow) 02/37: Update minor release command (#35751)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 70e2f419e69987715b195a5fae7b3f73d10024f9
Author: Ephraim Anierobi <sp...@gmail.com>
AuthorDate: Mon Nov 20 15:10:55 2023 +0100
Update minor release command (#35751)
When creating the constraints file for the new release, pull the constraints-main
first
---
dev/breeze/src/airflow_breeze/commands/minor_release_command.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/dev/breeze/src/airflow_breeze/commands/minor_release_command.py b/dev/breeze/src/airflow_breeze/commands/minor_release_command.py
index 0ccb9edb25..2fcbd621ac 100644
--- a/dev/breeze/src/airflow_breeze/commands/minor_release_command.py
+++ b/dev/breeze/src/airflow_breeze/commands/minor_release_command.py
@@ -138,6 +138,7 @@ def instruction_update_version_branch(version_branch):
def create_constraints(version_branch):
if confirm_action("Do you want to create branches from the constraints main?"):
run_command(["git", "checkout", "constraints-main"], dry_run_override=DRY_RUN, check=True)
+ run_command(["git", "pull", "origin", "constraints-main"], dry_run_override=DRY_RUN, check=True)
run_command(
["git", "checkout", "-b", f"constraints-{version_branch}"], dry_run_override=DRY_RUN, check=True
)