You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by pi...@apache.org on 2023/03/06 21:46:44 UTC
[airflow] branch v2-5-test updated (5c0aabb290 -> 2ec4d063b6)
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a change to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
from 5c0aabb290 Fix inconsitencies in checking edit permissions for a DAG (#20346)
new 7fe2ceaad6 Update doc for API clients release policy (#28521)
new b42ace20df Move provider issue generation to breeze (#28352)
new a0ec6fe9fc Improving the release process (#27829)
new 1456f11a99 listener plugin example added (#27905)
new 97912c84dd logging poke info when external dag is not none and task_id and task_ids are none (#28097)
new d6657fdcf4 Add Public Interface description to Airflow documentation (#28300)
new 2a19dd7741 Fix #28391 manual task trigger from UI fails for k8s executor (#28394)
new 13ce6725d4 introduce dag processor job (#28799)
new 76b2796817 Throttle streaming log reads (#28818)
new a833dfb720 Bump swagger-ui-dist from 3.52.0 to 4.1.3 in /airflow/www (#28824)
new 389c074b82 More robust cleanup of executors in test_kubernetes_executor (#28281)
new 9ad5779a54 Annotate KubernetesExecutor pods that we don't delete (#28844)
new 9f3fecbdbc Email Config docs more explicit env var examples (#28845)
new 934e169ed1 clarify that the total_entries property isn't impact by pagination (#28867)
new 1eba989f1b KubenetesExecutor sends state even when successful (#28871)
new 8719079355 Add dep context description for better log message (#28875)
new c22c42dbd4 Remove horizontal lines in TI logs (#28876)
new 0cb731c394 Don't get ES log template from airflow local settings unless necessary (#28882)
new 5a8ddaedd8 Be more selective when adopting pods with KubernetesExecutor (#28899)
new 459412b183 Switch pull-request-target to base branch of pull request (#28921)
new 31bffa8713 Allow URI without authority and host blocks in `airflow connections add` (#28922)
new 061338fad1 Refactor python operators/sensor tests (#28493)
new e9853c7794 Update how PythonSensor returns values from python_callable (#28932)
new 3959765789 improve quick start guide (#28949)
new 9bf17670dc Move project and license docs down in menu to start with developer-focused docs (#28956)
new c487daf254 Bump epoch on python dependencies in Airlfow Image (#28960)
new e64ac0464d Mark license block in doc as text (#28965)
new 20e6ee0d32 Update go client gen command (#28967)
new e2cf933058 Resolve all variables in pickled XCom iterator (#28982)
new 9891eef2ab Fix dag run trigger with a note. (#29228)
new 2a8e414364 Write action log to DB when DAG run is trigged via API (#28998)
new 147d5fe719 fixing import error for dataset (#29007)
new 8fe9d7e477 Remove erroneous TODO comment (#29015)
new 0bab483f43 change context example to no longer us as variable (#29021)
new 94bce2b763 Only skip provider integration tests for non-main builds (#29022)
new c2dd921d5c Annotate and simplify code samples in DAGs doc (#29027)
new 2ec4d063b6 Sanitize url_for arguments before they are passed (#29039)
The 37 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../airflow_helmchart_bug_report.yml | 2 +-
.../get-target-branch-build-scripts/action.yml | 44 +
.github/workflows/build-images.yml | 80 +-
.github/workflows/ci.yml | 42 +-
.gitignore | 3 +
BREEZE.rst | 9 +
Dockerfile.ci | 15 +-
README.md | 8 +-
airflow/__init__.py | 1 +
.../api_connexion/endpoints/dag_run_endpoint.py | 17 +-
airflow/api_connexion/openapi/v1.yaml | 4 +-
airflow/api_connexion/schemas/dag_run_schema.py | 2 +-
airflow/cli/commands/connection_command.py | 5 +-
airflow/cli/commands/dag_processor_command.py | 18 +-
airflow/cli/commands/task_command.py | 2 +-
airflow/config_templates/airflow_local_settings.py | 6 -
airflow/decorators/__init__.pyi | 2 +-
airflow/decorators/branch_python.py | 2 +-
airflow/decorators/sensor.py | 2 +
airflow/example_dags/plugins/event_listener.py | 156 +++
.../plugins/listener_plugin.py} | 12 +-
airflow/executors/kubernetes_executor.py | 87 +-
airflow/jobs/dag_processor_job.py | 69 ++
airflow/models/taskinstance.py | 12 +-
airflow/models/xcom.py | 7 +-
airflow/sensors/external_task.py | 28 +-
airflow/sensors/python.py | 5 +-
airflow/ti_deps/dep_context.py | 1 +
airflow/triggers/base.py | 2 +-
airflow/utils/log/log_reader.py | 17 +-
airflow/www/package.json | 252 ++---
airflow/www/static/js/types/api-generated.ts | 5 +-
airflow/www/views.py | 36 +-
airflow/www/yarn.lock | 8 +-
clients/gen/go.sh | 1 +
dev/README_RELEASE_AIRFLOW.md | 421 ++------
dev/README_RELEASE_PROVIDER_PACKAGES.md | 4 +-
dev/breeze/README.md | 2 +-
dev/breeze/setup.cfg | 7 +
.../commands/minor_release_command.py | 191 ++++
.../commands/release_candidate_command.py | 346 +++++++
.../src/airflow_breeze/commands/release_command.py | 291 ++++++
.../commands/release_management_commands.py | 208 +++-
.../airflow_breeze/commands/testing_commands.py | 9 +
.../src/airflow_breeze/params/shell_params.py | 1 +
.../provider_issue_TEMPLATE.md.jinja2} | 14 +-
dev/breeze/src/airflow_breeze/utils/confirm.py | 17 +
dev/breeze/src/airflow_breeze/utils/console.py | 4 +
dev/provider_packages/prepare_provider_packages.py | 164 +---
.../core-extensions/extra-links.rst | 2 +-
docs/apache-airflow-providers/index.rst | 2 +-
.../authoring-and-scheduling/plugins.rst | 2 +-
docs/apache-airflow/core-concepts/dags.rst | 120 ++-
docs/apache-airflow/howto/custom-operator.rst | 2 +-
...define_extra_link.rst => define-extra-link.rst} | 0
docs/apache-airflow/howto/email-config.rst | 22 +-
docs/apache-airflow/howto/index.rst | 3 +-
docs/apache-airflow/howto/listener-plugin.rst | 95 ++
docs/apache-airflow/howto/operator/index.rst | 4 +-
docs/apache-airflow/index.rst | 13 +-
docs/apache-airflow/license.rst | 2 +-
docs/apache-airflow/public-airflow-interface.rst | 369 +++++++
docs/apache-airflow/python-api-ref.rst | 175 ----
docs/apache-airflow/redirects.txt | 8 +-
docs/apache-airflow/start.rst | 2 +-
docs/conf.py | 9 +-
docs/spelling_wordlist.txt | 1 +
images/breeze/output-commands-hash.txt | 25 +-
images/breeze/output_ci-image.svg | 40 +-
images/breeze/output_release-management.svg | 58 +-
...t_release-management_generate-issue-content.svg | 176 ++++
images/breeze/output_setup.svg | 13 +-
.../output_setup_check-all-params-in-groups.svg | 172 ++++
.../output_setup_regenerate-command-images.svg | 44 +-
images/breeze/output_testing_integration-tests.svg | 18 +-
scripts/ci/docker-compose/_docker.env | 1 +
scripts/ci/docker-compose/base.yml | 1 +
scripts/docker/entrypoint_ci.sh | 13 +-
.../run_prepare_provider_documentation.sh | 5 -
.../endpoints/test_dag_run_endpoint.py | 16 +-
tests/cli/commands/test_connection_command.py | 67 +-
tests/cli/commands/test_dag_processor_command.py | 13 +-
tests/conftest.py | 10 +-
tests/decorators/test_python.py | 143 +--
tests/decorators/test_python_virtualenv.py | 13 -
tests/executors/test_kubernetes_executor.py | 409 ++++----
tests/models/test_taskinstance.py | 49 +-
tests/operators/test_python.py | 1003 +++++++-------------
tests/sensors/test_external_task_sensor.py | 48 +-
tests/sensors/test_python.py | 135 +--
90 files changed, 3737 insertions(+), 2207 deletions(-)
create mode 100644 .github/actions/get-target-branch-build-scripts/action.yml
create mode 100644 airflow/example_dags/plugins/event_listener.py
copy airflow/{providers/amazon/aws/utils/rds.py => example_dags/plugins/listener_plugin.py} (78%)
create mode 100644 airflow/jobs/dag_processor_job.py
create mode 100644 dev/breeze/src/airflow_breeze/commands/minor_release_command.py
create mode 100644 dev/breeze/src/airflow_breeze/commands/release_candidate_command.py
create mode 100644 dev/breeze/src/airflow_breeze/commands/release_command.py
rename dev/{provider_packages/PROVIDER_ISSUE_TEMPLATE.md.jinja2 => breeze/src/airflow_breeze/provider_issue_TEMPLATE.md.jinja2} (57%)
rename docs/apache-airflow/howto/{define_extra_link.rst => define-extra-link.rst} (100%)
create mode 100644 docs/apache-airflow/howto/listener-plugin.rst
create mode 100644 docs/apache-airflow/public-airflow-interface.rst
delete mode 100644 docs/apache-airflow/python-api-ref.rst
create mode 100644 images/breeze/output_release-management_generate-issue-content.svg
create mode 100644 images/breeze/output_setup_check-all-params-in-groups.svg
[airflow] 06/37: Add Public Interface description to Airflow documentation (#28300)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit d6657fdcf435eb541a936416097316ea34e0bf89
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Wed Jan 11 14:37:39 2023 +0100
Add Public Interface description to Airflow documentation (#28300)
Airflow is a complex system and since it is a platform, it is supposed
to be extended by the users by writing custom code wherever they miss
functionality in Airflow's core or its providers.
This page is an attempt to have a single place where we can express
our intention of what is the Public Interface of Airflow that the
user can depend on when implementing such customizations.
This is never 100% possible, we know that some users workflows might
depend on stuff that is internal implementation details or behaviours,
however the user doing so should be aware of the risk they are taking
by relying on something that was not intentionally exposed.
This page is intended to serve as a guideline for the users who would
like to make a decision to rely on some of the Airflow behaviours, so
that they know whether the API they want to rely on were explicitly
intended by Airflow community to expose as Public, or not.
Co-authored-by: Bas Harenslak <ba...@users.noreply.github.com>
Co-authored-by: Bas Harenslak <ba...@users.noreply.github.com>
(cherry picked from commit 352d492c66e69e816fb1547e46fc1e3b7ba32066)
---
.../airflow_helmchart_bug_report.yml | 2 +-
airflow/decorators/__init__.pyi | 2 +-
airflow/decorators/branch_python.py | 2 +-
airflow/decorators/sensor.py | 2 +
airflow/triggers/base.py | 2 +-
.../core-extensions/extra-links.rst | 2 +-
docs/apache-airflow-providers/index.rst | 2 +-
.../authoring-and-scheduling/plugins.rst | 2 +-
docs/apache-airflow/howto/custom-operator.rst | 2 +-
...define_extra_link.rst => define-extra-link.rst} | 0
docs/apache-airflow/howto/index.rst | 2 +-
docs/apache-airflow/howto/operator/index.rst | 4 +-
docs/apache-airflow/index.rst | 9 +-
docs/apache-airflow/public-airflow-interface.rst | 369 +++++++++++++++++++++
docs/apache-airflow/python-api-ref.rst | 175 ----------
docs/apache-airflow/redirects.txt | 8 +-
docs/conf.py | 9 +-
17 files changed, 400 insertions(+), 194 deletions(-)
diff --git a/.github/ISSUE_TEMPLATE/airflow_helmchart_bug_report.yml b/.github/ISSUE_TEMPLATE/airflow_helmchart_bug_report.yml
index 3111db5957..137fea97cf 100644
--- a/.github/ISSUE_TEMPLATE/airflow_helmchart_bug_report.yml
+++ b/.github/ISSUE_TEMPLATE/airflow_helmchart_bug_report.yml
@@ -63,7 +63,7 @@ body:
you paste with ``` ```.
- type: textarea
attributes:
- label: Docker Image customisations
+ label: Docker Image customizations
description: What are the specific modification you've made in your image?
placeholder: >
Did you extend or customise the official Airflow image? Did you add any packages? Maybe
diff --git a/airflow/decorators/__init__.pyi b/airflow/decorators/__init__.pyi
index 0a6d534b24..b0edc7d2c2 100644
--- a/airflow/decorators/__init__.pyi
+++ b/airflow/decorators/__init__.pyi
@@ -166,7 +166,7 @@ class TaskDecoratorCollection:
def branch(self, *, multiple_outputs: bool | None = None, **kwargs) -> TaskDecorator:
"""Create a decorator to wrap the decorated callable into a BranchPythonOperator.
- For more information on how to use this decorator, see :ref:`howto/operator:BranchPythonOperator`.
+ For more information on how to use this decorator, see :ref:`concepts:branching`.
Accepts arbitrary for operator kwarg. Can be reused in a single DAG.
:param multiple_outputs: If set, function return value will be unrolled to multiple XCom values.
diff --git a/airflow/decorators/branch_python.py b/airflow/decorators/branch_python.py
index b7e3a94826..1c90917ed5 100644
--- a/airflow/decorators/branch_python.py
+++ b/airflow/decorators/branch_python.py
@@ -73,7 +73,7 @@ def branch_task(
Wraps a python function into a BranchPythonOperator.
For more information on how to use this operator, take a look at the guide:
- :ref:`howto/operator:BranchPythonOperator`
+ :ref:`concepts:branching`
Accepts kwargs for operator kwarg. Can be reused in a single DAG.
diff --git a/airflow/decorators/sensor.py b/airflow/decorators/sensor.py
index 2033968620..884249c107 100644
--- a/airflow/decorators/sensor.py
+++ b/airflow/decorators/sensor.py
@@ -28,6 +28,7 @@ from airflow.sensors.python import PythonSensor
class DecoratedSensorOperator(PythonSensor):
"""
Wraps a Python callable and captures args/kwargs when called for execution.
+
:param python_callable: A reference to an object that is callable
:param task_id: task Id
:param op_args: a list of positional arguments that will get unpacked when
@@ -63,6 +64,7 @@ class DecoratedSensorOperator(PythonSensor):
def sensor_task(python_callable: Callable | None = None, **kwargs) -> TaskDecorator:
"""
Wraps a function into an Airflow operator.
+
Accepts kwargs for operator kwarg. Can be reused in a single DAG.
:param python_callable: Function to decorate
"""
diff --git a/airflow/triggers/base.py b/airflow/triggers/base.py
index 6bfc0883ef..06bce36a4c 100644
--- a/airflow/triggers/base.py
+++ b/airflow/triggers/base.py
@@ -32,7 +32,7 @@ class BaseTrigger(abc.ABC, LoggingMixin):
- Actively running in a trigger worker
We use the same class for both situations, and rely on all Trigger classes
- to be able to return the (Airflow-JSON-encodable) arguments that will
+ to be able to return the arguments (possible to encode with Airflow-JSON) that will
let them be re-instantiated elsewhere.
"""
diff --git a/docs/apache-airflow-providers/core-extensions/extra-links.rst b/docs/apache-airflow-providers/core-extensions/extra-links.rst
index 884f5906b2..5fe8421b1d 100644
--- a/docs/apache-airflow-providers/core-extensions/extra-links.rst
+++ b/docs/apache-airflow-providers/core-extensions/extra-links.rst
@@ -26,7 +26,7 @@ its own extra links that can redirect users to external systems. The extra link
will be available on the task page.
The operator extra links are explained in
-:doc:`apache-airflow:howto/define_extra_link` and here you can also see the extra links
+:doc:`apache-airflow:howto/define-extra-link` and here you can also see the extra links
provided by the community-managed providers:
.. airflow-extra-links::
diff --git a/docs/apache-airflow-providers/index.rst b/docs/apache-airflow-providers/index.rst
index e81e03b046..86bf050b81 100644
--- a/docs/apache-airflow-providers/index.rst
+++ b/docs/apache-airflow-providers/index.rst
@@ -205,7 +205,7 @@ Displaying package information in CLI/API:
Exposing customized functionality to the Airflow's core:
* ``extra-links`` - this field should contain the list of all operator class names that are adding extra links
- capability. See :doc:`apache-airflow:howto/define_extra_link` for description of how to add extra link
+ capability. See :doc:`apache-airflow:howto/define-extra-link` for description of how to add extra link
capability to the operators of yours.
* ``connection-types`` - this field should contain the list of all connection types together with hook
diff --git a/docs/apache-airflow/authoring-and-scheduling/plugins.rst b/docs/apache-airflow/authoring-and-scheduling/plugins.rst
index bd74de93d0..7c1ebb03b0 100644
--- a/docs/apache-airflow/authoring-and-scheduling/plugins.rst
+++ b/docs/apache-airflow/authoring-and-scheduling/plugins.rst
@@ -257,7 +257,7 @@ definitions in Airflow.
appbuilder_views = [v_appbuilder_package, v_appbuilder_nomenu_package]
appbuilder_menu_items = [appbuilder_mitem, appbuilder_mitem_toplevel]
-.. seealso:: :doc:`/howto/define_extra_link`
+.. seealso:: :doc:`/howto/define-extra-link`
Exclude views from CSRF protection
----------------------------------
diff --git a/docs/apache-airflow/howto/custom-operator.rst b/docs/apache-airflow/howto/custom-operator.rst
index 15f1e79e2c..ea49db33de 100644
--- a/docs/apache-airflow/howto/custom-operator.rst
+++ b/docs/apache-airflow/howto/custom-operator.rst
@@ -270,7 +270,7 @@ If you use a non-existing lexer then the value of the template field will be ren
Define an operator extra link
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-For your operator, you can :doc:`Define an extra link <define_extra_link>` that can
+For your operator, you can :doc:`Define an extra link <define-extra-link>` that can
redirect users to external systems. For example, you can add a link that redirects
the user to the operator's manual.
diff --git a/docs/apache-airflow/howto/define_extra_link.rst b/docs/apache-airflow/howto/define-extra-link.rst
similarity index 100%
rename from docs/apache-airflow/howto/define_extra_link.rst
rename to docs/apache-airflow/howto/define-extra-link.rst
diff --git a/docs/apache-airflow/howto/index.rst b/docs/apache-airflow/howto/index.rst
index 7a170828d5..a2bfcf1ce3 100644
--- a/docs/apache-airflow/howto/index.rst
+++ b/docs/apache-airflow/howto/index.rst
@@ -47,7 +47,7 @@ configuring an Airflow environment.
run-behind-proxy
run-with-systemd
use-test-config
- define_extra_link
+ define-extra-link
email-config
dynamic-dag-generation
docker-compose/index
diff --git a/docs/apache-airflow/howto/operator/index.rst b/docs/apache-airflow/howto/operator/index.rst
index e734e81b0d..2ca7a5e965 100644
--- a/docs/apache-airflow/howto/operator/index.rst
+++ b/docs/apache-airflow/howto/operator/index.rst
@@ -24,9 +24,7 @@ An operator represents a single, ideally idempotent, task. Operators
determine what actually executes when your DAG runs.
.. note::
- See the :doc:`Operators Concepts </core-concepts/operators>` documentation and the
- :doc:`Operators API Reference </python-api-ref>` for more
- information.
+ See the :doc:`Operators Concepts </core-concepts/operators>` documentation.
.. toctree::
:maxdepth: 2
diff --git a/docs/apache-airflow/index.rst b/docs/apache-airflow/index.rst
index 887479422b..eac8ef6cde 100644
--- a/docs/apache-airflow/index.rst
+++ b/docs/apache-airflow/index.rst
@@ -79,6 +79,7 @@ seen running over time:
Each column represents one DAG run. These are two of the most used views in Airflow, but there are several
other views which allow you to deep dive into the state of your workflows.
+
Why Airflow?
=========================================
Airflow is a batch workflow orchestration platform. The Airflow framework contains operators to connect with
@@ -107,8 +108,12 @@ The open-source nature of Airflow ensures you work on components developed, test
blogs posts, articles, conferences, books, and more. You can connect with other peers via several channels
such as `Slack <https://s.apache.org/airflow-slack>`_ and mailing lists.
+Airflow as a Platform is highly customizable. By utilizing :doc:`public-airflow-interface` you can extend
+and customize almost every aspect of Airflow.
+
Why not Airflow?
-=========================================
+================
+
Airflow was built for finite batch workflows. While the CLI and REST API do allow triggering workflows,
Airflow was not built for infinitely-running event-based workflows. Airflow is not a streaming solution.
However, a streaming system such as Apache Kafka is often seen working together with Apache Airflow. Kafka can
@@ -137,6 +142,7 @@ so coding will always be required.
authoring-and-scheduling/index
administration-and-deployment/index
integration
+ public-airflow-interface
best-practices
faq
Release Policies <release-process>
@@ -150,7 +156,6 @@ so coding will always be required.
Operators and hooks <operators-and-hooks-ref>
CLI <cli-and-env-variables-ref>
Templates <templates-ref>
- Python API <python-api-ref>
Stable REST API <stable-rest-api-ref>
deprecated-rest-api-ref
Configurations <configurations-ref>
diff --git a/docs/apache-airflow/public-airflow-interface.rst b/docs/apache-airflow/public-airflow-interface.rst
new file mode 100644
index 0000000000..523e6aa215
--- /dev/null
+++ b/docs/apache-airflow/public-airflow-interface.rst
@@ -0,0 +1,369 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Public Interface of Airflow
+...........................
+
+The Public Interface of Apache Airflow is a set of interfaces that allow developers to interact
+with and access certain features of the Apache Airflow system. This includes operations such as
+creating and managing DAGs (Directed Acyclic Graphs), managing tasks and their dependencies,
+and extending Airflow capabilities by writing new executors, plugins, operators and providers. The
+Public Interface can be useful for building custom tools and integrations with other systems,
+and for automating certain aspects of the Airflow workflow.
+
+Using Airflow Public Interfaces
+===============================
+
+Using Airflow Public Interfaces is needed when you want to interact with Airflow programmatically:
+
+* When you are extending Airflow classes such as Operators and Hooks. This can be done by DAG authors to add missing functionality in their DAGs or by those who write reusable custom operators for other DAG authors.
+* When writing new :doc:`Plugins <authoring-and-scheduling/plugins>` that extend Airflow's functionality beyond
+ DAG building blocks. Secrets, Timetables, Triggers, Listeners are all examples of such functionality. This
+ is usually done by users who manage Airflow instances.
+* Bundling custom Operators, Hooks, Plugins and releasing them together via
+ :doc:`provider packages <apache-airflow-providers:index>` - this is usually done by those who intend to
+ provide a reusable set of functionality for external services or applications Airflow integrates with.
+
+All the ways above involve extending or using Airflow Python classes and functions. The classes
+and functions mentioned below can be relied on to keep backwards-compatible signatures and behaviours within
+MAJOR version of Airflow. On the other hand, classes and methods starting with ``_`` (also known
+as protected Python methods) and ``__`` (also known as private Python methods) are not part of the Public
+Airflow Interface and might change at any time.
+
+You can also use Airflow's Public Interface via the `Stable REST API <stable-rest-api-ref>`_ (based on the
+OpenAPI specification). For specific needs you can also use the
+`Airflow Command Line Interface (CLI) <cli-and-env-variables-ref.rst>`_ though it's behaviour might change
+in details (such as output format and available flags) so if you want to rely on those in programmatic
+way, the Stable REST API is recommended.
+
+
+Using the Public Interface for DAG Authors
+==========================================
+
+DAGs
+----
+
+The DAG is Airflow's core entity that represents a recurring workflow. You can create a DAG by
+instantiating the :class:`~airflow.models.dag.DAG` class in your DAG file.
+
+Airflow has a set of example DAGs that you can use to learn how to write DAGs
+
+.. toctree::
+ :includehidden:
+ :glob:
+ :maxdepth: 1
+
+ _api/airflow/example_dags/index
+
+You can read more about DAGs in :doc:`DAGs <core-concepts/dags>`.
+
+.. _pythonapi:operators:
+
+Operators
+---------
+
+Operators allow for generation of certain types of tasks that become nodes in
+the DAG when instantiated.
+
+There are 3 main types of operators:
+
+- Operators that performs an **action**, or tell another system to
+ perform an action
+- **Transfer** operators move data from one system to another
+- **Sensors** are a certain type of operator that will keep running until a
+ certain criterion is met. Examples include a specific file landing in HDFS or
+ S3, a partition appearing in Hive, or a specific time of the day. Sensors
+ are derived from :class:`~airflow.sensors.base.BaseSensorOperator` and run a poke
+ method at a specified :attr:`~airflow.sensors.base.BaseSensorOperator.poke_interval` until it
+ returns ``True``.
+
+All operators are derived from :class:`~airflow.models.baseoperator.BaseOperator` and acquire much
+functionality through inheritance. Since this is the core of the engine,
+it's worth taking the time to understand the parameters of :class:`~airflow.models.baseoperator.BaseOperator`
+to understand the primitive features that can be leveraged in your DAGs.
+
+Airflow has a set of Operators that are considered public. You are also free to extend their functionality
+by extending them:
+
+.. toctree::
+ :includehidden:
+ :glob:
+ :maxdepth: 1
+
+ _api/airflow/operators/index
+
+ _api/airflow/sensors/index
+
+
+You can read more about the operators in :doc:`core-concepts/operators`, :doc:`core-concepts/sensors`.
+Also you can learn how to write a custom operator in :doc:`howto/custom-operator`.
+
+.. _pythonapi:hooks:
+
+Hooks
+-----
+
+Hooks are interfaces to external platforms and databases, implementing a common
+interface when possible and acting as building blocks for operators. All hooks
+are derived from :class:`~airflow.hooks.base.BaseHook`.
+
+Airflow has a set of Hooks that are considered public. You are free to extend their functionality
+by extending them:
+
+.. toctree::
+ :includehidden:
+ :glob:
+ :maxdepth: 1
+
+ _api/airflow/hooks/index
+
+Public Airflow utilities
+------------------------
+
+When writing or extending Hooks and Operators, DAG authors and developers can
+use the following classes:
+
+* The :class:`~airflow.models.connection.Connection`, which provides access to external service credentials and configuration.
+* The :class:`~airflow.models.variable.Variable`, which provides access to Airflow configuration variables.
+* The :class:`~airflow.models.xcom.XCom` which are used to access to inter-task communication data.
+
+You can read more about the public Airflow utilities in :doc:`howto/connection`,
+:doc:`core-concepts/variables`, :doc:`core-concepts/xcoms`
+
+Public Exceptions
+-----------------
+
+When writing the custom Operators and Hooks, you can handle and raise public Exceptions that Airflow
+exposes:
+
+.. toctree::
+ :includehidden:
+ :glob:
+ :maxdepth: 1
+
+ _api/airflow/exceptions/index
+
+
+Using Public Interface to extend Airflow capabilities
+=====================================================
+
+Airflow uses Plugin mechanism to extend Airflow platform capabilities. They allow to extend
+Airflow UI but also they are the way to expose the below customizations (Triggers, Timetables, Listeners, etc.).
+Providers can also implement plugin endpoints and customize Airflow UI and the customizations.
+
+You can read more about plugins in :doc:`authoring-and-scheduling/plugins`. You can read how to extend
+Airflow UI in :doc:`howto/custom-view-plugin`. Note that there are some simple customizations of the UI
+that do not require plugins - you can read more about them in :doc:`howto/customize-ui`.
+
+Here are the ways how Plugins can be used to extend Airflow:
+
+Triggers
+--------
+
+Airflow uses Triggers to implement ``asyncio`` compatible Deferrable Operators.
+All Triggers derive from :class:`~airflow.triggers.base.BaseTrigger`.
+
+Airflow has a set of Triggers that are considered public. You are free to extend their functionality
+by extending them:
+
+.. toctree::
+ :includehidden:
+ :glob:
+ :maxdepth: 1
+
+ _api/airflow/triggers/index
+
+You can read more about Triggers in :doc:`authoring-and-scheduling/deferring`.
+
+Timetables
+----------
+
+Custom timetable implementations provide Airflow's scheduler additional logic to
+schedule DAG runs in ways not possible with built-in schedule expressions.
+All Timetables derive from :class:`~airflow.timetables.base.Timetable`.
+
+Airflow has a set of Timetables that are considered public. You are free to extend their functionality
+by extending them:
+
+.. toctree::
+ :includehidden:
+ :maxdepth: 1
+
+ _api/airflow/timetables/index
+
+You can read more about Timetables in :doc:`howto/timetable`.
+
+Listeners
+---------
+
+Listeners enable you to respond to DAG/Task lifecycle events.
+
+This is implemented via :class:`~airflow.listeners.listener.ListenerManager` class that provides hooks that
+can be implemented to respond to DAG/Task lifecycle events.
+
+.. versionadded:: 2.5
+
+ Listener public interface has been added in version 2.5.
+
+You can read more about Listeners in :doc:`administration-and-deployment/listeners`.
+
+Extra Links
+-----------
+
+Extra links are dynamic links that could be added to Airflow independently from custom Operators. Normally
+they can be defined by the Operators, but plugins allow you to override the links on a global level.
+
+You can read more about the Extra Links in :doc:`/howto/define-extra-link`.
+
+Using Public Interface to integrate with external services and applications
+===========================================================================
+
+
+Tasks in Airflow can orchestrate external services via Hooks and Operators. The core functionality of
+Airflow (such as authentication) can also be extended to leverage external services.
+You can read more about providers :doc:`provider packages <apache-airflow-providers:index>` and core
+extensions they can provide in :doc:`provider packages <apache-airflow-providers:core-extensions/index>`.
+
+Executors
+---------
+
+Executors are the mechanism by which task instances get run. All executors are
+derived from :class:`~airflow.executors.base_executor.BaseExecutor`. There are several
+executor implementations built-in Airflow, each with its own unique characteristics and capabilities.
+
+Airflow has a set of Executors that are considered public. You are free to extend their functionality
+by extending them:
+
+.. toctree::
+ :includehidden:
+ :glob:
+ :maxdepth: 1
+
+ _api/airflow/executors/index
+
+You can read more about executors in :doc:`core-concepts/executor/index`.
+
+.. versionadded:: 2.6
+
+ Executor interface was available in earlier version of Airflow but only as of version 2.6 executors are
+ fully decoupled and Airflow does not rely on built-in set of executors.
+ You could have implemented (and succeeded) with implementing Executors before Airflow 2.6 and a number
+ of people succeeded in doing so, but there were some hard-coded behaviours that preferred in-built
+ executors, and custom executors could not provide full functionality that built-in executors had.
+
+Secrets Backends
+----------------
+
+Airflow can be configured to rely on secrets backends to retrieve
+:class:`~airflow.models.connection.Connection` and :class:`~airflow.models.Variables`.
+All secrets backends derive from :class:`~airflow.secrets.BaseSecretsBackend`.
+
+All Secrets Backend implementations are public. You can extend their functionality:
+
+.. toctree::
+ :includehidden:
+ :glob:
+ :maxdepth: 1
+
+ _api/airflow/secrets/index
+
+You can read more about Secret Backends in :doc:`administration-and-deployment/security/secrets/secrets-backend/index`.
+You can also find all the available Secrets Backends implemented in community providers
+in :doc:`apache-airflow-providers:core-extensions/secrets-backends`.
+
+Authentication Backends
+-----------------------
+
+Authentication backends can extend the way how Airflow authentication mechanism works. You can find out more
+about authentication in :doc:`apache-airflow-providers:core-extensions/auth-backends` that also shows available
+Authentication backends implemented in the community providers.
+
+Connections
+-----------
+
+When creating Hooks, you can add custom Connections. You can read more
+about connections in :doc:`apache-airflow-providers:core-extensions/connections` for available
+Connections implemented in the community providers.
+
+Extra Links
+-----------
+
+When creating Hooks, you can add custom Extra Links that are displayed when the tasks are run.
+You can find out more about extra links in :doc:`apache-airflow-providers:core-extensions/extra-links`
+that also shows available extra links implemented in the community providers.
+
+Logging and Monitoring
+----------------------
+
+You can extend the way how logs are written by Airflow. You can find out more about log writing in
+:doc:`administration-and-deployment/logging-monitoring/index`.
+
+The :doc:`apache-airflow-providers:core-extensions/logging` that also shows available log writers
+implemented in the community providers.
+
+Decorators
+----------
+DAG authors can use decorators to author DAGs using the :doc:`TaskFlow <core-concepts/taskflow>` concept.
+All Decorators derive from :class:`~airflow.decorators.base.TaskDecorator`.
+
+Airflow has a set of Decorators that are considered public. You are free to extend their functionality
+by extending them:
+
+.. toctree::
+ :includehidden:
+ :maxdepth: 1
+
+ _api/airflow/decorators/index
+
+You can read more about creating custom Decorators in :doc:`howto/create-custom-decorator`.
+
+Email notifications
+-------------------
+
+Airflow has a built-in way of sending email notifications and it allows to extend it by adding custom
+email notification classes. You can read more about email notifications in :doc:`howto/email-config`.
+
+Cluster Policies
+----------------
+
+Cluster Policies are the way to dynamically apply cluster-wide policies to the DAGs being parsed or tasks
+being executed. You can read more about Cluster Policies in :doc:`administration-and-deployment/cluster-policies`.
+
+Lineage
+-------
+
+Airflow can help track origins of data, what happens to it and where it moves over time. You can read more
+about lineage in :doc:`administration-and-deployment/lineage`.
+
+
+What is not part of the Public Interface of Apache Airflow?
+===========================================================
+
+Everything not mentioned in this document should be considered as non-Public Interface.
+
+Sometimes in other applications those components could be relied on to keep backwards compatibility,
+but in Airflow they are not parts of the Public Interface and might change any time:
+
+* `Database structure <database-erd-ref>`_ is considered to be an internal implementation
+ detail and you should not assume the structure is going to be maintained in a
+ backwards-compatible way.
+
+* `Web UI <ui>`_ is continuously evolving and there are no backwards compatibility guarantees on HTML elements.
+
+* Python classes except those explicitly mentioned in this document, are considered an
+ internal implementation detail and you should not assume they will be maintained
+ in a backwards-compatible way.
diff --git a/docs/apache-airflow/python-api-ref.rst b/docs/apache-airflow/python-api-ref.rst
deleted file mode 100644
index 9e385f846d..0000000000
--- a/docs/apache-airflow/python-api-ref.rst
+++ /dev/null
@@ -1,175 +0,0 @@
- .. Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- .. http://www.apache.org/licenses/LICENSE-2.0
-
- .. Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-
-
-
-Python API Reference
-====================
-
-.. _pythonapi:dags:
-
-DAGs
----------
-The DAG is Airflow's core model that represents a recurring workflow. Check out :class:`~airflow.models.dag.DAG` for details.
-
-.. _pythonapi:operators:
-
-Operators
----------
-Operators allow for generation of certain types of tasks that become nodes in
-the DAG when instantiated. All operators derive from :class:`~airflow.models.baseoperator.BaseOperator` and
-inherit many attributes and methods that way.
-
-There are 3 main types of operators:
-
-- Operators that performs an **action**, or tell another system to
- perform an action
-- **Transfer** operators move data from one system to another
-- **Sensors** are a certain type of operator that will keep running until a
- certain criterion is met. Examples include a specific file landing in HDFS or
- S3, a partition appearing in Hive, or a specific time of the day. Sensors
- are derived from :class:`~airflow.sensors.base.BaseSensorOperator` and run a poke
- method at a specified :attr:`~airflow.sensors.base.BaseSensorOperator.poke_interval` until it returns ``True``.
-
-BaseOperator
-''''''''''''
-All operators are derived from :class:`~airflow.models.baseoperator.BaseOperator` and acquire much
-functionality through inheritance. Since this is the core of the engine,
-it's worth taking the time to understand the parameters of :class:`~airflow.models.baseoperator.BaseOperator`
-to understand the primitive features that can be leveraged in your
-DAGs.
-
-BaseSensorOperator
-''''''''''''''''''
-All sensors are derived from :class:`~airflow.sensors.base.BaseSensorOperator`. All sensors inherit
-the :attr:`~airflow.sensors.base.BaseSensorOperator.timeout` and :attr:`~airflow.sensors.base.BaseSensorOperator.poke_interval` on top of the :class:`~airflow.models.baseoperator.BaseOperator`
-attributes.
-
-Operators packages
-''''''''''''''''''
-All operators are in the following packages:
-
-.. toctree::
- :includehidden:
- :glob:
- :maxdepth: 1
-
- _api/airflow/operators/index
-
- _api/airflow/sensors/index
-
-
-.. _pythonapi:hooks:
-
-Hooks
------
-Hooks are interfaces to external platforms and databases, implementing a common
-interface when possible and acting as building blocks for operators. All hooks
-are derived from :class:`~airflow.hooks.base.BaseHook`.
-
-Hooks packages
-''''''''''''''
-All hooks are in the following packages:
-
-.. toctree::
- :includehidden:
- :glob:
- :maxdepth: 1
-
- _api/airflow/hooks/index
-
-Executors
----------
-Executors are the mechanism by which task instances get run. All executors are
-derived from :class:`~airflow.executors.base_executor.BaseExecutor`.
-
-Executors packages
-''''''''''''''''''
-All executors are in the following packages:
-
-.. toctree::
- :includehidden:
- :glob:
- :maxdepth: 1
-
- _api/airflow/executors/index
-
-Models
-------
-Models are built on top of the SQLAlchemy ORM Base class, and instances are
-persisted in the database.
-
-.. toctree::
- :includehidden:
- :glob:
- :maxdepth: 1
-
- _api/airflow/models/index
-
-.. _pythonapi:exceptions:
-
-Exceptions
-----------
-
-.. toctree::
- :includehidden:
- :glob:
- :maxdepth: 1
-
- _api/airflow/exceptions/index
-
-Secrets Backends
-----------------
-Airflow relies on secrets backends to retrieve :class:`~airflow.models.connection.Connection` objects.
-All secrets backends derive from :class:`~airflow.secrets.BaseSecretsBackend`.
-
-.. toctree::
- :includehidden:
- :glob:
- :maxdepth: 1
-
- _api/airflow/secrets/index
-
-Timetables
-----------
-Custom timetable implementations provide Airflow's scheduler additional logic to
-schedule DAG runs in ways not possible with built-in schedule expressions.
-
-.. toctree::
- :includehidden:
- :maxdepth: 1
-
- _api/airflow/timetables/index
-
-Example DAGs
-------------
-
-.. toctree::
- :includehidden:
- :maxdepth: 1
-
- _api/airflow/example_dags/index
-
-Utils
------
-
-.. toctree::
- :hidden:
- :glob:
- :maxdepth: 1
-
- _api/airflow/utils/dag_parsing_context/index
diff --git a/docs/apache-airflow/redirects.txt b/docs/apache-airflow/redirects.txt
index 20e11a493a..b99a5b253c 100644
--- a/docs/apache-airflow/redirects.txt
+++ b/docs/apache-airflow/redirects.txt
@@ -46,7 +46,7 @@ start/index.rst start.rst
# References
cli-ref.rst cli-and-env-variables-ref.rst
-_api/index.rst python-api-ref.rst
+_api/index.rst public-airflow-interface.rst
rest-api-ref.rst deprecated-rest-api-ref.rst
macros-ref.rst templates-ref.rst
@@ -130,3 +130,9 @@ executor/celery.rst core-concepts/executor/celery.rst
executor/local.rst core-concepts/executor/local.rst
executor/sequential.rst core-concepts/executor/sequential.rst
upgrading-from-1-10/upgrade-check.rst howto/upgrading-from-1-10/upgrade-check.rst
+
+# Python API
+python-api-ref.rst public-airflow-interface.rst
+
+# Typos
+howto/define_extra_link.rst howto/define-extra-link.rst
diff --git a/docs/conf.py b/docs/conf.py
index 2e9340cfe8..c88fd2aeec 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -221,17 +221,18 @@ if PACKAGE_NAME == "apache-airflow":
browsable_packages = {
"hooks",
+ "decorators",
"example_dags",
"executors",
- "models",
"operators",
"providers",
"secrets",
"sensors",
"timetables",
+ "triggers",
"utils",
}
- browseable_utils = {"dag_parsing_context.py"}
+ browsable_utils: set[str] = set()
root = ROOT_DIR / "airflow"
for path in root.iterdir():
@@ -240,9 +241,9 @@ if PACKAGE_NAME == "apache-airflow":
if path.is_dir() and path.name not in browsable_packages:
exclude_patterns.append(f"_api/airflow/{path.name}")
- # Don't include all of utils, just the specific ones we include in python-api-ref
+ # Don't include all of utils, just the specific ones we decoded to include
for path in (root / "utils").iterdir():
- if path.name not in browseable_utils:
+ if path.name not in browsable_utils:
exclude_patterns.append(_get_rst_filepath_from_path(path))
elif PACKAGE_NAME != "docker-stack":
exclude_patterns.extend(
[airflow] 27/37: Mark license block in doc as text (#28965)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit e64ac0464de701265fbf1dddfe751b7a78a69295
Author: Tzu-ping Chung <ur...@gmail.com>
AuthorDate: Tue Jan 17 01:07:30 2023 +0800
Mark license block in doc as text (#28965)
Sphinx was helpfully (but incorrectly) adding Python syntax highlighting
to the block. This marks the block explicitly as pure text to avoid
that.
(cherry picked from commit 4fff0555b505f7a51efb2bbd108378780eb850d9)
---
docs/apache-airflow/license.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/apache-airflow/license.rst b/docs/apache-airflow/license.rst
index ceec3e78b6..e5bb13e36b 100644
--- a/docs/apache-airflow/license.rst
+++ b/docs/apache-airflow/license.rst
@@ -23,7 +23,7 @@ License
.. image:: img/apache.jpg
:width: 150
-::
+.. code-block:: text
Apache License
Version 2.0, January 2004
[airflow] 20/37: Switch pull-request-target to base branch of pull request (#28921)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 459412b1833665370b14f44dfbfb2a5845d971d5
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Fri Jan 13 14:34:32 2023 +0100
Switch pull-request-target to base branch of pull request (#28921)
When pull request target workflow runs, we used build scripts from
main - to couple it with github workflows, but GitHub actually uses
base branch now for the workflows, so we should also use the target
branch for that.
This PR changes it and also extracts the code to do the checkout
to a separate composite action.
(cherry picked from commit 50f20037e8793913c6da2b296462a744dd02e23c)
---
.../get-target-branch-build-scripts/action.yml | 44 ++++++++++++
.github/workflows/build-images.yml | 80 +++-------------------
.gitignore | 3 +
3 files changed, 55 insertions(+), 72 deletions(-)
diff --git a/.github/actions/get-target-branch-build-scripts/action.yml b/.github/actions/get-target-branch-build-scripts/action.yml
new file mode 100644
index 0000000000..3d32578a2f
--- /dev/null
+++ b/.github/actions/get-target-branch-build-scripts/action.yml
@@ -0,0 +1,44 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: 'Gets target branch build scripts'
+description: 'Checks out target branch build scripts (including breeze) and replaces the current version'
+runs:
+ using: "composite"
+ steps:
+ - name: Checkout target branch to 'target-airflow' folder to use ci/scripts and breeze from there.
+ uses: actions/checkout@v3
+ with:
+ path: "target-airflow"
+ ref: ${{ github.base_ref }}
+ persist-credentials: false
+ submodules: recursive
+ - name: >
+ Override "scripts/ci", "dev" and "./github/actions" with the target branch
+ so that the PR does not override it
+ # We should not override those scripts which become part of the image as they will not be
+ # changed in the image built - we should only override those that are executed to build
+ # the image.
+ shell: bash
+ run: |
+ rm -rfv "scripts/ci"
+ mv -v "target-airflow/scripts/ci" "scripts"
+ rm -rfv "dev"
+ mv -v "target-airflow/dev" "."
+ rm -rfv "./github/actions"
+ mv -v "target-airflow/.github/actions" "actions"
diff --git a/.github/workflows/build-images.yml b/.github/workflows/build-images.yml
index b6934b6555..78f8468493 100644
--- a/.github/workflows/build-images.yml
+++ b/.github/workflows/build-images.yml
@@ -138,12 +138,8 @@ jobs:
# Stdout is redirected to GITHUB_ENV but we also print it to stderr to see it in ci log
print(output, file=sys.stderr)
EOF
- - name: Checkout main branch to use breeze from there.
- uses: actions/checkout@v3
- with:
- ref: "main"
- persist-credentials: false
- submodules: recursive
+ - name: "Get target branch build scripts"
+ uses: ./.github/actions/get-target-branch-build-scripts
- name: "Install Breeze"
uses: ./.github/actions/breeze
- name: Selective checks
@@ -183,28 +179,8 @@ jobs:
ref: ${{ needs.build-info.outputs.target-commit-sha }}
persist-credentials: false
submodules: recursive
- - name: >
- Checkout "main branch to 'main-airflow' folder
- to use ci/scripts from there.
- uses: actions/checkout@v3
- with:
- path: "main-airflow"
- ref: "main"
- persist-credentials: false
- submodules: recursive
- - name: >
- Override "scripts/ci", "dev" and "./github/actions" with the "main" branch
- so that the PR does not override it
- # We should not override those scripts which become part of the image as they will not be
- # changed in the image built - we should only override those that are executed to build
- # the image.
- run: |
- rm -rfv "scripts/ci"
- mv -v "main-airflow/scripts/ci" "scripts"
- rm -rfv "dev"
- mv -v "main-airflow/dev" "."
- rm -rfv "./github/actions"
- mv -v "main-airflow/.github/actions" "actions"
+ - name: "Get target branch build scripts"
+ uses: ./.github/actions/get-target-branch-build-scripts
- name: >
Build CI Images ${{needs.build-info.outputs.all-python-versions-list-as-string}}:${{env.IMAGE_TAG}}
uses: ./.github/actions/build-ci-images
@@ -239,28 +215,8 @@ jobs:
ref: ${{ needs.build-info.outputs.target-commit-sha }}
persist-credentials: false
submodules: recursive
- - name: >
- Checkout "main" branch to 'main-airflow' folder
- to use ci/scripts from there.
- uses: actions/checkout@v3
- with:
- path: "main-airflow"
- ref: "main"
- persist-credentials: false
- submodules: recursive
- - name: >
- Override "scripts/ci", "dev" and "./github/actions" with the "main" branch
- so that the PR does not override it
- # We should not override those scripts which become part of the image as they will not be
- # changed in the image built - we should only override those that are executed to build
- # the image.
- run: |
- rm -rfv "scripts/ci"
- mv -v "main-airflow/scripts/ci" "scripts"
- rm -rfv "dev"
- mv -v "main-airflow/dev" "."
- rm -rfv "./github/actions"
- mv -v "main-airflow/.github/actions" "actions"
+ - name: "Get target branch build scripts"
+ uses: ./.github/actions/get-target-branch-build-scripts
- name: >
Build PROD Images
${{needs.build-info.outputs.all-python-versions-list-as-string}}:${{env.IMAGE_TAG}}
@@ -296,28 +252,8 @@ jobs:
ref: ${{ needs.build-info.outputs.target-commit-sha }}
persist-credentials: false
submodules: recursive
- - name: >
- Checkout "main" branch to 'main-airflow' folder
- to use ci/scripts from there.
- uses: actions/checkout@v3
- with:
- path: "main-airflow"
- ref: "main"
- persist-credentials: false
- submodules: recursive
- - name: >
- Override "scripts/ci", "dev" and "./github/actions" with the "main" branch
- so that the PR does not override it
- # We should not override those scripts which become part of the image as they will not be
- # changed in the image built - we should only override those that are executed to build
- # the image.
- run: |
- rm -rfv "scripts/ci"
- mv -v "main-airflow/scripts/ci" "scripts"
- rm -rfv "dev"
- mv -v "main-airflow/dev" "."
- rm -rfv "./github/actions"
- mv -v "main-airflow/.github/actions" "actions"
+ - name: "Get target branch build scripts"
+ uses: ./.github/actions/get-target-branch-build-scripts
- name: "Start ARM instance"
run: ./scripts/ci/images/ci_start_arm_instance_and_connect_to_docker.sh
- name: "Install Breeze"
diff --git a/.gitignore b/.gitignore
index edd1362f96..ebce7707b6 100644
--- a/.gitignore
+++ b/.gitignore
@@ -230,3 +230,6 @@ licenses/LICENSES-ui.txt
# files generated by memray
*.py.*.html
*.py.*.bin
+
+# used to checkout target-branch in CI
+/target-airflow
[airflow] 23/37: Update how PythonSensor returns values from python_callable (#28932)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit e9853c779428eb586955ffae9316ce89bfda510d
Author: SoxMax <ir...@gmail.com>
AuthorDate: Wed Jan 18 19:57:31 2023 -0500
Update how PythonSensor returns values from python_callable (#28932)
* Update how PythonSensor returns values from python_callable
* test poke returns the xcom value
* update test to only run poke
* reformat based on changes
* use full if rather than ternary
(cherry picked from commit b0f302e027d09a50493f4cdd808559984e433ee1)
---
airflow/sensors/python.py | 5 ++++-
tests/sensors/test_python.py | 11 +++++++++++
2 files changed, 15 insertions(+), 1 deletion(-)
diff --git a/airflow/sensors/python.py b/airflow/sensors/python.py
index 615e4e20ee..0a91031fd6 100644
--- a/airflow/sensors/python.py
+++ b/airflow/sensors/python.py
@@ -71,4 +71,7 @@ class PythonSensor(BaseSensorOperator):
self.log.info("Poking callable: %s", str(self.python_callable))
return_value = self.python_callable(*self.op_args, **self.op_kwargs)
- return PokeReturnValue(bool(return_value))
+ if isinstance(return_value, PokeReturnValue):
+ return return_value
+ else:
+ return PokeReturnValue(bool(return_value))
diff --git a/tests/sensors/test_python.py b/tests/sensors/test_python.py
index 73a0ddffe4..ec515d8dee 100644
--- a/tests/sensors/test_python.py
+++ b/tests/sensors/test_python.py
@@ -23,6 +23,7 @@ from datetime import date
import pytest
from airflow.exceptions import AirflowSensorTimeout
+from airflow.sensors.base import PokeReturnValue
from airflow.sensors.python import PythonSensor
from tests.operators.test_python import BasePythonTest
@@ -41,6 +42,16 @@ class TestPythonSensor(BasePythonTest):
with pytest.raises(ZeroDivisionError):
self.run_as_task(lambda: 1 / 0)
+ def test_python_sensor_xcom(self):
+ with self.dag:
+ task = self.opcls(
+ task_id=self.task_id,
+ python_callable=lambda: PokeReturnValue(True, "xcom"),
+ **self.default_kwargs(),
+ )
+ poke_result = task.poke({})
+ assert poke_result.xcom_value == "xcom"
+
def test_python_callable_arguments_are_templatized(self):
"""Test PythonSensor op_args are templatized"""
# Create a named tuple and ensure it is still preserved
[airflow] 05/37: logging poke info when external dag is not none and task_id and task_ids are none (#28097)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 97912c84ddef640bac2061e3838ff1dde3e66a28
Author: Srinivasa Surabathini <40...@users.noreply.github.com>
AuthorDate: Fri Jan 20 18:15:31 2023 +0000
logging poke info when external dag is not none and task_id and task_ids are none (#28097)
(cherry picked from commit 760c52949ac41ffa7a2357aa1af0cdca163ddac8)
---
airflow/sensors/external_task.py | 28 +++++++++++++----
tests/sensors/test_external_task_sensor.py | 48 ++++++++++++++++++++++++++++--
2 files changed, 68 insertions(+), 8 deletions(-)
diff --git a/airflow/sensors/external_task.py b/airflow/sensors/external_task.py
index 967bb5a276..d1c5443b92 100644
--- a/airflow/sensors/external_task.py
+++ b/airflow/sensors/external_task.py
@@ -143,18 +143,27 @@ class ExternalTaskSensor(BaseSensorOperator):
if external_task_id is not None and external_task_ids is not None:
raise ValueError(
"Only one of `external_task_id` or `external_task_ids` may "
- "be provided to ExternalTaskSensor; not both."
+ "be provided to ExternalTaskSensor; "
+ "use external_task_id or external_task_ids or external_task_group_id."
)
- if external_task_id is not None:
- external_task_ids = [external_task_id]
+ if external_task_group_id is not None and external_task_id is not None:
+ raise ValueError(
+ "Only one of `external_task_group_id` or `external_task_id` may "
+ "be provided to ExternalTaskSensor; "
+ "use external_task_id or external_task_ids or external_task_group_id."
+ )
- if external_task_group_id and external_task_ids:
+ if external_task_group_id is not None and external_task_ids is not None:
raise ValueError(
- "Values for `external_task_group_id` and `external_task_id` or `external_task_ids` "
- "can't be set at the same time"
+ "Only one of `external_task_group_id` or `external_task_ids` may "
+ "be provided to ExternalTaskSensor; "
+ "use external_task_id or external_task_ids or external_task_group_id."
)
+ if external_task_id is not None:
+ external_task_ids = [external_task_id]
+
if external_task_ids or external_task_group_id:
if not total_states <= set(State.task_states):
raise ValueError(
@@ -217,6 +226,13 @@ class ExternalTaskSensor(BaseSensorOperator):
serialized_dttm_filter,
)
+ if self.external_dag_id and not self.external_task_group_id and not self.external_task_ids:
+ self.log.info(
+ "Poking for DAG '%s' on %s ... ",
+ self.external_dag_id,
+ serialized_dttm_filter,
+ )
+
# In poke mode this will check dag existence only once
if self.check_existence and not self._has_checked_existence:
self._check_for_existence(session=session)
diff --git a/tests/sensors/test_external_task_sensor.py b/tests/sensors/test_external_task_sensor.py
index b594210b13..1b5a5032cf 100644
--- a/tests/sensors/test_external_task_sensor.py
+++ b/tests/sensors/test_external_task_sensor.py
@@ -145,18 +145,49 @@ class TestExternalTaskSensor(unittest.TestCase):
)
op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE, ignore_ti_state=True)
+ def test_raise_with_external_task_sensor_task_id_and_task_ids(self):
+ with pytest.raises(ValueError) as ctx:
+ ExternalTaskSensor(
+ task_id="test_external_task_sensor_task_id_with_task_ids_failed_status",
+ external_dag_id=TEST_DAG_ID,
+ external_task_id=TEST_TASK_ID,
+ external_task_ids=TEST_TASK_ID,
+ dag=self.dag,
+ )
+ assert (
+ str(ctx.value) == "Only one of `external_task_id` or `external_task_ids` may "
+ "be provided to ExternalTaskSensor; "
+ "use external_task_id or external_task_ids or external_task_group_id."
+ )
+
def test_raise_with_external_task_sensor_task_group_and_task_id(self):
with pytest.raises(ValueError) as ctx:
ExternalTaskSensor(
task_id="test_external_task_sensor_task_group_with_task_id_failed_status",
external_dag_id=TEST_DAG_ID,
+ external_task_id=TEST_TASK_ID,
+ external_task_group_id=TEST_TASK_GROUP_ID,
+ dag=self.dag,
+ )
+ assert (
+ str(ctx.value) == "Only one of `external_task_group_id` or `external_task_id` may "
+ "be provided to ExternalTaskSensor; "
+ "use external_task_id or external_task_ids or external_task_group_id."
+ )
+
+ def test_raise_with_external_task_sensor_task_group_and_task_ids(self):
+ with pytest.raises(ValueError) as ctx:
+ ExternalTaskSensor(
+ task_id="test_external_task_sensor_task_group_with_task_ids_failed_status",
+ external_dag_id=TEST_DAG_ID,
external_task_ids=TEST_TASK_ID,
external_task_group_id=TEST_TASK_GROUP_ID,
dag=self.dag,
)
assert (
- str(ctx.value) == "Values for `external_task_group_id` and `external_task_id` or "
- "`external_task_ids` can't be set at the same time"
+ str(ctx.value) == "Only one of `external_task_group_id` or `external_task_ids` may "
+ "be provided to ExternalTaskSensor; "
+ "use external_task_id or external_task_ids or external_task_group_id."
)
# by default i.e. check_existence=False, if task_group doesn't exist, the sensor will run till timeout,
@@ -354,6 +385,19 @@ class TestExternalTaskSensor(unittest.TestCase):
)
op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE, ignore_ti_state=True)
+ def test_external_dag_sensor_log(self, caplog):
+ other_dag = DAG("other_dag", default_args=self.args, end_date=DEFAULT_DATE, schedule="@once")
+ other_dag.create_dagrun(
+ run_id="test", start_date=DEFAULT_DATE, execution_date=DEFAULT_DATE, state=State.SUCCESS
+ )
+ op = ExternalTaskSensor(
+ task_id="test_external_dag_sensor_check",
+ external_dag_id="other_dag",
+ dag=self.dag,
+ )
+ op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE, ignore_ti_state=True)
+ assert (f"Poking for DAG 'other_dag' on {DEFAULT_DATE.isoformat()} ... ") in caplog.messages
+
def test_external_dag_sensor_soft_fail_as_skipped(self):
other_dag = DAG("other_dag", default_args=self.args, end_date=DEFAULT_DATE, schedule="@once")
other_dag.create_dagrun(
[airflow] 32/37: fixing import error for dataset (#29007)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 147d5fe7193227228bc5a45b5a003f91c6b63f10
Author: bharat99k <12...@users.noreply.github.com>
AuthorDate: Sat Jan 21 01:28:15 2023 +0530
fixing import error for dataset (#29007)
(cherry picked from commit 481f27170673cb1e4fc8210341c8d341b11925e9)
---
airflow/__init__.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/airflow/__init__.py b/airflow/__init__.py
index 19624252a3..68437a2e5f 100644
--- a/airflow/__init__.py
+++ b/airflow/__init__.py
@@ -115,3 +115,4 @@ if STATICA_HACK: # pragma: no cover
from airflow.models.dag import DAG
from airflow.models.xcom_arg import XComArg
from airflow.exceptions import AirflowException
+ from airflow.models.dataset import Dataset
[airflow] 22/37: Refactor python operators/sensor tests (#28493)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 061338fad1a9ec4bf12b1aad482b3a72f7d3551c
Author: Andrey Anshin <An...@taragol.is>
AuthorDate: Thu Dec 22 12:32:06 2022 +0400
Refactor python operators/sensor tests (#28493)
(cherry picked from commit 884fca8d114ce8e0c982747937a1014f3b5e7491)
---
tests/conftest.py | 8 +-
tests/decorators/test_python.py | 143 +---
tests/decorators/test_python_virtualenv.py | 13 -
tests/operators/test_python.py | 1003 ++++++++++------------------
tests/sensors/test_python.py | 124 +---
5 files changed, 426 insertions(+), 865 deletions(-)
diff --git a/tests/conftest.py b/tests/conftest.py
index 0d4d1170f0..d71d8eb0f0 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -22,6 +22,7 @@ import subprocess
import sys
from contextlib import ExitStack, suppress
from datetime import datetime, timedelta
+from typing import TYPE_CHECKING
import freezegun
import pytest
@@ -46,6 +47,9 @@ from tests.test_utils.perf.perf_kit.sqlalchemy import ( # noqa isort:skip
trace_queries,
)
+if TYPE_CHECKING:
+ from airflow.models.taskinstance import TaskInstance
+
@pytest.fixture()
def reset_environment():
@@ -741,7 +745,7 @@ def create_task_instance(dag_maker, create_dummy_dag):
run_type=None,
data_interval=None,
**kwargs,
- ):
+ ) -> TaskInstance:
if execution_date is None:
from airflow.utils import timezone
@@ -775,7 +779,7 @@ def create_task_instance_of_operator(dag_maker):
execution_date=None,
session=None,
**operator_kwargs,
- ):
+ ) -> TaskInstance:
with dag_maker(dag_id=dag_id, session=session):
operator_class(**operator_kwargs)
if execution_date is None:
diff --git a/tests/decorators/test_python.py b/tests/decorators/test_python.py
index 47a908db77..1bbad51a0b 100644
--- a/tests/decorators/test_python.py
+++ b/tests/decorators/test_python.py
@@ -37,41 +37,20 @@ from airflow.utils import timezone
from airflow.utils.state import State
from airflow.utils.task_group import TaskGroup
from airflow.utils.types import DagRunType
-from tests.operators.test_python import Call, assert_calls_equal, build_recording_function
-from tests.test_utils.db import clear_db_runs
+from tests.operators.test_python import BasePythonTest
DEFAULT_DATE = timezone.datetime(2016, 1, 1)
-END_DATE = timezone.datetime(2016, 1, 2)
-INTERVAL = timedelta(hours=12)
-FROZEN_NOW = timezone.datetime(2016, 1, 2, 12, 1, 1)
-TI_CONTEXT_ENV_VARS = [
- "AIRFLOW_CTX_DAG_ID",
- "AIRFLOW_CTX_TASK_ID",
- "AIRFLOW_CTX_EXECUTION_DATE",
- "AIRFLOW_CTX_DAG_RUN_ID",
-]
-
-class TestAirflowTaskDecorator:
- def setup_class(self):
- clear_db_runs()
-
- def setup_method(self):
- self.dag = DAG("test_dag", default_args={"owner": "airflow", "start_date": DEFAULT_DATE})
- self.run = False
-
- def teardown_method(self):
- self.dag.clear()
- self.run = False
- clear_db_runs()
+class TestAirflowTaskDecorator(BasePythonTest):
+ default_date = DEFAULT_DATE
def test_python_operator_python_callable_is_callable(self):
"""Tests that @task will only instantiate if
the python_callable argument is callable."""
not_callable = {}
with pytest.raises(TypeError):
- task_decorator(not_callable, dag=self.dag)
+ task_decorator(not_callable)
@pytest.mark.parametrize(
"resolve",
@@ -155,13 +134,7 @@ class TestAirflowTaskDecorator:
with self.dag:
res = identity2(8, 4)
- dr = self.dag.create_dagrun(
- run_id=DagRunType.MANUAL.value,
- start_date=timezone.utcnow(),
- execution_date=DEFAULT_DATE,
- state=State.RUNNING,
- )
-
+ dr = self.create_dag_run()
res.operator.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
ti = dr.get_task_instances()[0]
@@ -179,13 +152,7 @@ class TestAirflowTaskDecorator:
with self.dag:
ident = identity_tuple(35, 36)
- dr = self.dag.create_dagrun(
- run_id=DagRunType.MANUAL.value,
- start_date=timezone.utcnow(),
- execution_date=DEFAULT_DATE,
- state=State.RUNNING,
- )
-
+ dr = self.create_dag_run()
ident.operator.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
ti = dr.get_task_instances()[0]
@@ -227,15 +194,9 @@ class TestAirflowTaskDecorator:
with self.dag:
ret = add_number(2)
- self.dag.create_dagrun(
- run_id=DagRunType.MANUAL,
- execution_date=DEFAULT_DATE,
- start_date=DEFAULT_DATE,
- state=State.RUNNING,
- )
+ self.create_dag_run()
with pytest.raises(AirflowException):
-
ret.operator.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
def test_fail_multiple_outputs_no_dict(self):
@@ -245,84 +206,53 @@ class TestAirflowTaskDecorator:
with self.dag:
ret = add_number(2)
- self.dag.create_dagrun(
- run_id=DagRunType.MANUAL,
- execution_date=DEFAULT_DATE,
- start_date=DEFAULT_DATE,
- state=State.RUNNING,
- )
+ self.create_dag_run()
with pytest.raises(AirflowException):
-
ret.operator.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
def test_python_callable_arguments_are_templatized(self):
"""Test @task op_args are templatized"""
- recorded_calls = []
+
+ @task_decorator
+ def arg_task(*args):
+ raise RuntimeError("Should not executed")
# Create a named tuple and ensure it is still preserved
# after the rendering is done
Named = namedtuple("Named", ["var1", "var2"])
named_tuple = Named("{{ ds }}", "unchanged")
- task = task_decorator(
- # a Mock instance cannot be used as a callable function or test fails with a
- # TypeError: Object of type Mock is not JSON serializable
- build_recording_function(recorded_calls),
- dag=self.dag,
- )
- ret = task(4, date(2019, 1, 1), "dag {{dag.dag_id}} ran on {{ds}}.", named_tuple)
-
- self.dag.create_dagrun(
- run_id=DagRunType.MANUAL,
- execution_date=DEFAULT_DATE,
- data_interval=(DEFAULT_DATE, DEFAULT_DATE),
- start_date=DEFAULT_DATE,
- state=State.RUNNING,
- )
- ret.operator.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
+ with self.dag:
+ ret = arg_task(4, date(2019, 1, 1), "dag {{dag.dag_id}} ran on {{ds}}.", named_tuple)
- ds_templated = DEFAULT_DATE.date().isoformat()
- assert len(recorded_calls) == 1
- assert_calls_equal(
- recorded_calls[0],
- Call(
- 4,
- date(2019, 1, 1),
- f"dag {self.dag.dag_id} ran on {ds_templated}.",
- Named(ds_templated, "unchanged"),
- ),
- )
+ dr = self.create_dag_run()
+ ti = TaskInstance(task=ret.operator, run_id=dr.run_id)
+ rendered_op_args = ti.render_templates().op_args
+ assert len(rendered_op_args) == 4
+ assert rendered_op_args[0] == 4
+ assert rendered_op_args[1] == date(2019, 1, 1)
+ assert rendered_op_args[2] == f"dag {self.dag_id} ran on {self.ds_templated}."
+ assert rendered_op_args[3] == Named(self.ds_templated, "unchanged")
def test_python_callable_keyword_arguments_are_templatized(self):
"""Test PythonOperator op_kwargs are templatized"""
- recorded_calls = []
- task = task_decorator(
- # a Mock instance cannot be used as a callable function or test fails with a
- # TypeError: Object of type Mock is not JSON serializable
- build_recording_function(recorded_calls),
- dag=self.dag,
- )
- ret = task(an_int=4, a_date=date(2019, 1, 1), a_templated_string="dag {{dag.dag_id}} ran on {{ds}}.")
- self.dag.create_dagrun(
- run_id=DagRunType.MANUAL,
- execution_date=DEFAULT_DATE,
- data_interval=(DEFAULT_DATE, DEFAULT_DATE),
- start_date=DEFAULT_DATE,
- state=State.RUNNING,
- )
- ret.operator.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
+ @task_decorator
+ def kwargs_task(an_int, a_date, a_templated_string):
+ raise RuntimeError("Should not executed")
- assert len(recorded_calls) == 1
- assert_calls_equal(
- recorded_calls[0],
- Call(
- an_int=4,
- a_date=date(2019, 1, 1),
- a_templated_string=f"dag {self.dag.dag_id} ran on {DEFAULT_DATE.date().isoformat()}.",
- ),
- )
+ with self.dag:
+ ret = kwargs_task(
+ an_int=4, a_date=date(2019, 1, 1), a_templated_string="dag {{dag.dag_id}} ran on {{ds}}."
+ )
+
+ dr = self.create_dag_run()
+ ti = TaskInstance(task=ret.operator, run_id=dr.run_id)
+ rendered_op_kwargs = ti.render_templates().op_kwargs
+ assert rendered_op_kwargs["an_int"] == 4
+ assert rendered_op_kwargs["a_date"] == date(2019, 1, 1)
+ assert rendered_op_kwargs["a_templated_string"] == f"dag {self.dag_id} ran on {self.ds_templated}."
def test_manual_task_id(self):
"""Test manually setting task_id"""
@@ -415,6 +345,7 @@ class TestAirflowTaskDecorator:
def do_run():
return 4
+ self.dag.default_args["owner"] = "airflow"
with self.dag:
ret = do_run()
assert ret.operator.owner == "airflow"
diff --git a/tests/decorators/test_python_virtualenv.py b/tests/decorators/test_python_virtualenv.py
index 032ec34aa5..88121c5db3 100644
--- a/tests/decorators/test_python_virtualenv.py
+++ b/tests/decorators/test_python_virtualenv.py
@@ -19,7 +19,6 @@ from __future__ import annotations
import datetime
import sys
-from datetime import timedelta
from subprocess import CalledProcessError
import pytest
@@ -28,18 +27,6 @@ from airflow.decorators import task
from airflow.utils import timezone
DEFAULT_DATE = timezone.datetime(2016, 1, 1)
-END_DATE = timezone.datetime(2016, 1, 2)
-INTERVAL = timedelta(hours=12)
-FROZEN_NOW = timezone.datetime(2016, 1, 2, 12, 1, 1)
-
-TI_CONTEXT_ENV_VARS = [
- "AIRFLOW_CTX_DAG_ID",
- "AIRFLOW_CTX_TASK_ID",
- "AIRFLOW_CTX_EXECUTION_DATE",
- "AIRFLOW_CTX_DAG_RUN_ID",
-]
-
-
PYTHON_VERSION = sys.version_info[0]
diff --git a/tests/operators/test_python.py b/tests/operators/test_python.py
index c011c5cb35..8f6c089f08 100644
--- a/tests/operators/test_python.py
+++ b/tests/operators/test_python.py
@@ -20,16 +20,18 @@ from __future__ import annotations
import copy
import logging
import os
+import re
import sys
-import unittest.mock
import warnings
from collections import namedtuple
from datetime import date, datetime, timedelta
from subprocess import CalledProcessError
+from unittest import mock
import pytest
+from slugify import slugify
-from airflow.exceptions import AirflowException
+from airflow.exceptions import AirflowException, RemovedInAirflow3Warning
from airflow.models import DAG, DagRun, TaskInstance as TI
from airflow.models.baseoperator import BaseOperator
from airflow.models.taskinstance import clear_task_instances, set_current_context
@@ -45,88 +47,108 @@ from airflow.utils import timezone
from airflow.utils.context import AirflowContextDeprecationWarning, Context
from airflow.utils.python_virtualenv import prepare_virtualenv
from airflow.utils.session import create_session
-from airflow.utils.state import State
+from airflow.utils.state import DagRunState, State
from airflow.utils.trigger_rule import TriggerRule
-from airflow.utils.types import DagRunType
+from airflow.utils.types import NOTSET, DagRunType
from tests.test_utils import AIRFLOW_MAIN_FOLDER
from tests.test_utils.db import clear_db_runs
DEFAULT_DATE = timezone.datetime(2016, 1, 1)
-END_DATE = timezone.datetime(2016, 1, 2)
-INTERVAL = timedelta(hours=12)
-FROZEN_NOW = timezone.datetime(2016, 1, 2, 12, 1, 1)
-
-TI_CONTEXT_ENV_VARS = [
- "AIRFLOW_CTX_DAG_ID",
- "AIRFLOW_CTX_TASK_ID",
- "AIRFLOW_CTX_EXECUTION_DATE",
- "AIRFLOW_CTX_DAG_RUN_ID",
-]
-
TEMPLATE_SEARCHPATH = os.path.join(AIRFLOW_MAIN_FOLDER, "tests", "config_templates")
+LOGGER_NAME = "airflow.task.operators"
-class Call:
- def __init__(self, *args, **kwargs):
- self.args = args
- self.kwargs = kwargs
-
-
-def build_recording_function(calls_collection):
- """
- We can not use a Mock instance as a PythonOperator callable function or some tests fail with a
- TypeError: Object of type Mock is not JSON serializable
- Then using this custom function recording custom Call objects for further testing
- (replacing Mock.assert_called_with assertion method)
- """
-
- def recording_function(*args, **kwargs):
- calls_collection.append(Call(*args, **kwargs))
-
- return recording_function
-
+class BasePythonTest:
+ """Base test class for TestPythonOperator and TestPythonSensor classes"""
-def assert_calls_equal(first: Call, second: Call) -> None:
- assert isinstance(first, Call)
- assert isinstance(second, Call)
- assert first.args == second.args
- # eliminate context (conf, dag_run, task_instance, etc.)
- test_args = ["an_int", "a_date", "a_templated_string"]
- first.kwargs = {key: value for (key, value) in first.kwargs.items() if key in test_args}
- second.kwargs = {key: value for (key, value) in second.kwargs.items() if key in test_args}
- assert first.kwargs == second.kwargs
+ opcls: type[BaseOperator]
+ dag_id: str
+ task_id: str
+ run_id: str
+ dag: DAG
+ ds_templated: str
+ default_date: datetime = DEFAULT_DATE
+
+ @pytest.fixture(autouse=True)
+ def base_tests_setup(self, request, create_task_instance_of_operator, dag_maker):
+ self.dag_id = f"dag_{slugify(request.cls.__name__)}"
+ self.task_id = f"task_{slugify(request.node.name, max_length=40)}"
+ self.run_id = f"run_{slugify(request.node.name, max_length=40)}"
+ self.ds_templated = self.default_date.date().isoformat()
+ self.ti_maker = create_task_instance_of_operator
+ self.dag_maker = dag_maker
+ self.dag = self.dag_maker(self.dag_id, template_searchpath=TEMPLATE_SEARCHPATH).dag
+ clear_db_runs()
+ yield
+ clear_db_runs()
+
+ @staticmethod
+ def assert_expected_task_states(dag_run: DagRun, expected_states: dict):
+ """Helper function that asserts `TaskInstances` of a given `task_id` are in a given state."""
+ asserts = []
+ for ti in dag_run.get_task_instances():
+ try:
+ expected = expected_states[ti.task_id]
+ except KeyError:
+ asserts.append(f"Unexpected task id {ti.task_id!r} found, expected {expected_states.keys()}")
+ continue
+
+ if ti.state != expected:
+ asserts.append(f"Task {ti.task_id!r} has state {ti.state!r} instead of expected {expected!r}")
+ if asserts:
+ pytest.fail("\n".join(asserts))
+
+ @staticmethod
+ def default_kwargs(**kwargs):
+ """Default arguments for specific Operator."""
+ return kwargs
+
+ def create_dag_run(self) -> DagRun:
+ return self.dag.create_dagrun(
+ state=DagRunState.RUNNING,
+ start_date=self.dag_maker.start_date,
+ session=self.dag_maker.session,
+ execution_date=self.default_date,
+ run_type=DagRunType.MANUAL,
+ )
+ def create_ti(self, fn, **kwargs) -> TI:
+ """Create TaskInstance for class defined Operator."""
+ return self.ti_maker(
+ self.opcls,
+ python_callable=fn,
+ **self.default_kwargs(**kwargs),
+ dag_id=self.dag_id,
+ task_id=self.task_id,
+ execution_date=self.default_date,
+ )
-class TestPythonBase(unittest.TestCase):
- """Base test class for TestPythonOperator and TestPythonSensor classes"""
+ def run_as_operator(self, fn, **kwargs):
+ """Run task by direct call ``run`` method."""
+ with self.dag:
+ task = self.opcls(task_id=self.task_id, python_callable=fn, **self.default_kwargs(**kwargs))
- @classmethod
- def setUpClass(cls):
- super().setUpClass()
+ task.run(start_date=self.default_date, end_date=self.default_date)
+ return task
- with create_session() as session:
- session.query(DagRun).delete()
- session.query(TI).delete()
+ def run_as_task(self, fn, **kwargs):
+ """Create TaskInstance and run it."""
+ ti = self.create_ti(fn, **kwargs)
+ ti.run()
+ return ti.task
- def setUp(self):
- super().setUp()
- self.dag = DAG("test_dag", default_args={"owner": "airflow", "start_date": DEFAULT_DATE})
- self.addCleanup(self.dag.clear)
- self.clear_run()
- self.addCleanup(self.clear_run)
+ def render_templates(self, fn, **kwargs):
+ """Create TaskInstance and render templates without actual run."""
+ return self.create_ti(fn, **kwargs).render_templates()
- def tearDown(self):
- super().tearDown()
- with create_session() as session:
- session.query(DagRun).delete()
- session.query(TI).delete()
+class TestPythonOperator(BasePythonTest):
+ opcls = PythonOperator
- def clear_run(self):
+ @pytest.fixture(autouse=True)
+ def setup_tests(self):
self.run = False
-
-class TestPythonOperator(TestPythonBase):
def do_run(self):
self.run = True
@@ -135,105 +157,58 @@ class TestPythonOperator(TestPythonBase):
def test_python_operator_run(self):
"""Tests that the python callable is invoked on task run."""
- task = PythonOperator(python_callable=self.do_run, task_id="python_operator", dag=self.dag)
+ ti = self.create_ti(self.do_run)
assert not self.is_run()
- task.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
+ ti.run()
assert self.is_run()
- def test_python_operator_python_callable_is_callable(self):
- """Tests that PythonOperator will only instantiate if
- the python_callable argument is callable."""
- not_callable = {}
- with pytest.raises(AirflowException):
- PythonOperator(python_callable=not_callable, task_id="python_operator", dag=self.dag)
- not_callable = None
- with pytest.raises(AirflowException):
- PythonOperator(python_callable=not_callable, task_id="python_operator", dag=self.dag)
+ @pytest.mark.parametrize("not_callable", [{}, None])
+ def test_python_operator_python_callable_is_callable(self, not_callable):
+ """Tests that PythonOperator will only instantiate if the python_callable argument is callable."""
+ with pytest.raises(AirflowException, match="`python_callable` param must be callable"):
+ PythonOperator(python_callable=not_callable, task_id="python_operator")
def test_python_callable_arguments_are_templatized(self):
"""Test PythonOperator op_args are templatized"""
- recorded_calls = []
-
# Create a named tuple and ensure it is still preserved
# after the rendering is done
Named = namedtuple("Named", ["var1", "var2"])
named_tuple = Named("{{ ds }}", "unchanged")
- task = PythonOperator(
- task_id="python_operator",
- # a Mock instance cannot be used as a callable function or test fails with a
- # TypeError: Object of type Mock is not JSON serializable
- python_callable=build_recording_function(recorded_calls),
+ task = self.render_templates(
+ lambda: 0,
op_args=[4, date(2019, 1, 1), "dag {{dag.dag_id}} ran on {{ds}}.", named_tuple],
- dag=self.dag,
- )
-
- self.dag.create_dagrun(
- run_type=DagRunType.MANUAL,
- execution_date=DEFAULT_DATE,
- data_interval=(DEFAULT_DATE, DEFAULT_DATE),
- start_date=DEFAULT_DATE,
- state=State.RUNNING,
- )
- task.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
-
- ds_templated = DEFAULT_DATE.date().isoformat()
- assert 1 == len(recorded_calls)
- assert_calls_equal(
- recorded_calls[0],
- Call(
- 4,
- date(2019, 1, 1),
- f"dag {self.dag.dag_id} ran on {ds_templated}.",
- Named(ds_templated, "unchanged"),
- ),
)
+ rendered_op_args = task.op_args
+ assert len(rendered_op_args) == 4
+ assert rendered_op_args[0] == 4
+ assert rendered_op_args[1] == date(2019, 1, 1)
+ assert rendered_op_args[2] == f"dag {self.dag_id} ran on {self.ds_templated}."
+ assert rendered_op_args[3] == Named(self.ds_templated, "unchanged")
def test_python_callable_keyword_arguments_are_templatized(self):
"""Test PythonOperator op_kwargs are templatized"""
- recorded_calls = []
-
- task = PythonOperator(
- task_id="python_operator",
- # a Mock instance cannot be used as a callable function or test fails with a
- # TypeError: Object of type Mock is not JSON serializable
- python_callable=build_recording_function(recorded_calls),
+ task = self.render_templates(
+ lambda: 0,
op_kwargs={
"an_int": 4,
"a_date": date(2019, 1, 1),
"a_templated_string": "dag {{dag.dag_id}} ran on {{ds}}.",
},
- dag=self.dag,
- )
-
- self.dag.create_dagrun(
- run_type=DagRunType.MANUAL,
- execution_date=DEFAULT_DATE,
- data_interval=(DEFAULT_DATE, DEFAULT_DATE),
- start_date=DEFAULT_DATE,
- state=State.RUNNING,
- )
- task.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
-
- assert 1 == len(recorded_calls)
- assert_calls_equal(
- recorded_calls[0],
- Call(
- an_int=4,
- a_date=date(2019, 1, 1),
- a_templated_string=f"dag {self.dag.dag_id} ran on {DEFAULT_DATE.date().isoformat()}.",
- ),
)
+ rendered_op_kwargs = task.op_kwargs
+ assert rendered_op_kwargs["an_int"] == 4
+ assert rendered_op_kwargs["a_date"] == date(2019, 1, 1)
+ assert rendered_op_kwargs["a_templated_string"] == f"dag {self.dag_id} ran on {self.ds_templated}."
def test_python_operator_shallow_copy_attr(self):
def not_callable(x):
- return x
+ assert False, "Should not be triggered"
original_task = PythonOperator(
python_callable=not_callable,
- task_id="python_operator",
op_kwargs={"certain_attrs": ""},
- dag=self.dag,
+ task_id=self.task_id,
)
new_task = copy.deepcopy(original_task)
# shallow copy op_kwargs
@@ -242,383 +217,213 @@ class TestPythonOperator(TestPythonBase):
assert id(original_task.python_callable) == id(new_task.python_callable)
def test_conflicting_kwargs(self):
- self.dag.create_dagrun(
- run_type=DagRunType.MANUAL,
- execution_date=DEFAULT_DATE,
- start_date=DEFAULT_DATE,
- state=State.RUNNING,
- external_trigger=False,
- )
-
# dag is not allowed since it is a reserved keyword
def func(dag):
- # An ValueError should be triggered since we're using dag as a
- # reserved keyword
+ # An ValueError should be triggered since we're using dag as a reserved keyword
raise RuntimeError(f"Should not be triggered, dag: {dag}")
- python_operator = PythonOperator(
- task_id="python_operator", op_args=[1], python_callable=func, dag=self.dag
- )
-
- with pytest.raises(ValueError) as ctx:
- python_operator.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
- assert "dag" in str(ctx.value), "'dag' not found in the exception"
+ ti = self.create_ti(func, op_args=[1])
+ error_message = re.escape("The key 'dag' in args is a part of kwargs and therefore reserved.")
+ with pytest.raises(ValueError, match=error_message):
+ ti.run()
def test_provide_context_does_not_fail(self):
- """
- ensures that provide_context doesn't break dags in 2.0
- """
- self.dag.create_dagrun(
- run_type=DagRunType.MANUAL,
- execution_date=DEFAULT_DATE,
- start_date=DEFAULT_DATE,
- state=State.RUNNING,
- external_trigger=False,
- )
+ """Ensures that provide_context doesn't break dags in 2.0."""
def func(custom, dag):
assert 1 == custom, "custom should be 1"
assert dag is not None, "dag should be set"
- python_operator = PythonOperator(
- task_id="python_operator",
- op_kwargs={"custom": 1},
- python_callable=func,
- provide_context=True,
- dag=self.dag,
- )
- python_operator.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
+ with pytest.warns(RemovedInAirflow3Warning):
+ self.run_as_task(func, op_kwargs={"custom": 1}, provide_context=True)
def test_context_with_conflicting_op_args(self):
- self.dag.create_dagrun(
- run_type=DagRunType.MANUAL,
- execution_date=DEFAULT_DATE,
- start_date=DEFAULT_DATE,
- state=State.RUNNING,
- external_trigger=False,
- )
-
def func(custom, dag):
assert 1 == custom, "custom should be 1"
assert dag is not None, "dag should be set"
- python_operator = PythonOperator(
- task_id="python_operator", op_kwargs={"custom": 1}, python_callable=func, dag=self.dag
- )
- python_operator.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
+ self.run_as_task(func, op_kwargs={"custom": 1})
def test_context_with_kwargs(self):
- self.dag.create_dagrun(
- run_type=DagRunType.MANUAL,
- execution_date=DEFAULT_DATE,
- start_date=DEFAULT_DATE,
- state=State.RUNNING,
- external_trigger=False,
- )
-
def func(**context):
# check if context is being set
assert len(context) > 0, "Context has not been injected"
- python_operator = PythonOperator(
- task_id="python_operator", op_kwargs={"custom": 1}, python_callable=func, dag=self.dag
- )
- python_operator.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
-
- def test_return_value_log_with_show_return_value_in_logs_default(self):
- self.dag.create_dagrun(
- run_type=DagRunType.MANUAL,
- execution_date=DEFAULT_DATE,
- start_date=DEFAULT_DATE,
- state=State.RUNNING,
- external_trigger=False,
- )
-
- def func():
- return "test_return_value"
-
- python_operator = PythonOperator(task_id="python_operator", python_callable=func, dag=self.dag)
-
- with self.assertLogs("airflow.task.operators", level=logging.INFO) as cm:
- python_operator.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
+ self.run_as_task(func, op_kwargs={"custom": 1})
- assert (
- "INFO:airflow.task.operators:Done. Returned value was: test_return_value" in cm.output
- ), "Return value should be shown"
-
- def test_return_value_log_with_show_return_value_in_logs_false(self):
- self.dag.create_dagrun(
- run_type=DagRunType.MANUAL,
- execution_date=DEFAULT_DATE,
- start_date=DEFAULT_DATE,
- state=State.RUNNING,
- external_trigger=False,
- )
+ @pytest.mark.parametrize(
+ "show_return_value_in_logs, should_shown",
+ [
+ pytest.param(NOTSET, True, id="default"),
+ pytest.param(True, True, id="show"),
+ pytest.param(False, False, id="hide"),
+ ],
+ )
+ def test_return_value_log(self, show_return_value_in_logs, should_shown, caplog):
+ caplog.set_level(logging.INFO, logger=LOGGER_NAME)
def func():
return "test_return_value"
- python_operator = PythonOperator(
- task_id="python_operator",
- python_callable=func,
- dag=self.dag,
- show_return_value_in_logs=False,
- )
-
- with self.assertLogs("airflow.task.operators", level=logging.INFO) as cm:
- python_operator.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
-
- assert (
- "INFO:airflow.task.operators:Done. Returned value was: test_return_value" not in cm.output
- ), "Return value should not be shown"
- assert (
- "INFO:airflow.task.operators:Done. Returned value not shown" in cm.output
- ), "Log message that the option is turned off should be shown"
+ if show_return_value_in_logs is NOTSET:
+ self.run_as_task(func)
+ else:
+ self.run_as_task(func, show_return_value_in_logs=show_return_value_in_logs)
+ if should_shown:
+ assert "Done. Returned value was: test_return_value" in caplog.messages
+ assert "Done. Returned value not shown" not in caplog.messages
+ else:
+ assert "Done. Returned value was: test_return_value" not in caplog.messages
+ assert "Done. Returned value not shown" in caplog.messages
-class TestBranchOperator(unittest.TestCase):
- @classmethod
- def setUpClass(cls):
- super().setUpClass()
-
- with create_session() as session:
- session.query(DagRun).delete()
- session.query(TI).delete()
-
- def setUp(self):
- self.dag = DAG(
- "branch_operator_test",
- default_args={"owner": "airflow", "start_date": DEFAULT_DATE},
- schedule=INTERVAL,
- )
- self.branch_1 = EmptyOperator(task_id="branch_1", dag=self.dag)
- self.branch_2 = EmptyOperator(task_id="branch_2", dag=self.dag)
- self.branch_3 = None
+class TestBranchOperator(BasePythonTest):
+ opcls = BranchPythonOperator
- def tearDown(self):
- super().tearDown()
-
- with create_session() as session:
- session.query(DagRun).delete()
- session.query(TI).delete()
+ @pytest.fixture(autouse=True)
+ def setup_tests(self):
+ self.branch_1 = EmptyOperator(task_id="branch_1")
+ self.branch_2 = EmptyOperator(task_id="branch_2")
def test_with_dag_run(self):
- branch_op = BranchPythonOperator(
- task_id="make_choice", dag=self.dag, python_callable=lambda: "branch_1"
- )
-
- self.branch_1.set_upstream(branch_op)
- self.branch_2.set_upstream(branch_op)
- self.dag.clear()
+ with self.dag:
+ branch_op = BranchPythonOperator(task_id=self.task_id, python_callable=lambda: "branch_1")
+ branch_op >> [self.branch_1, self.branch_2]
- dr = self.dag.create_dagrun(
- run_type=DagRunType.MANUAL,
- start_date=timezone.utcnow(),
- execution_date=DEFAULT_DATE,
- state=State.RUNNING,
+ dr = self.create_dag_run()
+ branch_op.run(start_date=self.default_date, end_date=self.default_date)
+ self.assert_expected_task_states(
+ dr, {self.task_id: State.SUCCESS, "branch_1": State.NONE, "branch_2": State.SKIPPED}
)
- branch_op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
-
- tis = dr.get_task_instances()
- for ti in tis:
- if ti.task_id == "make_choice":
- assert ti.state == State.SUCCESS
- elif ti.task_id == "branch_1":
- assert ti.state == State.NONE
- elif ti.task_id == "branch_2":
- assert ti.state == State.SKIPPED
- else:
- raise ValueError(f"Invalid task id {ti.task_id} found!")
-
def test_with_skip_in_branch_downstream_dependencies(self):
- branch_op = BranchPythonOperator(
- task_id="make_choice", dag=self.dag, python_callable=lambda: "branch_1"
- )
-
- branch_op >> self.branch_1 >> self.branch_2
- branch_op >> self.branch_2
- self.dag.clear()
+ with self.dag:
+ branch_op = BranchPythonOperator(task_id=self.task_id, python_callable=lambda: "branch_1")
+ branch_op >> self.branch_1 >> self.branch_2
+ branch_op >> self.branch_2
- dr = self.dag.create_dagrun(
- run_type=DagRunType.MANUAL,
- start_date=timezone.utcnow(),
- execution_date=DEFAULT_DATE,
- state=State.RUNNING,
+ dr = self.create_dag_run()
+ branch_op.run(start_date=self.default_date, end_date=self.default_date)
+ self.assert_expected_task_states(
+ dr, {self.task_id: State.SUCCESS, "branch_1": State.NONE, "branch_2": State.NONE}
)
- branch_op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
-
- tis = dr.get_task_instances()
- for ti in tis:
- if ti.task_id == "make_choice":
- assert ti.state == State.SUCCESS
- elif ti.task_id == "branch_1":
- assert ti.state == State.NONE
- elif ti.task_id == "branch_2":
- assert ti.state == State.NONE
- else:
- raise ValueError(f"Invalid task id {ti.task_id} found!")
-
def test_with_skip_in_branch_downstream_dependencies2(self):
- branch_op = BranchPythonOperator(
- task_id="make_choice", dag=self.dag, python_callable=lambda: "branch_2"
- )
-
- branch_op >> self.branch_1 >> self.branch_2
- branch_op >> self.branch_2
- self.dag.clear()
+ with self.dag:
+ branch_op = BranchPythonOperator(task_id=self.task_id, python_callable=lambda: "branch_2")
+ branch_op >> self.branch_1 >> self.branch_2
+ branch_op >> self.branch_2
- dr = self.dag.create_dagrun(
- run_type=DagRunType.MANUAL,
- start_date=timezone.utcnow(),
- execution_date=DEFAULT_DATE,
- state=State.RUNNING,
+ dr = self.create_dag_run()
+ branch_op.run(start_date=self.default_date, end_date=self.default_date)
+ self.assert_expected_task_states(
+ dr, {self.task_id: State.SUCCESS, "branch_1": State.SKIPPED, "branch_2": State.NONE}
)
- branch_op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
-
- tis = dr.get_task_instances()
- for ti in tis:
- if ti.task_id == "make_choice":
- assert ti.state == State.SUCCESS
- elif ti.task_id == "branch_1":
- assert ti.state == State.SKIPPED
- elif ti.task_id == "branch_2":
- assert ti.state == State.NONE
- else:
- raise ValueError(f"Invalid task id {ti.task_id} found!")
-
def test_xcom_push(self):
- branch_op = BranchPythonOperator(
- task_id="make_choice", dag=self.dag, python_callable=lambda: "branch_1"
- )
-
- self.branch_1.set_upstream(branch_op)
- self.branch_2.set_upstream(branch_op)
- self.dag.clear()
-
- dr = self.dag.create_dagrun(
- run_type=DagRunType.MANUAL,
- start_date=timezone.utcnow(),
- execution_date=DEFAULT_DATE,
- state=State.RUNNING,
- )
-
- branch_op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
+ with self.dag:
+ branch_op = BranchPythonOperator(task_id=self.task_id, python_callable=lambda: "branch_1")
+ branch_op >> [self.branch_1, self.branch_2]
- tis = dr.get_task_instances()
- for ti in tis:
- if ti.task_id == "make_choice":
- assert ti.xcom_pull(task_ids="make_choice") == "branch_1"
+ dr = self.create_dag_run()
+ branch_op.run(start_date=self.default_date, end_date=self.default_date)
+ for ti in dr.get_task_instances():
+ if ti.task_id == self.task_id:
+ assert ti.xcom_pull(task_ids=self.task_id) == "branch_1"
+ break
+ else:
+ pytest.fail(f"{self.task_id!r} not found.")
def test_clear_skipped_downstream_task(self):
"""
After a downstream task is skipped by BranchPythonOperator, clearing the skipped task
should not cause it to be executed.
"""
- branch_op = BranchPythonOperator(
- task_id="make_choice", dag=self.dag, python_callable=lambda: "branch_1"
- )
- branches = [self.branch_1, self.branch_2]
- branch_op >> branches
- self.dag.clear()
-
- dr = self.dag.create_dagrun(
- run_type=DagRunType.MANUAL,
- start_date=timezone.utcnow(),
- execution_date=DEFAULT_DATE,
- state=State.RUNNING,
- )
-
- branch_op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
+ with self.dag:
+ branch_op = BranchPythonOperator(task_id=self.task_id, python_callable=lambda: "branch_1")
+ branches = [self.branch_1, self.branch_2]
+ branch_op >> branches
+ dr = self.create_dag_run()
+ branch_op.run(start_date=self.default_date, end_date=self.default_date)
for task in branches:
- task.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
+ task.run(start_date=self.default_date, end_date=self.default_date)
- tis = dr.get_task_instances()
- for ti in tis:
- if ti.task_id == "make_choice":
- assert ti.state == State.SUCCESS
- elif ti.task_id == "branch_1":
- assert ti.state == State.SUCCESS
- elif ti.task_id == "branch_2":
- assert ti.state == State.SKIPPED
- else:
- raise ValueError(f"Invalid task id {ti.task_id} found!")
+ expected_states = {
+ self.task_id: State.SUCCESS,
+ "branch_1": State.SUCCESS,
+ "branch_2": State.SKIPPED,
+ }
- children_tis = [ti for ti in tis if ti.task_id in branch_op.get_direct_relative_ids()]
+ self.assert_expected_task_states(dr, expected_states)
# Clear the children tasks.
+ tis = dr.get_task_instances()
+ children_tis = [ti for ti in tis if ti.task_id in branch_op.get_direct_relative_ids()]
with create_session() as session:
- clear_task_instances(children_tis, session=session, dag=self.dag)
+ clear_task_instances(children_tis, session=session, dag=branch_op.dag)
# Run the cleared tasks again.
for task in branches:
- task.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
+ task.run(start_date=self.default_date, end_date=self.default_date)
# Check if the states are correct after children tasks are cleared.
- for ti in dr.get_task_instances():
- if ti.task_id == "make_choice":
- assert ti.state == State.SUCCESS
- elif ti.task_id == "branch_1":
- assert ti.state == State.SUCCESS
- elif ti.task_id == "branch_2":
- assert ti.state == State.SKIPPED
- else:
- raise ValueError(f"Invalid task id {ti.task_id} found!")
+ self.assert_expected_task_states(dr, expected_states)
def test_raise_exception_on_no_accepted_type_return(self):
- branch_op = BranchPythonOperator(task_id="make_choice", dag=self.dag, python_callable=lambda: 5)
- self.dag.clear()
- with pytest.raises(AirflowException) as ctx:
- branch_op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
- assert "must be either None, a task ID, or an Iterable of IDs" in str(ctx.value)
+ ti = self.create_ti(lambda: 5)
+ with pytest.raises(AirflowException, match="must be either None, a task ID, or an Iterable of IDs"):
+ ti.run()
def test_raise_exception_on_invalid_task_id(self):
- branch_op = BranchPythonOperator(
- task_id="make_choice", dag=self.dag, python_callable=lambda: "some_task_id"
- )
- self.dag.clear()
- with pytest.raises(AirflowException) as ctx:
- branch_op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
- assert "Invalid tasks found: {'some_task_id'}" in str(ctx.value)
+ ti = self.create_ti(lambda: "some_task_id")
+ with pytest.raises(AirflowException, match="Invalid tasks found: {'some_task_id'}"):
+ ti.run()
+ @pytest.mark.parametrize(
+ "choice,expected_states",
+ [
+ ("task1", [State.SUCCESS, State.SUCCESS, State.SUCCESS]),
+ ("join", [State.SUCCESS, State.SKIPPED, State.SUCCESS]),
+ ],
+ )
+ def test_empty_branch(self, choice, expected_states):
+ """
+ Tests that BranchPythonOperator handles empty branches properly.
+ """
+ with self.dag:
+ branch = BranchPythonOperator(task_id=self.task_id, python_callable=lambda: choice)
+ task1 = EmptyOperator(task_id="task1")
+ join = EmptyOperator(task_id="join", trigger_rule="none_failed_min_one_success")
-class TestShortCircuitOperator:
- def setup(self):
- with create_session() as session:
- session.query(DagRun).delete()
- session.query(TI).delete()
+ branch >> [task1, join]
+ task1 >> join
- self.dag = DAG(
- "short_circuit_op_test",
- start_date=DEFAULT_DATE,
- schedule=INTERVAL,
- )
+ dr = self.create_dag_run()
+ task_ids = [self.task_id, "task1", "join"]
+ tis = {ti.task_id: ti for ti in dr.task_instances}
- with self.dag:
- self.op1 = EmptyOperator(task_id="op1")
- self.op2 = EmptyOperator(task_id="op2")
- self.op1.set_downstream(self.op2)
+ for task_id in task_ids: # Mimic the specific order the scheduling would run the tests.
+ task_instance = tis[task_id]
+ task_instance.refresh_from_task(self.dag.get_task(task_id))
+ task_instance.run()
- def teardown(self):
- with create_session() as session:
- session.query(DagRun).delete()
- session.query(TI).delete()
+ def get_state(ti):
+ ti.refresh_from_db()
+ return ti.state
- def _assert_expected_task_states(self, dagrun, expected_states):
- """Helper function that asserts `TaskInstances` of a given `task_id` are in a given state."""
+ assert [get_state(tis[task_id]) for task_id in task_ids] == expected_states
- tis = dagrun.get_task_instances()
- for ti in tis:
- try:
- expected_state = expected_states[ti.task_id]
- except KeyError:
- raise ValueError(f"Invalid task id {ti.task_id} found!")
- else:
- assert ti.state == expected_state
+
+class TestShortCircuitOperator(BasePythonTest):
+ opcls = ShortCircuitOperator
+
+ @pytest.fixture(autouse=True)
+ def setup_tests(self):
+ self.task_id = "short_circuit"
+ self.op1 = EmptyOperator(task_id="op1")
+ self.op2 = EmptyOperator(task_id="op2")
all_downstream_skipped_states = {
"short_circuit": State.SUCCESS,
@@ -725,62 +530,41 @@ class TestShortCircuitOperator:
Checking the behavior of the ShortCircuitOperator in several scenarios enabling/disabling the skipping
of downstream tasks, both short-circuiting modes, and various trigger rules of downstream tasks.
"""
-
- self.short_circuit = ShortCircuitOperator(
- task_id="short_circuit",
- python_callable=lambda: callable_return,
- ignore_downstream_trigger_rules=test_ignore_downstream_trigger_rules,
- dag=self.dag,
- )
- self.short_circuit.set_downstream(self.op1)
- self.op2.trigger_rule = test_trigger_rule
- self.dag.clear()
-
- dagrun = self.dag.create_dagrun(
- run_type=DagRunType.MANUAL,
- start_date=timezone.utcnow(),
- execution_date=DEFAULT_DATE,
- state=State.RUNNING,
- )
-
- self.short_circuit.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
- self.op1.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
- self.op2.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
-
- assert self.short_circuit.ignore_downstream_trigger_rules == test_ignore_downstream_trigger_rules
- assert self.short_circuit.trigger_rule == TriggerRule.ALL_SUCCESS
+ with self.dag:
+ short_circuit = ShortCircuitOperator(
+ task_id="short_circuit",
+ python_callable=lambda: callable_return,
+ ignore_downstream_trigger_rules=test_ignore_downstream_trigger_rules,
+ )
+ short_circuit >> self.op1 >> self.op2
+ self.op2.trigger_rule = test_trigger_rule
+
+ dr = self.create_dag_run()
+ short_circuit.run(start_date=self.default_date, end_date=self.default_date)
+ self.op1.run(start_date=self.default_date, end_date=self.default_date)
+ self.op2.run(start_date=self.default_date, end_date=self.default_date)
+
+ assert short_circuit.ignore_downstream_trigger_rules == test_ignore_downstream_trigger_rules
+ assert short_circuit.trigger_rule == TriggerRule.ALL_SUCCESS
assert self.op1.trigger_rule == TriggerRule.ALL_SUCCESS
assert self.op2.trigger_rule == test_trigger_rule
-
- self._assert_expected_task_states(dagrun, expected_task_states)
+ self.assert_expected_task_states(dr, expected_task_states)
def test_clear_skipped_downstream_task(self):
"""
After a downstream task is skipped by ShortCircuitOperator, clearing the skipped task
should not cause it to be executed.
"""
-
- self.short_circuit = ShortCircuitOperator(
- task_id="short_circuit",
- python_callable=lambda: False,
- dag=self.dag,
- )
- self.short_circuit.set_downstream(self.op1)
- self.dag.clear()
-
- dagrun = self.dag.create_dagrun(
- run_type=DagRunType.MANUAL,
- start_date=timezone.utcnow(),
- execution_date=DEFAULT_DATE,
- state=State.RUNNING,
- )
-
- self.short_circuit.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
- self.op1.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
- self.op2.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
-
- assert self.short_circuit.ignore_downstream_trigger_rules
- assert self.short_circuit.trigger_rule == TriggerRule.ALL_SUCCESS
+ with self.dag:
+ short_circuit = ShortCircuitOperator(task_id="short_circuit", python_callable=lambda: False)
+ short_circuit >> self.op1 >> self.op2
+ dr = self.create_dag_run()
+
+ short_circuit.run(start_date=self.default_date, end_date=self.default_date)
+ self.op1.run(start_date=self.default_date, end_date=self.default_date)
+ self.op2.run(start_date=self.default_date, end_date=self.default_date)
+ assert short_circuit.ignore_downstream_trigger_rules
+ assert short_circuit.trigger_rule == TriggerRule.ALL_SUCCESS
assert self.op1.trigger_rule == TriggerRule.ALL_SUCCESS
assert self.op2.trigger_rule == TriggerRule.ALL_SUCCESS
@@ -789,82 +573,45 @@ class TestShortCircuitOperator:
"op1": State.SKIPPED,
"op2": State.SKIPPED,
}
- self._assert_expected_task_states(dagrun, expected_states)
+ self.assert_expected_task_states(dr, expected_states)
# Clear downstream task "op1" that was previously executed.
- tis = dagrun.get_task_instances()
-
+ tis = dr.get_task_instances()
with create_session() as session:
- clear_task_instances([ti for ti in tis if ti.task_id == "op1"], session=session, dag=self.dag)
-
- self.op1.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
- self._assert_expected_task_states(dagrun, expected_states)
+ clear_task_instances(
+ [ti for ti in tis if ti.task_id == "op1"], session=session, dag=short_circuit.dag
+ )
+ self.op1.run(start_date=self.default_date, end_date=self.default_date)
+ self.assert_expected_task_states(dr, expected_states)
def test_xcom_push(self):
- short_op_push_xcom = ShortCircuitOperator(
- task_id="push_xcom_from_shortcircuit", dag=self.dag, python_callable=lambda: "signature"
- )
-
- short_op_no_push_xcom = ShortCircuitOperator(
- task_id="do_not_push_xcom_from_shortcircuit", dag=self.dag, python_callable=lambda: False
- )
-
- self.dag.clear()
- dr = self.dag.create_dagrun(
- run_type=DagRunType.MANUAL,
- start_date=timezone.utcnow(),
- execution_date=DEFAULT_DATE,
- state=State.RUNNING,
- )
+ with self.dag:
+ short_op_push_xcom = ShortCircuitOperator(
+ task_id="push_xcom_from_shortcircuit", python_callable=lambda: "signature"
+ )
+ short_op_no_push_xcom = ShortCircuitOperator(
+ task_id="do_not_push_xcom_from_shortcircuit", python_callable=lambda: False
+ )
- short_op_push_xcom.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
- short_op_no_push_xcom.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
+ dr = self.create_dag_run()
+ short_op_push_xcom.run(start_date=self.default_date, end_date=self.default_date)
+ short_op_no_push_xcom.run(start_date=self.default_date, end_date=self.default_date)
tis = dr.get_task_instances()
- xcom_value_short_op_push_xcom = tis[0].xcom_pull(
- task_ids="push_xcom_from_shortcircuit", key="return_value"
- )
- assert xcom_value_short_op_push_xcom == "signature"
-
- xcom_value_short_op_no_push_xcom = tis[0].xcom_pull(
- task_ids="do_not_push_xcom_from_shortcircuit", key="return_value"
- )
- assert xcom_value_short_op_no_push_xcom is None
+ assert tis[0].xcom_pull(task_ids=short_op_push_xcom.task_id, key="return_value") == "signature"
+ assert tis[0].xcom_pull(task_ids=short_op_no_push_xcom.task_id, key="return_value") is None
virtualenv_string_args: list[str] = []
-class TestPythonVirtualenvOperator(unittest.TestCase):
- def setUp(self):
- super().setUp()
- self.dag = DAG(
- "test_dag",
- default_args={"owner": "airflow", "start_date": DEFAULT_DATE},
- template_searchpath=TEMPLATE_SEARCHPATH,
- schedule=INTERVAL,
- )
- self.dag.create_dagrun(
- run_type=DagRunType.MANUAL,
- start_date=timezone.utcnow(),
- execution_date=DEFAULT_DATE,
- state=State.RUNNING,
- )
- self.addCleanup(self.dag.clear)
-
- def tearDown(self):
- super().tearDown()
- with create_session() as session:
- session.query(DagRun).delete()
- session.query(TI).delete()
-
- def _run_as_operator(self, fn, python_version=sys.version_info[0], **kwargs):
+class TestPythonVirtualenvOperator(BasePythonTest):
+ opcls = PythonVirtualenvOperator
- task = PythonVirtualenvOperator(
- python_callable=fn, python_version=python_version, task_id="task", dag=self.dag, **kwargs
- )
- task.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
- return task
+ @staticmethod
+ def default_kwargs(*, python_version=sys.version_info[0], **kwargs):
+ kwargs["python_version"] = python_version
+ return kwargs
def test_template_fields(self):
assert set(PythonOperator.template_fields).issubset(PythonVirtualenvOperator.template_fields)
@@ -874,7 +621,7 @@ class TestPythonVirtualenvOperator(unittest.TestCase):
"""Ensure dill is correctly installed."""
import dill # noqa: F401
- self._run_as_operator(f, use_dill=True, system_site_packages=False)
+ self.run_as_task(f, use_dill=True, system_site_packages=False)
def test_no_requirements(self):
"""Tests that the python callable is invoked on task run."""
@@ -882,7 +629,7 @@ class TestPythonVirtualenvOperator(unittest.TestCase):
def f():
pass
- self._run_as_operator(f)
+ self.run_as_task(f)
def test_no_system_site_packages(self):
def f():
@@ -892,13 +639,13 @@ class TestPythonVirtualenvOperator(unittest.TestCase):
return True
raise Exception
- self._run_as_operator(f, system_site_packages=False, requirements=["dill"])
+ self.run_as_task(f, system_site_packages=False, requirements=["dill"])
def test_system_site_packages(self):
def f():
import funcsigs # noqa: F401
- self._run_as_operator(f, requirements=["funcsigs"], system_site_packages=True)
+ self.run_as_task(f, requirements=["funcsigs"], system_site_packages=True)
def test_with_requirements_pinned(self):
def f():
@@ -907,44 +654,44 @@ class TestPythonVirtualenvOperator(unittest.TestCase):
if funcsigs.__version__ != "0.4":
raise Exception
- self._run_as_operator(f, requirements=["funcsigs==0.4"])
+ self.run_as_task(f, requirements=["funcsigs==0.4"])
def test_unpinned_requirements(self):
def f():
import funcsigs # noqa: F401
- self._run_as_operator(f, requirements=["funcsigs", "dill"], system_site_packages=False)
+ self.run_as_task(f, requirements=["funcsigs", "dill"], system_site_packages=False)
def test_range_requirements(self):
def f():
import funcsigs # noqa: F401
- self._run_as_operator(f, requirements=["funcsigs>1.0", "dill"], system_site_packages=False)
+ self.run_as_task(f, requirements=["funcsigs>1.0", "dill"], system_site_packages=False)
def test_requirements_file(self):
def f():
import funcsigs # noqa: F401
- self._run_as_operator(f, requirements="requirements.txt", system_site_packages=False)
+ self.run_as_operator(f, requirements="requirements.txt", system_site_packages=False)
- @unittest.mock.patch("airflow.operators.python.prepare_virtualenv")
+ @mock.patch("airflow.operators.python.prepare_virtualenv")
def test_pip_install_options(self, mocked_prepare_virtualenv):
def f():
import funcsigs # noqa: F401
mocked_prepare_virtualenv.side_effect = prepare_virtualenv
- self._run_as_operator(
+ self.run_as_task(
f,
requirements=["funcsigs==0.4"],
system_site_packages=False,
pip_install_options=["--no-deps"],
)
mocked_prepare_virtualenv.assert_called_with(
- venv_directory=unittest.mock.ANY,
- python_bin=unittest.mock.ANY,
+ venv_directory=mock.ANY,
+ python_bin=mock.ANY,
system_site_packages=False,
- requirements_file_path=unittest.mock.ANY,
+ requirements_file_path=mock.ANY,
pip_install_options=["--no-deps"],
)
@@ -954,7 +701,7 @@ class TestPythonVirtualenvOperator(unittest.TestCase):
assert funcsigs.__version__ == "1.0.2"
- self._run_as_operator(
+ self.run_as_operator(
f,
requirements="requirements.txt",
use_dill=True,
@@ -967,7 +714,7 @@ class TestPythonVirtualenvOperator(unittest.TestCase):
raise Exception
with pytest.raises(CalledProcessError):
- self._run_as_operator(f)
+ self.run_as_task(f)
def test_python_3(self):
def f():
@@ -980,13 +727,13 @@ class TestPythonVirtualenvOperator(unittest.TestCase):
return
raise Exception
- self._run_as_operator(f, python_version=3, use_dill=False, requirements=["dill"])
+ self.run_as_task(f, python_version=3, use_dill=False, requirements=["dill"])
def test_without_dill(self):
def f(a):
return a
- self._run_as_operator(f, system_site_packages=False, use_dill=False, op_args=[4])
+ self.run_as_task(f, system_site_packages=False, use_dill=False, op_args=[4])
def test_string_args(self):
def f():
@@ -995,7 +742,7 @@ class TestPythonVirtualenvOperator(unittest.TestCase):
if virtualenv_string_args[0] != virtualenv_string_args[2]:
raise Exception
- self._run_as_operator(f, string_args=[1, 2, 1])
+ self.run_as_task(f, string_args=[1, 2, 1])
def test_with_args(self):
def f(a, b, c=False, d=False):
@@ -1004,37 +751,38 @@ class TestPythonVirtualenvOperator(unittest.TestCase):
else:
raise Exception
- self._run_as_operator(f, op_args=[0, 1], op_kwargs={"c": True})
+ self.run_as_task(f, op_args=[0, 1], op_kwargs={"c": True})
def test_return_none(self):
def f():
return None
- task = self._run_as_operator(f)
+ task = self.run_as_task(f)
assert task.execute_callable() is None
def test_return_false(self):
def f():
return False
- task = self._run_as_operator(f)
+ task = self.run_as_task(f)
assert task.execute_callable() is False
def test_lambda(self):
with pytest.raises(AirflowException):
- PythonVirtualenvOperator(python_callable=lambda x: 4, task_id="task", dag=self.dag)
+ PythonVirtualenvOperator(python_callable=lambda x: 4, task_id=self.task_id)
def test_nonimported_as_arg(self):
def f(_):
return None
- self._run_as_operator(f, op_args=[datetime.utcnow()])
+ self.run_as_task(f, op_args=[datetime.utcnow()])
def test_context(self):
def f(templates_dict):
return templates_dict["ds"]
- self._run_as_operator(f, templates_dict={"ds": "{{ ds }}"})
+ task = self.run_as_task(f, templates_dict={"ds": "{{ ds }}"})
+ assert task.templates_dict == {"ds": self.ds_templated}
# This tests might take longer than default 60 seconds as it is serializing a lot of
# context using dill (which is slow apparently).
@@ -1078,7 +826,7 @@ class TestPythonVirtualenvOperator(unittest.TestCase):
):
pass
- self._run_as_operator(f, use_dill=True, system_site_packages=True, requirements=None)
+ self.run_as_operator(f, use_dill=True, system_site_packages=True, requirements=None)
@pytest.mark.filterwarnings("ignore::airflow.utils.context.AirflowContextDeprecationWarning")
def test_pendulum_context(self):
@@ -1112,7 +860,7 @@ class TestPythonVirtualenvOperator(unittest.TestCase):
):
pass
- self._run_as_operator(f, use_dill=True, system_site_packages=False, requirements=["pendulum"])
+ self.run_as_task(f, use_dill=True, system_site_packages=False, requirements=["pendulum"])
@pytest.mark.filterwarnings("ignore::airflow.utils.context.AirflowContextDeprecationWarning")
def test_base_context(self):
@@ -1140,7 +888,7 @@ class TestPythonVirtualenvOperator(unittest.TestCase):
):
pass
- self._run_as_operator(f, use_dill=True, system_site_packages=False, requirements=None)
+ self.run_as_task(f, use_dill=True, system_site_packages=False, requirements=None)
def test_deepcopy(self):
"""Test that PythonVirtualenvOperator are deep-copyable."""
@@ -1148,13 +896,37 @@ class TestPythonVirtualenvOperator(unittest.TestCase):
def f():
return 1
- task = PythonVirtualenvOperator(
- python_callable=f,
- task_id="task",
- dag=self.dag,
- )
+ task = PythonVirtualenvOperator(python_callable=f, task_id="task")
copy.deepcopy(task)
+ def test_virtualenv_serializable_context_fields(self, create_task_instance):
+ """Ensure all template context fields are listed in the operator.
+
+ This exists mainly so when a field is added to the context, we remember to
+ also add it to PythonVirtualenvOperator.
+ """
+ # These are intentionally NOT serialized into the virtual environment:
+ # * Variables pointing to the task instance itself.
+ # * Variables that are accessor instances.
+ intentionally_excluded_context_keys = [
+ "task_instance",
+ "ti",
+ "var", # Accessor for Variable; var->json and var->value.
+ "conn", # Accessor for Connection.
+ ]
+
+ ti = create_task_instance(dag_id=self.dag_id, task_id=self.task_id, schedule=None)
+ context = ti.get_template_context()
+
+ declared_keys = {
+ *PythonVirtualenvOperator.BASE_SERIALIZABLE_CONTEXT_KEYS,
+ *PythonVirtualenvOperator.PENDULUM_SERIALIZABLE_CONTEXT_KEYS,
+ *PythonVirtualenvOperator.AIRFLOW_SERIALIZABLE_CONTEXT_KEYS,
+ *intentionally_excluded_context_keys,
+ }
+
+ assert set(context) == declared_keys
+
DEFAULT_ARGS = {
"owner": "test",
@@ -1221,7 +993,7 @@ def get_all_the_context(**context):
assert context == current_context._context
-@pytest.fixture()
+@pytest.fixture
def clear_db():
clear_db_runs()
yield
@@ -1239,76 +1011,3 @@ class TestCurrentContextRuntime:
with DAG(dag_id="edge_case_context_dag", default_args=DEFAULT_ARGS):
op = PythonOperator(python_callable=get_all_the_context, task_id="get_all_the_context")
op.run(ignore_first_depends_on_past=True, ignore_ti_state=True)
-
-
-@pytest.mark.parametrize(
- "choice,expected_states",
- [
- ("task1", [State.SUCCESS, State.SUCCESS, State.SUCCESS]),
- ("join", [State.SUCCESS, State.SKIPPED, State.SUCCESS]),
- ],
-)
-def test_empty_branch(dag_maker, choice, expected_states):
- """
- Tests that BranchPythonOperator handles empty branches properly.
- """
- with dag_maker(
- "test_empty_branch",
- start_date=DEFAULT_DATE,
- ) as dag:
- branch = BranchPythonOperator(task_id="branch", python_callable=lambda: choice)
- task1 = EmptyOperator(task_id="task1")
- join = EmptyOperator(task_id="join", trigger_rule="none_failed_min_one_success")
-
- branch >> [task1, join]
- task1 >> join
-
- dag.clear(start_date=DEFAULT_DATE)
- dag_run = dag_maker.create_dagrun()
-
- task_ids = ["branch", "task1", "join"]
- tis = {ti.task_id: ti for ti in dag_run.task_instances}
-
- for task_id in task_ids: # Mimic the specific order the scheduling would run the tests.
- task_instance = tis[task_id]
- task_instance.refresh_from_task(dag.get_task(task_id))
- task_instance.run()
-
- def get_state(ti):
- ti.refresh_from_db()
- return ti.state
-
- assert [get_state(tis[task_id]) for task_id in task_ids] == expected_states
-
-
-def test_virtualenv_serializable_context_fields(create_task_instance):
- """Ensure all template context fields are listed in the operator.
-
- This exists mainly so when a field is added to the context, we remember to
- also add it to PythonVirtualenvOperator.
- """
- # These are intentionally NOT serialized into the virtual environment:
- # * Variables pointing to the task instance itself.
- # * Variables that are accessor instances.
- intentionally_excluded_context_keys = [
- "task_instance",
- "ti",
- "var", # Accessor for Variable; var->json and var->value.
- "conn", # Accessor for Connection.
- ]
-
- ti = create_task_instance(
- dag_id="test_virtualenv_serializable_context_fields",
- task_id="test_virtualenv_serializable_context_fields_task",
- schedule=None,
- )
- context = ti.get_template_context()
-
- declared_keys = {
- *PythonVirtualenvOperator.BASE_SERIALIZABLE_CONTEXT_KEYS,
- *PythonVirtualenvOperator.PENDULUM_SERIALIZABLE_CONTEXT_KEYS,
- *PythonVirtualenvOperator.AIRFLOW_SERIALIZABLE_CONTEXT_KEYS,
- *intentionally_excluded_context_keys,
- }
-
- assert set(context) == declared_keys
diff --git a/tests/sensors/test_python.py b/tests/sensors/test_python.py
index f3c258185d..73a0ddffe4 100644
--- a/tests/sensors/test_python.py
+++ b/tests/sensors/test_python.py
@@ -24,112 +24,52 @@ import pytest
from airflow.exceptions import AirflowSensorTimeout
from airflow.sensors.python import PythonSensor
-from airflow.utils.state import State
-from airflow.utils.timezone import datetime
-from airflow.utils.types import DagRunType
-from tests.operators.test_python import Call, assert_calls_equal, build_recording_function
+from tests.operators.test_python import BasePythonTest
-DEFAULT_DATE = datetime(2015, 1, 1)
+class TestPythonSensor(BasePythonTest):
+ opcls = PythonSensor
-class TestPythonSensor:
- def test_python_sensor_true(self, dag_maker):
- with dag_maker():
- op = PythonSensor(task_id="python_sensor_check_true", python_callable=lambda: True)
- op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE, ignore_ti_state=True)
+ def test_python_sensor_true(self):
+ self.run_as_task(fn=lambda: True)
- def test_python_sensor_false(self, dag_maker):
- with dag_maker():
- op = PythonSensor(
- task_id="python_sensor_check_false",
- timeout=0.01,
- poke_interval=0.01,
- python_callable=lambda: False,
- )
+ def test_python_sensor_false(self):
with pytest.raises(AirflowSensorTimeout):
- op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE, ignore_ti_state=True)
+ self.run_as_task(lambda: False, timeout=0.01, poke_interval=0.01)
- def test_python_sensor_raise(self, dag_maker):
- with dag_maker():
- op = PythonSensor(task_id="python_sensor_check_raise", python_callable=lambda: 1 / 0)
+ def test_python_sensor_raise(self):
with pytest.raises(ZeroDivisionError):
- op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE, ignore_ti_state=True)
+ self.run_as_task(lambda: 1 / 0)
- def test_python_callable_arguments_are_templatized(self, dag_maker):
+ def test_python_callable_arguments_are_templatized(self):
"""Test PythonSensor op_args are templatized"""
- recorded_calls = []
-
# Create a named tuple and ensure it is still preserved
# after the rendering is done
Named = namedtuple("Named", ["var1", "var2"])
named_tuple = Named("{{ ds }}", "unchanged")
- with dag_maker() as dag:
- task = PythonSensor(
- task_id="python_sensor",
- timeout=0.01,
- poke_interval=0.3,
- # a Mock instance cannot be used as a callable function or test fails with a
- # TypeError: Object of type Mock is not JSON serializable
- python_callable=build_recording_function(recorded_calls),
- op_args=[4, date(2019, 1, 1), "dag {{dag.dag_id}} ran on {{ds}}.", named_tuple],
- )
-
- dag.create_dagrun(
- run_type=DagRunType.MANUAL,
- execution_date=DEFAULT_DATE,
- data_interval=(DEFAULT_DATE, DEFAULT_DATE),
- start_date=DEFAULT_DATE,
- state=State.RUNNING,
- )
- with pytest.raises(AirflowSensorTimeout):
- task.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
-
- ds_templated = DEFAULT_DATE.date().isoformat()
- assert_calls_equal(
- recorded_calls[0],
- Call(
- 4,
- date(2019, 1, 1),
- f"dag {dag.dag_id} ran on {ds_templated}.",
- Named(ds_templated, "unchanged"),
- ),
+ task = self.render_templates(
+ lambda: 0,
+ op_args=[4, date(2019, 1, 1), "dag {{dag.dag_id}} ran on {{ds}}.", named_tuple],
)
-
- def test_python_callable_keyword_arguments_are_templatized(self, dag_maker):
+ rendered_op_args = task.op_args
+ assert len(rendered_op_args) == 4
+ assert rendered_op_args[0] == 4
+ assert rendered_op_args[1] == date(2019, 1, 1)
+ assert rendered_op_args[2] == f"dag {self.dag_id} ran on {self.ds_templated}."
+ assert rendered_op_args[3] == Named(self.ds_templated, "unchanged")
+
+ def test_python_callable_keyword_arguments_are_templatized(self):
"""Test PythonSensor op_kwargs are templatized"""
- recorded_calls = []
-
- with dag_maker() as dag:
- task = PythonSensor(
- task_id="python_sensor",
- timeout=0.01,
- poke_interval=0.01,
- # a Mock instance cannot be used as a callable function or test fails with a
- # TypeError: Object of type Mock is not JSON serializable
- python_callable=build_recording_function(recorded_calls),
- op_kwargs={
- "an_int": 4,
- "a_date": date(2019, 1, 1),
- "a_templated_string": "dag {{dag.dag_id}} ran on {{ds}}.",
- },
- )
-
- dag.create_dagrun(
- run_type=DagRunType.MANUAL,
- execution_date=DEFAULT_DATE,
- data_interval=(DEFAULT_DATE, DEFAULT_DATE),
- start_date=DEFAULT_DATE,
- state=State.RUNNING,
- )
- with pytest.raises(AirflowSensorTimeout):
- task.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
-
- assert_calls_equal(
- recorded_calls[0],
- Call(
- an_int=4,
- a_date=date(2019, 1, 1),
- a_templated_string=f"dag {dag.dag_id} ran on {DEFAULT_DATE.date().isoformat()}.",
- ),
+ task = self.render_templates(
+ lambda: 0,
+ op_kwargs={
+ "an_int": 4,
+ "a_date": date(2019, 1, 1),
+ "a_templated_string": "dag {{dag.dag_id}} ran on {{ds}}.",
+ },
)
+ rendered_op_kwargs = task.op_kwargs
+ assert rendered_op_kwargs["an_int"] == 4
+ assert rendered_op_kwargs["a_date"] == date(2019, 1, 1)
+ assert rendered_op_kwargs["a_templated_string"] == f"dag {self.dag_id} ran on {self.ds_templated}."
[airflow] 36/37: Annotate and simplify code samples in DAGs doc (#29027)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit c2dd921d5c540bfe7b68fb72abb8393d12e11834
Author: Bas Harenslak <Ba...@users.noreply.github.com>
AuthorDate: Fri Jan 20 16:48:58 2023 +0100
Annotate and simplify code samples in DAGs doc (#29027)
(cherry picked from commit 80dbfbc7ad8f63db8565baefa282bc01146803fe)
---
docs/apache-airflow/core-concepts/dags.rst | 118 ++++++++++++++++++++---------
1 file changed, 82 insertions(+), 36 deletions(-)
diff --git a/docs/apache-airflow/core-concepts/dags.rst b/docs/apache-airflow/core-concepts/dags.rst
index 527965b9e6..b5cf27361b 100644
--- a/docs/apache-airflow/core-concepts/dags.rst
+++ b/docs/apache-airflow/core-concepts/dags.rst
@@ -35,29 +35,60 @@ Declaring a DAG
---------------
There are three ways to declare a DAG - either you can use a context manager,
-which will add the DAG to anything inside it implicitly::
+which will add the DAG to anything inside it implicitly:
+
+.. code-block:: python
+ :emphasize-lines: 6-10
+
+ import datetime
+
+ from airflow import DAG
+ from airflow.operators.empty import EmptyOperator
with DAG(
- "my_dag_name", start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
- schedule="@daily", catchup=False
+ dag_id="my_dag_name",
+ start_date=datetime.datetime(2021, 1, 1),
+ schedule="@daily",
):
- op = EmptyOperator(task_id="task")
+ EmptyOperator(task_id="task")
+
+
+Or, you can use a standard constructor, passing the DAG into any operators you use:
+
+.. code-block:: python
+ :emphasize-lines: 6-11
+
+ import datetime
+
+ from airflow import DAG
+ from airflow.operators.empty import EmptyOperator
+
+ my_dag = DAG(
+ dag_id="my_dag_name",
+ start_date=datetime.datetime(2021, 1, 1),
+ schedule="@daily",
+ )
+ EmptyOperator(task_id="task", dag=my_dag)
+
+
+Or, you can use the ``@dag`` decorator to :ref:`turn a function into a DAG generator <concepts-dag-decorator>`:
+
+.. code-block:: python
+ :emphasize-lines: 7,8,12
-Or, you can use a standard constructor, passing the dag into any
-operators you use::
+ import datetime
- my_dag = DAG("my_dag_name", start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
- schedule="@daily", catchup=False)
- op = EmptyOperator(task_id="task", dag=my_dag)
+ from airflow.decorators import dag
+ from airflow.operators.empty import EmptyOperator
-Or, you can use the ``@dag`` decorator to :ref:`turn a function into a DAG generator <concepts-dag-decorator>`::
- @dag(start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
- schedule="@daily", catchup=False)
+ @dag(start_date=datetime.datetime(2021, 1, 1), schedule="@daily")
def generate_dag():
- op = EmptyOperator(task_id="task")
+ EmptyOperator(task_id="task")
+
+
+ generate_dag()
- dag = generate_dag()
DAGs are nothing without :doc:`tasks` to run, and those will usually come in the form of either :doc:`operators`, :doc:`sensors` or :doc:`taskflow`.
@@ -214,19 +245,20 @@ Otherwise, you must pass it into each Operator with ``dag=``.
Default Arguments
-----------------
-Often, many Operators inside a DAG need the same set of default arguments (such as their ``retries``). Rather than having to specify this individually for every Operator, you can instead pass ``default_args`` to the DAG when you create it, and it will auto-apply them to any operator tied to it::
+Often, many Operators inside a DAG need the same set of default arguments (such as their ``retries``). Rather than having to specify this individually for every Operator, you can instead pass ``default_args`` to the DAG when you create it, and it will auto-apply them to any operator tied to it:
+.. code-block:: python
+ :emphasize-lines: 7
import pendulum
with DAG(
- dag_id='my_dag',
- start_date=pendulum.datetime(2016, 1, 1, tz="UTC"),
- schedule='@daily',
- catchup=False,
- default_args={'retries': 2},
- ) as dag:
- op = BashOperator(task_id='dummy', bash_command='Hello World!')
+ dag_id="my_dag",
+ start_date=pendulum.datetime(2016, 1, 1),
+ schedule="@daily",
+ default_args={"retries": 2},
+ ):
+ op = BashOperator(task_id="dummy", bash_command="Hello World!")
print(op.retries) # 2
@@ -448,9 +480,12 @@ Dynamic DAGs
Since a DAG is defined by Python code, there is no need for it to be purely declarative; you are free to use loops, functions, and more to define your DAG.
-For example, here is a DAG that uses a ``for`` loop to define some Tasks::
+For example, here is a DAG that uses a ``for`` loop to define some tasks:
+
+.. code-block:: python
+ :emphasize-lines: 7
- with DAG("loop_example") as dag:
+ with DAG("loop_example", ...):
first = EmptyOperator(task_id="first")
last = EmptyOperator(task_id="last")
@@ -487,39 +522,50 @@ Unlike :ref:`concepts:subdags`, TaskGroups are purely a UI grouping concept. Tas
.. image:: /img/task_group.gif
-Dependency relationships can be applied across all tasks in a TaskGroup with the ``>>`` and ``<<`` operators. For example, the following code puts ``task1`` and ``task2`` in TaskGroup ``group1`` and then puts both tasks upstream of ``task3``::
+Dependency relationships can be applied across all tasks in a TaskGroup with the ``>>`` and ``<<`` operators. For example, the following code puts ``task1`` and ``task2`` in TaskGroup ``group1`` and then puts both tasks upstream of ``task3``:
+
+.. code-block:: python
+ :emphasize-lines: 10
from airflow.decorators import task_group
+
@task_group()
def group1():
task1 = EmptyOperator(task_id="task1")
task2 = EmptyOperator(task_id="task2")
+
task3 = EmptyOperator(task_id="task3")
group1() >> task3
-TaskGroup also supports ``default_args`` like DAG, it will overwrite the ``default_args`` in DAG level::
+TaskGroup also supports ``default_args`` like DAG, it will overwrite the ``default_args`` in DAG level:
- import pendulum
+.. code-block:: python
+ :emphasize-lines: 15
+
+ import datetime
+ from airflow import DAG
from airflow.decorators import task_group
+ from airflow.operators.bash import BashOperator
+ from airflow.operators.empty import EmptyOperator
with DAG(
- dag_id='dag1',
- start_date=pendulum.datetime(2016, 1, 1, tz="UTC"),
+ dag_id="dag1",
+ start_date=datetime.datetime(2016, 1, 1),
schedule="@daily",
- catchup=False,
- default_args={'retries': 1},
+ default_args={"retries": 1},
):
- @task_group(default_args={'retries': 3}):
+
+ @task_group(default_args={"retries": 3})
def group1():
"""This docstring will become the tooltip for the TaskGroup."""
- task1 = EmptyOperator(task_id='task1')
- task2 = BashOperator(task_id='task2', bash_command='echo Hello World!', retries=2)
- print(task1.retries) # 3
- print(task2.retries) # 2
+ task1 = EmptyOperator(task_id="task1")
+ task2 = BashOperator(task_id="task2", bash_command="echo Hello World!", retries=2)
+ print(task1.retries) # 3
+ print(task2.retries) # 2
If you want to see a more advanced use of TaskGroup, you can look at the ``example_task_group_decorator.py`` example DAG that comes with Airflow.
[airflow] 37/37: Sanitize url_for arguments before they are passed (#29039)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 2ec4d063b663cef85dc51ea422dae8904dca66d3
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Thu Jan 19 16:37:47 2023 +0100
Sanitize url_for arguments before they are passed (#29039)
The url_for of flask has special arguments that start with `_` and we
should sanitize the ones that come with the request before passing them.
(cherry picked from commit 7f2b065ccd01071cff8f298b944d81f3ff3384b5)
---
airflow/www/views.py | 34 ++++++++++++++++++++++------------
1 file changed, 22 insertions(+), 12 deletions(-)
diff --git a/airflow/www/views.py b/airflow/www/views.py
index 85e4f710cb..33d3997994 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -154,6 +154,16 @@ def truncate_task_duration(task_duration):
return int(task_duration) if task_duration > 10.0 else round(task_duration, 3)
+def sanitize_args(args: dict[str, str]) -> dict[str, str]:
+ """
+ Remove all parameters starting with `_`
+
+ :param args: arguments of request
+ :return: copy of the dictionary passed as input with args starting with `_` removed.
+ """
+ return {key: value for key, value in args.items() if not key.startswith("_")}
+
+
def get_safe_url(url):
"""Given a user-supplied URL, ensure it points to our web server"""
if not url:
@@ -1099,7 +1109,7 @@ class Airflow(AirflowBaseView):
)
def legacy_code(self):
"""Redirect from url param."""
- return redirect(url_for("Airflow.code", **request.args))
+ return redirect(url_for("Airflow.code", **sanitize_args(request.args)))
@expose("/dags/<string:dag_id>/code")
@auth.has_access(
@@ -1146,7 +1156,7 @@ class Airflow(AirflowBaseView):
)
def legacy_dag_details(self):
"""Redirect from url param."""
- return redirect(url_for("Airflow.dag_details", **request.args))
+ return redirect(url_for("Airflow.dag_details", **sanitize_args(request.args)))
@expose("/dags/<string:dag_id>/details")
@auth.has_access(
@@ -2538,7 +2548,7 @@ class Airflow(AirflowBaseView):
@action_logging
def dag(self, dag_id):
"""Redirect to default DAG view."""
- kwargs = {**request.args, "dag_id": dag_id}
+ kwargs = {**sanitize_args(request.args), "dag_id": dag_id}
return redirect(url_for("Airflow.grid", **kwargs))
@expose("/legacy_tree")
@@ -2553,7 +2563,7 @@ class Airflow(AirflowBaseView):
@action_logging
def legacy_tree(self):
"""Redirect to the replacement - grid view."""
- return redirect(url_for("Airflow.grid", **request.args))
+ return redirect(url_for("Airflow.grid", **sanitize_args(request.args)))
@expose("/tree")
@auth.has_access(
@@ -2567,7 +2577,7 @@ class Airflow(AirflowBaseView):
@action_logging
def tree(self):
"""Redirect to the replacement - grid view. Kept for backwards compatibility."""
- return redirect(url_for("Airflow.grid", **request.args))
+ return redirect(url_for("Airflow.grid", **sanitize_args(request.args)))
@expose("/dags/<string:dag_id>/grid")
@auth.has_access(
@@ -2646,7 +2656,7 @@ class Airflow(AirflowBaseView):
@action_logging
def legacy_calendar(self):
"""Redirect from url param."""
- return redirect(url_for("Airflow.calendar", **request.args))
+ return redirect(url_for("Airflow.calendar", **sanitize_args(request.args)))
@expose("/dags/<string:dag_id>/calendar")
@auth.has_access(
@@ -2787,7 +2797,7 @@ class Airflow(AirflowBaseView):
@action_logging
def legacy_graph(self):
"""Redirect from url param."""
- return redirect(url_for("Airflow.graph", **request.args))
+ return redirect(url_for("Airflow.graph", **sanitize_args(request.args)))
@expose("/dags/<string:dag_id>/graph")
@auth.has_access(
@@ -2904,7 +2914,7 @@ class Airflow(AirflowBaseView):
@action_logging
def legacy_duration(self):
"""Redirect from url param."""
- return redirect(url_for("Airflow.duration", **request.args))
+ return redirect(url_for("Airflow.duration", **sanitize_args(request.args)))
@expose("/dags/<string:dag_id>/duration")
@auth.has_access(
@@ -3065,7 +3075,7 @@ class Airflow(AirflowBaseView):
@action_logging
def legacy_tries(self):
"""Redirect from url param."""
- return redirect(url_for("Airflow.tries", **request.args))
+ return redirect(url_for("Airflow.tries", **sanitize_args(request.args)))
@expose("/dags/<string:dag_id>/tries")
@auth.has_access(
@@ -3160,7 +3170,7 @@ class Airflow(AirflowBaseView):
@action_logging
def legacy_landing_times(self):
"""Redirect from url param."""
- return redirect(url_for("Airflow.landing_times", **request.args))
+ return redirect(url_for("Airflow.landing_times", **sanitize_args(request.args)))
@expose("/dags/<string:dag_id>/landing-times")
@auth.has_access(
@@ -3282,7 +3292,7 @@ class Airflow(AirflowBaseView):
@action_logging
def legacy_gantt(self):
"""Redirect from url param."""
- return redirect(url_for("Airflow.gantt", **request.args))
+ return redirect(url_for("Airflow.gantt", **sanitize_args(request.args)))
@expose("/dags/<string:dag_id>/gantt")
@auth.has_access(
@@ -3730,7 +3740,7 @@ class Airflow(AirflowBaseView):
)
def legacy_audit_log(self):
"""Redirect from url param."""
- return redirect(url_for("Airflow.audit_log", **request.args))
+ return redirect(url_for("Airflow.audit_log", **sanitize_args(request.args)))
@expose("/dags/<string:dag_id>/audit_log")
@auth.has_access(
[airflow] 24/37: improve quick start guide (#28949)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 395976578925724de46515b58673a9ba3560ca07
Author: gxcuit <gx...@163.com>
AuthorDate: Mon Jan 16 05:01:08 2023 +0800
improve quick start guide (#28949)
change task run to task test
(cherry picked from commit a4f6f3d6fe614457ff95ac803fd15e9f0bd38d27)
---
docs/apache-airflow/start.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/apache-airflow/start.rst b/docs/apache-airflow/start.rst
index 1c7625dbc2..35c62bdc74 100644
--- a/docs/apache-airflow/start.rst
+++ b/docs/apache-airflow/start.rst
@@ -90,7 +90,7 @@ run the commands below.
.. code-block:: bash
# run your first task instance
- airflow tasks run example_bash_operator runme_0 2015-01-01
+ airflow tasks test example_bash_operator runme_0 2015-01-01
# run a backfill over 2 days
airflow dags backfill example_bash_operator \
--start-date 2015-01-01 \
[airflow] 15/37: KubenetesExecutor sends state even when successful (#28871)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 1eba989f1bd077d563e3a1b07f950367398a8281
Author: Jed Cunningham <66...@users.noreply.github.com>
AuthorDate: Fri Jan 13 00:35:16 2023 -0600
KubenetesExecutor sends state even when successful (#28871)
Co-authored-by: Tzu-ping Chung <ur...@gmail.com>
(cherry picked from commit 8a9959cc1ead49785b10b9b28c101e3d94cb4176)
---
airflow/executors/kubernetes_executor.py | 37 ++++++++++++++++++-----------
tests/executors/test_kubernetes_executor.py | 25 +++++++++++++++++--
2 files changed, 46 insertions(+), 16 deletions(-)
diff --git a/airflow/executors/kubernetes_executor.py b/airflow/executors/kubernetes_executor.py
index 8098486ded..34204539fa 100644
--- a/airflow/executors/kubernetes_executor.py
+++ b/airflow/executors/kubernetes_executor.py
@@ -214,7 +214,7 @@ class KubernetesJobWatcher(multiprocessing.Process, LoggingMixin):
self.watcher_queue.put((pod_id, namespace, State.FAILED, annotations, resource_version))
elif status == "Succeeded":
self.log.info("Event: %s Succeeded", pod_id)
- self.watcher_queue.put((pod_id, namespace, None, annotations, resource_version))
+ self.watcher_queue.put((pod_id, namespace, State.SUCCESS, annotations, resource_version))
elif status == "Running":
if event["type"] == "DELETED":
self.log.info("Event: Pod %s deleted before it could complete", pod_id)
@@ -719,19 +719,26 @@ class KubernetesExecutor(BaseExecutor):
if TYPE_CHECKING:
assert self.kube_scheduler
- if state != State.RUNNING:
- if self.kube_config.delete_worker_pods:
- if state != State.FAILED or self.kube_config.delete_worker_pods_on_failure:
- self.kube_scheduler.delete_pod(pod_id, namespace)
- self.log.info("Deleted pod: %s in namespace %s", str(key), str(namespace))
- else:
- self.kube_scheduler.patch_pod_executor_done(pod_id=pod_id, namespace=namespace)
- self.log.info("Patched pod %s in namespace %s to mark it as done", str(key), str(namespace))
- try:
- self.running.remove(key)
- except KeyError:
- self.log.debug("Could not find key: %s", str(key))
- self.event_buffer[key] = state, None
+ if state == State.RUNNING:
+ self.event_buffer[key] = state, None
+ return
+
+ if self.kube_config.delete_worker_pods:
+ if state != State.FAILED or self.kube_config.delete_worker_pods_on_failure:
+ self.kube_scheduler.delete_pod(pod_id, namespace)
+ self.log.info("Deleted pod: %s in namespace %s", str(key), str(namespace))
+ else:
+ self.kube_scheduler.patch_pod_executor_done(pod_id=pod_id, namespace=namespace)
+ self.log.info("Patched pod %s in namespace %s to mark it as done", str(key), str(namespace))
+
+ try:
+ self.running.remove(key)
+ except KeyError:
+ self.log.debug("TI key not in running, not adding to event_buffer: %s", key)
+ else:
+ # We get multiple events once the pod hits a terminal state, and we only want to
+ # do this once, so only do it when we remove the task from running
+ self.event_buffer[key] = state, None
def try_adopt_task_instances(self, tis: Sequence[TaskInstance]) -> Sequence[TaskInstance]:
tis_to_flush = [ti for ti in tis if not ti.queued_by_job_id]
@@ -809,6 +816,8 @@ class KubernetesExecutor(BaseExecutor):
)
except ApiException as e:
self.log.info("Failed to adopt pod %s. Reason: %s", pod.metadata.name, e)
+ pod_id = annotations_to_key(pod.metadata.annotations)
+ self.running.add(pod_id)
def _flush_task_queue(self) -> None:
if TYPE_CHECKING:
diff --git a/tests/executors/test_kubernetes_executor.py b/tests/executors/test_kubernetes_executor.py
index 9f5304ba4a..d24f3e8fd9 100644
--- a/tests/executors/test_kubernetes_executor.py
+++ b/tests/executors/test_kubernetes_executor.py
@@ -510,8 +510,10 @@ class TestKubernetesExecutor:
executor.start()
try:
key = ("dag_id", "task_id", "run_id", "try_number1")
+ executor.running = {key}
executor._change_state(key, State.RUNNING, "pod_id", "default")
assert executor.event_buffer[key][0] == State.RUNNING
+ assert executor.running == {key}
finally:
executor.end()
@@ -523,8 +525,10 @@ class TestKubernetesExecutor:
executor.start()
try:
key = ("dag_id", "task_id", "run_id", "try_number2")
+ executor.running = {key}
executor._change_state(key, State.SUCCESS, "pod_id", "default")
assert executor.event_buffer[key][0] == State.SUCCESS
+ assert executor.running == set()
mock_delete_pod.assert_called_once_with("pod_id", "default")
finally:
executor.end()
@@ -541,8 +545,10 @@ class TestKubernetesExecutor:
executor.start()
try:
key = ("dag_id", "task_id", "run_id", "try_number3")
+ executor.running = {key}
executor._change_state(key, State.FAILED, "pod_id", "default")
assert executor.event_buffer[key][0] == State.FAILED
+ assert executor.running == set()
mock_delete_pod.assert_not_called()
finally:
executor.end()
@@ -562,8 +568,10 @@ class TestKubernetesExecutor:
executor.start()
try:
key = ("dag_id", "task_id", "run_id", "try_number2")
+ executor.running = {key}
executor._change_state(key, State.SUCCESS, "pod_id", "test-namespace")
assert executor.event_buffer[key][0] == State.SUCCESS
+ assert executor.running == set()
mock_delete_pod.assert_not_called()
mock_patch_pod.assert_called_once_with(pod_id="pod_id", namespace="test-namespace")
finally:
@@ -583,8 +591,10 @@ class TestKubernetesExecutor:
executor.start()
try:
key = ("dag_id", "task_id", "run_id", "try_number2")
+ executor.running = {key}
executor._change_state(key, State.FAILED, "pod_id", "test-namespace")
assert executor.event_buffer[key][0] == State.FAILED
+ assert executor.running == set()
mock_delete_pod.assert_called_once_with("pod_id", "test-namespace")
mock_patch_pod.assert_not_called()
finally:
@@ -733,17 +743,27 @@ class TestKubernetesExecutor:
executor.kube_client = mock_kube_client
executor.kube_config.kube_namespace = "somens"
pod_names = ["one", "two"]
+
+ def get_annotations(pod_name):
+ return {
+ "dag_id": "dag",
+ "run_id": "run_id",
+ "task_id": pod_name,
+ "try_number": "1",
+ }
+
mock_kube_client.list_namespaced_pod.return_value.items = [
k8s.V1Pod(
metadata=k8s.V1ObjectMeta(
name=pod_name,
labels={"airflow-worker": pod_name},
- annotations={"some_annotation": "hello"},
+ annotations=get_annotations(pod_name),
namespace="somens",
)
)
for pod_name in pod_names
]
+ expected_running_ti_keys = {annotations_to_key(get_annotations(pod_name)) for pod_name in pod_names}
executor._adopt_completed_pods(mock_kube_client)
mock_kube_client.list_namespaced_pod.assert_called_once_with(
@@ -763,6 +783,7 @@ class TestKubernetesExecutor:
],
any_order=True,
)
+ assert executor.running == expected_running_ti_keys
@mock.patch("airflow.executors.kubernetes_executor.get_kube_client")
def test_not_adopt_unassigned_task(self, mock_kube_client):
@@ -1144,7 +1165,7 @@ class TestKubernetesJobWatcher:
self.events.append({"type": "MODIFIED", "object": self.pod})
self._run()
- self.assert_watcher_queue_called_once_with_state(None)
+ self.assert_watcher_queue_called_once_with_state(State.SUCCESS)
def test_process_status_running_deleted(self):
self.pod.status.phase = "Running"
[airflow] 07/37: Fix #28391 manual task trigger from UI fails for k8s executor (#28394)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 2a19dd7741e76eb45be87a7e55d0ea1c736b73dd
Author: sanjayp <sa...@gmail.com>
AuthorDate: Tue Jan 24 09:18:45 2023 -0600
Fix #28391 manual task trigger from UI fails for k8s executor (#28394)
Manual task trigger from UI fails for k8s executor. the executor.job_id
is currently set to "manual". the task instance queued_by_job_id field
is expected to be None|Integer. this causes the filter query in
clear_not_launched_queued_tasks method in kubernetes_executor to fail
with psycopg2.errors.InvalidTextRepresentation invalid input syntax for integer: "manual" error.
setting the job_id to None fixes the issue.
(cherry picked from commit 9510043546d1ac8ac56b67bafa537e4b940d68a4)
---
airflow/cli/commands/task_command.py | 2 +-
airflow/executors/kubernetes_executor.py | 2 --
airflow/www/views.py | 2 +-
3 files changed, 2 insertions(+), 4 deletions(-)
diff --git a/airflow/cli/commands/task_command.py b/airflow/cli/commands/task_command.py
index 2f37579c35..b514754f65 100644
--- a/airflow/cli/commands/task_command.py
+++ b/airflow/cli/commands/task_command.py
@@ -218,7 +218,7 @@ def _run_task_by_executor(args, dag, ti):
print(e)
raise e
executor = ExecutorLoader.get_default_executor()
- executor.job_id = "manual"
+ executor.job_id = None
executor.start()
print("Sending to executor.")
executor.queue_task_instance(
diff --git a/airflow/executors/kubernetes_executor.py b/airflow/executors/kubernetes_executor.py
index 28f720f35e..7dcba12968 100644
--- a/airflow/executors/kubernetes_executor.py
+++ b/airflow/executors/kubernetes_executor.py
@@ -526,8 +526,6 @@ class KubernetesExecutor(BaseExecutor):
def start(self) -> None:
"""Starts the executor"""
self.log.info("Start Kubernetes executor")
- if not self.job_id:
- raise AirflowException("Could not get scheduler_job_id")
self.scheduler_job_id = str(self.job_id)
self.log.debug("Start with scheduler_job_id: %s", self.scheduler_job_id)
self.kube_client = get_kube_client()
diff --git a/airflow/www/views.py b/airflow/www/views.py
index 7e8abd02a4..85e4f710cb 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -1795,7 +1795,7 @@ class Airflow(AirflowBaseView):
msg = f"Could not queue task instance for execution, dependencies not met: {failed_deps_str}"
return redirect_or_json(origin, msg, "error", 400)
- executor.job_id = "manual"
+ executor.job_id = None
executor.start()
executor.queue_task_instance(
ti,
[airflow] 09/37: Throttle streaming log reads (#28818)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 76b279681723b626063f5425934beca6cb43b1c7
Author: Daniel Standish <15...@users.noreply.github.com>
AuthorDate: Tue Jan 10 22:16:11 2023 -0800
Throttle streaming log reads (#28818)
Co-authored-by: Jed Cunningham <66...@users.noreply.github.com>
(cherry picked from commit 46704eed5340ef0e71cf828bb560b3d00aa88691)
---
airflow/utils/log/log_reader.py | 17 ++++++++++++++---
1 file changed, 14 insertions(+), 3 deletions(-)
diff --git a/airflow/utils/log/log_reader.py b/airflow/utils/log/log_reader.py
index 7ad4700195..5cc8b9377e 100644
--- a/airflow/utils/log/log_reader.py
+++ b/airflow/utils/log/log_reader.py
@@ -17,6 +17,7 @@
from __future__ import annotations
import logging
+import time
from typing import Iterator
from sqlalchemy.orm.session import Session
@@ -33,6 +34,9 @@ from airflow.utils.state import State
class TaskLogReader:
"""Task log reader"""
+ STREAM_LOOP_SLEEP_SECONDS = 0.5
+ """Time to sleep between loops while waiting for more logs"""
+
def read_log_chunks(
self, ti: TaskInstance, try_number: int | None, metadata
) -> tuple[list[tuple[tuple[str, str]]], dict[str, str]]:
@@ -77,12 +81,19 @@ class TaskLogReader:
metadata.pop("max_offset", None)
metadata.pop("offset", None)
metadata.pop("log_pos", None)
- while "end_of_log" not in metadata or (
- not metadata["end_of_log"] and ti.state not in [State.RUNNING, State.DEFERRED]
- ):
+ while True:
logs, metadata = self.read_log_chunks(ti, current_try_number, metadata)
for host, log in logs[0]:
yield "\n".join([host or "", log]) + "\n"
+ if "end_of_log" not in metadata or (
+ not metadata["end_of_log"] and ti.state not in [State.RUNNING, State.DEFERRED]
+ ):
+ if not logs[0]:
+ # we did not receive any logs in this loop
+ # sleeping to conserve resources / limit requests on external services
+ time.sleep(self.STREAM_LOOP_SLEEP_SECONDS)
+ else:
+ break
@cached_property
def log_handler(self):
[airflow] 26/37: Bump epoch on python dependencies in Airlfow Image (#28960)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit c487daf254739d6d4d9881f7907110268728a80f
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Mon Jan 16 11:43:20 2023 +0100
Bump epoch on python dependencies in Airlfow Image (#28960)
Recently some packages were removed from Airflow but they are still
available in cached image in Github and on clean install of the
image it causes conflicting dependencies when `pip check` is run.
This change will invalidate cache and reinstall base dependencies
from the scratch - thus removing some dependencies.
(cherry picked from commit fe46a934fc298ec3ac74e6c40ca7e1872d2482f9)
---
Dockerfile.ci | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Dockerfile.ci b/Dockerfile.ci
index cc28c355d9..f3d9bff8aa 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -1112,7 +1112,7 @@ ARG AIRFLOW_CONSTRAINTS_LOCATION=""
ARG DEFAULT_CONSTRAINTS_BRANCH="constraints-main"
# By changing the epoch we can force reinstalling Airflow and pip all dependencies
# It can also be overwritten manually by setting the AIRFLOW_CI_BUILD_EPOCH environment variable.
-ARG AIRFLOW_CI_BUILD_EPOCH="3"
+ARG AIRFLOW_CI_BUILD_EPOCH="4"
ARG AIRFLOW_PRE_CACHED_PIP_PACKAGES="true"
# By default in the image, we are installing all providers when installing from sources
ARG INSTALL_PROVIDERS_FROM_SOURCES="true"
[airflow] 03/37: Improving the release process (#27829)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit a0ec6fe9fc755dbf224735edd808e396d8383760
Author: Ephraim Anierobi <sp...@gmail.com>
AuthorDate: Tue Jan 24 09:21:28 2023 +0100
Improving the release process (#27829)
* Improving the release process
This PR aims to automate some parts of the release process.
For RC, we have a command to run the release process:
breeze release-management start-rc-process --version 2.4.3rc1 --previous-version 2.4.2
For release, the below command runs the process:
breeze release-management start-release --release-candidate 2.4.3rc1 --previous-release <PREVIOUS RELEASE>
And there's a command for creating a version branch when releasing a minor version:
breeze release-management create-minor-branch --version-branch ${VERSION_BRANCH}
Use breeze's user_confirm & get_console.print
Skip some steps in CI
Run command in CI
add user_confirm_bool function, a version of user_confirm
Remove retagging of images
simplify console.print calls
Update readme config hash
Use dry-run for commands that shouldn't run in CI
Remove outdated instruction
Remove single quotes on double quote
* fixup! Improving the release process
* fixup! fixup! Improving the release process
* remove hidden arg from commands
* Add check=True to all the run_command to raise when there's a failure
* fixup! Add check=True to all the run_command to raise when there's a failure
* Update minor release command
* Update the ci
(cherry picked from commit 9411e395ecc3bf54ddc341b0f0e227608bd7f972)
---
.github/workflows/ci.yml | 36 ++
dev/README_RELEASE_AIRFLOW.md | 367 ++-------------------
dev/breeze/README.md | 2 +-
dev/breeze/setup.cfg | 5 +
.../commands/minor_release_command.py | 191 +++++++++++
.../commands/release_candidate_command.py | 346 +++++++++++++++++++
.../src/airflow_breeze/commands/release_command.py | 291 ++++++++++++++++
.../commands/release_management_commands.py | 9 +
dev/breeze/src/airflow_breeze/utils/confirm.py | 17 +
dev/breeze/src/airflow_breeze/utils/console.py | 4 +
images/breeze/output-commands-hash.txt | 12 +-
images/breeze/output-commands.svg | 88 ++---
images/breeze/output_ci-image.svg | 40 +--
images/breeze/output_release-management.svg | 30 +-
images/breeze/output_setup.svg | 27 +-
.../output_setup_check-all-params-in-groups.svg | 172 ++++++++++
.../output_setup_regenerate-command-images.svg | 68 ++--
17 files changed, 1232 insertions(+), 473 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index d4163671bd..86440754e3 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -756,6 +756,42 @@ jobs:
run: breeze ci fix-ownership
if: always()
+ test-airflow-release-commands:
+ timeout-minutes: 80
+ name: "Test Airflow release commands"
+ runs-on: "${{needs.build-info.outputs.runs-on}}"
+ needs: [build-info, wait-for-ci-images]
+ if: needs.build-info.outputs.runs-on == 'self-hosted'
+ env:
+ RUNS_ON: "${{needs.build-info.outputs.runs-on}}"
+ PYTHON_MAJOR_MINOR_VERSION: "${{needs.build-info.outputs.default-python-version}}"
+ steps:
+ - name: Cleanup repo
+ run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
+ uses: actions/checkout@v3
+ with:
+ persist-credentials: false
+ - name: >
+ Prepare breeze & CI image: ${{needs.build-info.outputs.default-python-version}}:${{env.IMAGE_TAG}}
+ uses: ./.github/actions/prepare_breeze_and_image
+ - name: "Cleanup dist files"
+ run: rm -fv ./dist/*
+ - name: "Install required packages"
+ run: python -m pip install --editable ./dev/breeze/
+ - name: "Check Airflow create minor branch command"
+ run: breeze release-management create-minor-branch --version-branch 2-4 -a y
+ - name: "Check Airflow RC process command"
+ run: |
+ breeze release-management start-rc-process --version 2.4.3rc1 --previous-version 2.4.2 -a y
+ - name: "Check Airflow release process command"
+ run: |
+ breeze release-management start-release --release-candidate 2.4.3rc1 --previous-release 2.4.2 -a y
+ - name: "Fix ownership"
+ run: breeze ci fix-ownership
+ if: always()
+
+
tests-helm:
timeout-minutes: 80
name: "Python unit tests for Helm chart"
diff --git a/dev/README_RELEASE_AIRFLOW.md b/dev/README_RELEASE_AIRFLOW.md
index 0e905cf706..f4eb79a437 100644
--- a/dev/README_RELEASE_AIRFLOW.md
+++ b/dev/README_RELEASE_AIRFLOW.md
@@ -26,8 +26,6 @@
- [Prepare the Apache Airflow Package RC](#prepare-the-apache-airflow-package-rc)
- [Update the milestone](#update-the-milestone)
- [Build RC artifacts](#build-rc-artifacts)
- - [Prepare new release branches and cache - optional when first minor version is released](#prepare-new-release-branches-and-cache---optional-when-first-minor-version-is-released)
- - [Prepare PyPI convenience "snapshot" packages](#prepare-pypi-convenience-snapshot-packages)
- [Prepare production Docker Image RC](#prepare-production-docker-image-rc)
- [Prepare API clients RC packages](#prepare-api-clients-rc-packages)
- [Prepare issue for testing status of rc](#prepare-issue-for-testing-status-of-rc)
@@ -41,7 +39,6 @@
- [Publish the final Apache Airflow release](#publish-the-final-apache-airflow-release)
- [Summarize the voting for the Apache Airflow release](#summarize-the-voting-for-the-apache-airflow-release)
- [Publish release to SVN](#publish-release-to-svn)
- - [Prepare PyPI "release" packages](#prepare-pypi-release-packages)
- [Manually release API clients](#manually-release-api-clients)
- [Manually prepare production Docker Image](#manually-prepare-production-docker-image)
- [Verify production images](#verify-production-images)
@@ -212,11 +209,19 @@ The Release Candidate artifacts we vote upon should be the exact ones we vote ag
export AIRFLOW_REPO_ROOT=$(pwd)
```
-- Check out the 'test' branch
+- Install `breeze` command:
+
+ ```shell script
+ pipx install -e ./dev/breeze
+ ```
+
+- For major/minor version release,run the following commands to create the 'test' and 'stable' branches.
- For major/minor version release, please follow the instructions at
- [Prepare new release branches and cache](#prepare-new-release-branches-and-cache---optional-when-first-minor-version-is-released)
- to create the 'test' and 'stable' branches.
+ ```shell script
+ breeze release-management create-minor-branch --version-branch ${VERSION_BRANCH}
+ ```
+
+- Check out the 'test' branch
```shell script
git checkout v${VERSION_BRANCH}-test
@@ -240,266 +245,22 @@ The Release Candidate artifacts we vote upon should be the exact ones we vote ag
- Update the `REVISION_HEADS_MAP` at airflow/utils/db.py to include the revision head of the release even if there are no migrations.
- Commit the version change.
-- Check out the 'stable' branch
-
- ```shell script
- git checkout v${VERSION_BRANCH}-stable
- git reset --hard origin/v${VERSION_BRANCH}-stable
- ```
-
-- PR from the 'test' branch to the 'stable' branch, and manually merge it once approved. Here's how to manually merge the PR:
-
- ```shell script
- git merge --ff-only v${VERSION_BRANCH}-test
- git push origin v${VERSION_BRANCH}-stable
- ```
-
-- Tag your release
-
- ```shell script
- git tag -s ${VERSION} -m "Apache Airflow ${VERSION}"
- ```
-
-- Clean the checkout repo
-
- ```shell script
- git clean -fxd
- ```
-
-- Restore breeze installation (The breeze's `.egginfo` is cleared by git-clean)
-
- ```shell script
- pipx install -e ./dev/breeze --force
- ```
-
-- Make sure you have the latest CI image
-
- ```shell script
- breeze ci-image build --python 3.7
- ```
-
-- Tarball the repo
-
- ```shell script
- git archive --format=tar.gz ${VERSION} \
- --prefix=apache-airflow-${VERSION_WITHOUT_RC}/ \
- -o dist/apache-airflow-${VERSION_WITHOUT_RC}-source.tar.gz
- ```
-
- Copy the tarball to a location outside of the repo and verify licences.
-
-- Generate SHA512/ASC (If you have not generated a key yet, generate it by following instructions on http://www.apache.org/dev/openpgp.html#key-gen-generate-key)
-
- ```shell script
- breeze release-management prepare-airflow-package --package-format both
- pushd dist
- ${AIRFLOW_REPO_ROOT}/dev/sign.sh *
- popd
- ```
-
-- If you aren't using Breeze for packaging, build the distribution and wheel files directly
-
- ```shell script
- python setup.py compile_assets sdist bdist_wheel
- pushd dist
- ${AIRFLOW_REPO_ROOT}/dev/sign.sh *
- popd
- ```
-
-- Tag & Push the constraints files. This pushes constraints with rc suffix (this is expected)!
-
- ```shell script
- git checkout origin/constraints-${VERSION_BRANCH}
- git tag -s "constraints-${VERSION}" -m "Constraints for Apache Airflow ${VERSION}"
- git push origin tag "constraints-${VERSION}"
- ```
-
-- Push the artifacts to ASF dev dist repo
-
- ```shell script
- # First clone the repo
-
- [ -d asf-dist ] || svn checkout --depth=immediates https://dist.apache.org/repos/dist asf-dist
- svn update --set-depth=infinity asf-dist/dev/airflow
- cd asf-dist/dev/airflow
-
- # Create new folder for the release
- svn mkdir ${VERSION}
-
- # Move the artifacts to svn folder & commit
- mv ${AIRFLOW_REPO_ROOT}/dist/* ${VERSION}/
- cd ${VERSION}
- svn add *
- svn commit -m "Add artifacts for Airflow ${VERSION}"
- cd ${AIRFLOW_REPO_ROOT}
- rm -rf asf-dist
- ```
-
-## Prepare new release branches and cache - optional when first minor version is released
-
-When you just released the `X.Y.0` version (first release of new minor version) you need to create release
-branches: `vX-Y-test` and `vX-Y-stable` (for example with `2.1.0rc1` release you need to create v2-1-test and
-`v2-1-stable` branches). You also need to configure the branch
-
-### Create test source branch
-
- ```shell script
- # First clone the repo
- export BRANCH_PREFIX=2-1
- git branch v${BRANCH_PREFIX}-test
- ```
-
-### Re-tag images from main
-
-Run script to re-tag images from the ``main`` branch to the ``vX-Y-test`` branch:
-
- ```shell script
- ./dev/retag_docker_images.py --source-branch main --target-branch v${BRANCH_PREFIX}-test
- ```
-
-
-### Update default branches
-
-#### In Breeze
-
-In ``./dev/breeze/src/airflow_breeze/branch_defaults.py`` update branches to reflect the new branch:
-
-```python
-AIRFLOW_BRANCH = "main"
-DEFAULT_AIRFLOW_CONSTRAINTS_BRANCH = "constraints-main"
-```
-
-should become this, where ``X-Y`` is your new branch version:
-
-```python
-AIRFLOW_BRANCH = "vX-Y-test"
-DEFAULT_AIRFLOW_CONSTRAINTS_BRANCH = "constraints-X-Y"
-```
-
-### Commit the changes to the test branch
-
-```bash
-git add -p .
-git commit "Update default branches for ${BRANCH_PREFIX}"
-```
-
-### Create stable branch
-
-```bash
-git branch v${BRANCH_PREFIX}-stable
-````
-### Push test and stable branch
+- PR from the 'test' branch to the 'stable' branch
-```bash
-git checkout v${BRANCH_PREFIX}-test
-git push --set-upstream origin v${BRANCH_PREFIX}-test
-git checkout v${BRANCH_PREFIX}-stable
-git push --set-upstream origin v${BRANCH_PREFIX}-stable
-````
-
-### Add branches in the main branch
-
-You have to do those steps in the `main` branch of the repository:
-
-```bash
-git checkout main
-git pull
-```
-
-Add ``vX-Y-stable`` and ``vX-Y-test`` branches in ``codecov.yml`` (there are 2 places in the file!)
-
-```yaml
- branches:
- - main
- - v2-0-stable
- - v2-0-test
- - v2-1-stable
- - v2-1-test
- - v2-2-stable
- - v2-2-test
-```
-
-Add vX-Y-stable to `.asf.yaml` (X-Y is your new branch)
-
-```yaml
-protected_branches:
- main:
- required_pull_request_reviews:
- required_approving_review_count: 1
- ...
- vX-Y-stable:
- required_pull_request_reviews:
- required_approving_review_count: 1
-
-```
-
-### Create constraints branch out of the constraints-main one
-
- ```shell script
- # First clone the repo
- export BRANCH_PREFIX=2-1
- git checkout constraints-main
- git checkout -b constraints-${BRANCH_PREFIX}
- git push --set-upstream origin constraints-${BRANCH_PREFIX}
- ```
-
-
-## Prepare PyPI convenience "snapshot" packages
-
-At this point we have the artifact that we vote on, but as a convenience to developers we also want to
-publish "snapshots" of the RC builds to PyPI for installing via pip:
-
-To do this we need to
-
-- Checkout the rc tag:
+- When the PR is approved, install `dev/breeze` in a virtualenv:
```shell script
- cd "${AIRFLOW_REPO_ROOT}"
- git checkout ${VERSION}
+ pip install -e ./dev/breeze
```
-- Build the package:
-
- ```shell script
- breeze release-management prepare-airflow-package --version-suffix-for-pypi "${VERSION_SUFFIX}" --package-format both
- ```
-
-- Verify the artifacts that would be uploaded:
-
- ```shell script
- twine check dist/*
- ```
-
-- Upload the package to PyPI's test environment:
-
- ```shell script
- twine upload -r pypitest dist/*
- ```
-
-- Verify that the test package looks good by downloading it and installing it into a virtual environment. The package download link is available at:
-https://test.pypi.org/project/apache-airflow/#files
-
-- Upload the package to PyPI's production environment:
-
- ```shell script
- twine upload -r pypi dist/*
- ```
-
-- Again, confirm that the package is available here:
-https://pypi.python.org/pypi/apache-airflow
-
-It is important to stress that this snapshot should not be named "release", and it
-is not supposed to be used by and advertised to the end-users who do not read the devlist.
-
-- Push Tag for the release candidate
+- Set `GITHUB_TOKEN` environment variable. Needed in patch release for generating issue for testing of the RC
- This step should only be done now and not before, because it triggers an automated build of
- the production docker image, using the packages that are currently released in PyPI
- (both airflow and latest provider packages).
+- Start the release candidate process by running the below command (If you have not generated a key yet, generate it by following instructions on
+ http://www.apache.org/dev/openpgp.html#key-gen-generate-key):
```shell script
- git push origin tag ${VERSION}
+ breeze release-management start-rc-process --version ${VERSION} --previous-version <PREVIOUS_VERSION>
```
## Prepare production Docker Image RC
@@ -908,7 +669,6 @@ Cheers,
<your name>
```
-
## Publish release to SVN
You need to migrate the RC artifacts that passed to this repository:
@@ -918,96 +678,11 @@ https://dist.apache.org/repos/dist/release/airflow/
The best way of doing this is to svn cp between the two repos (this avoids having to upload the binaries again, and gives a clearer history in the svn commit logs):
```shell script
-# GO to Airflow Sources first
-cd <YOUR_AIRFLOW_REPO_ROOT>
-export AIRFLOW_REPO_ROOT="$(pwd)"
-cd ..
-
-[ -d asf-dist ] || svn checkout --depth=immediates https://dist.apache.org/repos/dist asf-dist
-svn update --set-depth=infinity asf-dist/{release,dev}/airflow
-AIRFLOW_DEV_SVN="${PWD}/asf-dist/dev/airflow"
-AIRFLOW_RELEASE_SVN="${PWD}/asf-dist/release/airflow"
-cd "${AIRFLOW_RELEASE_SVN}"
-
export RC=2.0.2rc5
-export VERSION=${RC/rc?/}
-
-# Create new folder for the release
-svn mkdir "${VERSION}"
-cd "${VERSION}"
-
-# Move the artifacts to svn folder & commit
-for f in ${AIRFLOW_DEV_SVN}/$RC/*; do
- svn cp "$f" "${$(basename $f)/}"
-done
-svn commit -m "Release Airflow ${VERSION} from ${RC}"
-
-# Remove old release
-# See http://www.apache.org/legal/release-policy.html#when-to-archive
-cd ..
-export PREVIOUS_VERSION=2.0.2
-svn rm "${PREVIOUS_VERSION}"
-svn commit -m "Remove old release: ${PREVIOUS_VERSION}"
+# start the release process by running the below command
+breeze release-management start-release --release-candidate ${RC} --previous-release <PREVIOUS RELEASE>
```
-Verify that the packages appear in [airflow](https://dist.apache.org/repos/dist/release/airflow/)
-
-## Prepare PyPI "release" packages
-
-At this point we release an official package:
-
-- Verify the artifacts that would be uploaded:
-
- ```shell script
- cd "${AIRFLOW_RELEASE_SVN}/${VERSION}"
- twine check *.whl *${VERSION}.tar.gz
- ```
-
-- Upload the package to PyPI's test environment:
-
- ```shell script
- twine upload -r pypitest *.whl *${VERSION}.tar.gz
- ```
-
-- Verify that the test package looks good by downloading it and installing it into a virtual environment.
- The package download link is available at: https://test.pypi.org/project/apache-airflow/#files
-
-- Upload the package to PyPI's production environment:
-
- ```shell script
- twine upload -r pypi *.whl *${VERSION}.tar.gz
- ```
-
-- Again, confirm that the package is available here: https://pypi.python.org/pypi/apache-airflow
-
-- Re-Tag & Push the constraints files with the final release version.
-
- ```shell script
- cd "${AIRFLOW_REPO_ROOT}"
- git checkout constraints-${RC}
- git tag -s "constraints-${VERSION}" -m "Constraints for Apache Airflow ${VERSION}"
- git push origin tag "constraints-${VERSION}"
- ```
-
-- In case you release "latest stable" version, also update "latest" constraints
-
- ```shell script
- git tag -f -s "constraints-latest" -m "Latest constraints set to Apache Airflow ${VERSION}"
- git push -f origin tag "constraints-latest"
- ```
-
-- Push Tag for the final version
-
- This step should only be done now and not before, because it triggers an automated build of
- the production docker image, using the packages that are currently released in PyPI
- (both airflow and latest provider packages).
-
- ```shell script
- git checkout ${RC}
- git tag -s ${VERSION} -m "Apache Airflow ${VERSION}"
- git push origin tag ${VERSION}
- ```
-
## Manually release API clients
If API clients are part of the release, publish new versions of the clients to PyPI without the rc suffix
diff --git a/dev/breeze/README.md b/dev/breeze/README.md
index 0c86259305..94238f30ef 100644
--- a/dev/breeze/README.md
+++ b/dev/breeze/README.md
@@ -52,6 +52,6 @@ PLEASE DO NOT MODIFY THE HASH BELOW! IT IS AUTOMATICALLY UPDATED BY PRE-COMMIT.
---------------------------------------------------------------------------------------------------------
-Package config hash: f28f0d555b81a0f48d6b29b3cf8bba132b8c6a8f3d290a25ad4fd62019a9adbf86c0dc913c474e23ae110f3f433db0214bf46b21000f0d2bdd0884134923ae91
+Package config hash: a98c6dc9a12523e5aaacaade98c721e869377ab59eead92344006f7fd71fe2978398b87703dfc5efa61fac372a4e798cef9de26d6f4283a47f919cd7e0dc4dfa
---------------------------------------------------------------------------------------------------------
diff --git a/dev/breeze/setup.cfg b/dev/breeze/setup.cfg
index fe4fef0225..7075da8b4c 100644
--- a/dev/breeze/setup.cfg
+++ b/dev/breeze/setup.cfg
@@ -69,6 +69,11 @@ install_requires =
requests
rich>=12.6.0
rich-click>=1.5
+ gitpython
+ twine
+ wheel
+ setuptools
+ jinja2
[options.packages.find]
where=src
diff --git a/dev/breeze/src/airflow_breeze/commands/minor_release_command.py b/dev/breeze/src/airflow_breeze/commands/minor_release_command.py
new file mode 100644
index 0000000000..76ff59c34c
--- /dev/null
+++ b/dev/breeze/src/airflow_breeze/commands/minor_release_command.py
@@ -0,0 +1,191 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+
+import click
+
+from airflow_breeze.utils.common_options import option_answer
+from airflow_breeze.utils.confirm import confirm_action
+from airflow_breeze.utils.console import console_print
+from airflow_breeze.utils.path_utils import AIRFLOW_SOURCES_ROOT
+from airflow_breeze.utils.run_utils import run_command
+
+CI = os.environ.get("CI")
+DRY_RUN = True if CI else False
+
+
+def create_branch(version_branch):
+ if confirm_action(f"Create version branch: {version_branch}?"):
+ run_command(["git", "checkout", "main"], dry_run_override=DRY_RUN, check=True)
+ run_command(
+ ["git", "checkout", "-b", f"v{version_branch}-test"], dry_run_override=DRY_RUN, check=True
+ )
+ console_print(f"Created branch: v{version_branch}-test")
+
+
+def update_default_branch(version_branch):
+ if confirm_action("Update default branches?"):
+ console_print()
+ console_print("You need to update the default branch at:")
+ console_print("./dev/breeze/src/airflow_breeze/branch_defaults.py")
+ console_print("Change the following:")
+ console_print("AIRFLOW_BRANCH = 'main'")
+ console_print("DEFAULT_AIRFLOW_CONSTRAINTS_BRANCH = 'constraints-main'")
+ console_print()
+ console_print("To:")
+ console_print()
+ console_print(f"AIRFLOW_BRANCH = 'v{version_branch}-test'")
+ console_print(f"DEFAULT_AIRFLOW_CONSTRAINTS_BRANCH = 'constraints-{version_branch}'")
+
+
+def commit_changes(version_branch):
+ if confirm_action("Commit the above changes?"):
+ run_command(["git", "add", "-p", "."], dry_run_override=DRY_RUN, check=True)
+ run_command(
+ ["git", "commit", "-m", f"Update default branches for {version_branch}"],
+ dry_run_override=DRY_RUN,
+ check=True,
+ )
+
+
+def create_stable_branch(version_branch):
+ if confirm_action(f"Create stable branch: v{version_branch}-stable?"):
+ run_command(
+ ["git", "checkout", "-b", f"v{version_branch}-stable"], dry_run_override=DRY_RUN, check=True
+ )
+ console_print(f"Created branch: v{version_branch}-stable")
+ else:
+ run_command(["git", "checkout", f"v{version_branch}-stable"], check=True)
+
+
+def push_test_and_stable_branch(version_branch):
+ if confirm_action("Push test and stable branches?"):
+ run_command(["git", "checkout", f"v{version_branch}-test"], dry_run_override=DRY_RUN, check=True)
+ run_command(
+ ["git", "push", "--set-upstream", "origin", f"v{version_branch}-test"],
+ dry_run_override=DRY_RUN,
+ check=True,
+ )
+ run_command(["git", "checkout", f"v{version_branch}-stable"], dry_run_override=DRY_RUN, check=True)
+ run_command(
+ ["git", "push", "--set-upstream", "origin", f"v{version_branch}-stable"],
+ dry_run_override=DRY_RUN,
+ check=True,
+ )
+
+
+def checkout_main():
+ if confirm_action("We now need to checkout main. Continue?"):
+ run_command(["git", "checkout", "main"], dry_run_override=DRY_RUN, check=True)
+ run_command(["git", "pull"])
+
+
+def instruction_update_version_branch(version_branch):
+ if confirm_action("Now, we need to manually update the version branches in main. Continue?"):
+ console_print()
+ console_print(
+ f"Add v{version_branch}-stable and v{version_branch}-test branches "
+ "in codecov.yml (there are 2 places in the file!)"
+ )
+ console_print("Areas to add the branches will look like this:")
+ console_print(
+ """
+ branches:
+ - main
+ - v2-0-stable
+ - v2-0-test
+ - v2-1-stable
+ - v2-1-test
+ - v2-2-stable
+ - v2-2-test
+ """
+ )
+ console_print()
+ console_print(f"Add v{version_branch}-stable to .asf.yaml ({version_branch} is your new branch)")
+ console_print(
+ f"""
+ protected_branches:
+ main:
+ required_pull_request_reviews:
+ required_approving_review_count: 1
+ ...
+ v{version_branch}-stable:
+ required_pull_request_reviews:
+ required_approving_review_count: 1
+ """
+ )
+ console_print("Once you finish with the above. Commit the changes and make a PR against main")
+ confirm_action("I'm done with the changes. Continue?", abort=True)
+
+
+def create_constraints(version_branch):
+ if confirm_action("Do you want to create branches from the constraints main?"):
+ run_command(["git", "checkout", "constraints-main"], dry_run_override=DRY_RUN, check=True)
+ run_command(
+ ["git", "checkout", "-b", f"constraints-{version_branch}"], dry_run_override=DRY_RUN, check=True
+ )
+ if confirm_action("Push the new branch?"):
+ run_command(
+ ["git", "push", "--set-upstream", "origin", f"constraints-{version_branch}"],
+ dry_run_override=DRY_RUN,
+ check=True,
+ )
+
+
+@click.command(
+ name="create-minor-branch",
+ help="Create a new version branch and update the default branches in main",
+ hidden=True,
+)
+@click.option("--version-branch", help="The version branch you want to create e.g 2-4", required=True)
+@option_answer
+def create_minor_version_branch(version_branch):
+ for obj in version_branch.split("-"):
+ assert isinstance(int(obj), int)
+ os.chdir(AIRFLOW_SOURCES_ROOT)
+ repo_root = os.getcwd()
+ console_print()
+ console_print(f"Repo root: {repo_root}")
+ console_print(f"Version branch: {version_branch}")
+ console_print("Below are your git remotes. We will push to origin:")
+ run_command(["git", "remote", "-v"])
+ console_print()
+ confirm_action("Verify that the above information is correct. Do you want to continue?", abort=True)
+ # Final confirmation
+ confirm_action("Pushes will be made to origin. Do you want to continue?", abort=True)
+ # Create a new branch from main
+ create_branch(version_branch)
+ # Build ci image
+ if confirm_action("Build latest breeze image?"):
+ run_command(["breeze", "ci-image", "build", "--python", "3.7"], dry_run_override=DRY_RUN, check=True)
+ # Update default branches
+ update_default_branch(version_branch)
+ # Commit changes
+ commit_changes(version_branch)
+ # Create stable branch
+ create_stable_branch(version_branch)
+ # Push test and stable branches
+ push_test_and_stable_branch(version_branch)
+ # Checkout main
+ checkout_main()
+ # Update version branches in main
+ instruction_update_version_branch(version_branch)
+ # Create constraints branch
+ create_constraints(version_branch)
+ console_print("Done!")
diff --git a/dev/breeze/src/airflow_breeze/commands/release_candidate_command.py b/dev/breeze/src/airflow_breeze/commands/release_candidate_command.py
new file mode 100644
index 0000000000..d1f0ed8fdd
--- /dev/null
+++ b/dev/breeze/src/airflow_breeze/commands/release_candidate_command.py
@@ -0,0 +1,346 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+
+import click
+
+from airflow_breeze.utils.common_options import option_answer
+from airflow_breeze.utils.confirm import confirm_action
+from airflow_breeze.utils.console import console_print
+from airflow_breeze.utils.path_utils import AIRFLOW_SOURCES_ROOT
+from airflow_breeze.utils.run_utils import run_command
+
+CI = os.environ.get("CI")
+DRY_RUN = True if CI else False
+
+
+def merge_pr(version_branch):
+ if confirm_action("Do you want to merge the Sync PR?"):
+ run_command(
+ [
+ "git",
+ "checkout",
+ f"v{version_branch}-stable",
+ ],
+ dry_run_override=DRY_RUN,
+ check=True,
+ )
+ run_command(
+ ["git", "reset", "--hard", f"origin/v{version_branch}-stable"],
+ dry_run_override=DRY_RUN,
+ check=True,
+ )
+ run_command(
+ ["git", "merge", "--ff-only", f"v{version_branch}-test"], dry_run_override=DRY_RUN, check=True
+ )
+ if confirm_action("Do you want to push the changes? Pushing the changes closes the PR"):
+ run_command(
+ ["git", "push", "origin", f"v{version_branch}-stable"], dry_run_override=DRY_RUN, check=True
+ )
+
+
+def git_tag(version):
+ if confirm_action(f"Tag {version}?"):
+ run_command(["git", "tag", "-s", f"{version}", "-m", f"Apache Airflow {version}"], check=True)
+ console_print("Tagged")
+
+
+def git_clean():
+ if confirm_action("Clean git repo?"):
+ run_command(["git", "clean", "-fxd"], dry_run_override=DRY_RUN, check=True)
+ console_print("Git repo cleaned")
+
+
+def tarball_release(version, version_without_rc):
+ if confirm_action("Create tarball?"):
+ run_command(["rm", "-rf", "dist"], check=True)
+
+ run_command(["mkdir", "dist"], check=True)
+ run_command(
+ [
+ "git",
+ "archive",
+ "--format=tar.gz",
+ f"{version}",
+ f"--prefix=apache-airflow-{version_without_rc}/",
+ "-o",
+ f"dist/apache-airflow-{version_without_rc}-source.tar.gz",
+ ],
+ check=True,
+ )
+ console_print("Tarball created")
+
+
+def create_artifacts_with_sdist():
+ run_command(["python3", "setup.py", "compile_assets", "sdist", "bdist_wheel"], check=True)
+ console_print("Artifacts created")
+
+
+def create_artifacts_with_breeze():
+ run_command(
+ ["breeze", "release-management", "prepare-airflow-package", "--package-format", "both"], check=True
+ )
+ console_print("Artifacts created")
+
+
+def sign_the_release(repo_root):
+ if confirm_action("Do you want to sign the release?"):
+ os.chdir(repo_root)
+ run_command(["pushd", "dist"], dry_run_override=DRY_RUN, check=True)
+ run_command(["./dev/sign.sh", "*"], dry_run_override=DRY_RUN, check=True)
+ run_command(["popd"], dry_run_override=DRY_RUN, check=True)
+ console_print("Release signed")
+
+
+def tag_and_push_constraints(version, version_branch):
+ if confirm_action("Do you want to tag and push constraints?"):
+ run_command(
+ ["git", "checkout", f"origin/constraints-{version_branch}"], dry_run_override=DRY_RUN, check=True
+ )
+ run_command(
+ [
+ "git",
+ "tag",
+ "-s",
+ f"constraints-{version}",
+ "-m",
+ f"Constraints for Apache Airflow {version}",
+ ],
+ dry_run_override=DRY_RUN,
+ check=True,
+ )
+ run_command(
+ ["git", "push", "origin", "tag", f"constraints-{version}"], dry_run_override=DRY_RUN, check=True
+ )
+ console_print("Constraints tagged and pushed")
+
+
+def clone_asf_repo(version, repo_root):
+ if confirm_action("Do you want to clone asf repo?"):
+ os.chdir(repo_root)
+ run_command(
+ ["svn", "checkout", "--depth=immediates", "https://dist.apache.org/repos/dist", "asf-dist"],
+ check=True,
+ )
+ run_command(["svn", "update", "--set-depth=infinity", "asf-dist/dev/airflow"], check=True)
+ console_print("Cloned ASF repo successfully")
+
+
+def move_artifacts_to_svn(version, repo_root):
+ if confirm_action("Do you want to move artifacts to SVN?"):
+ os.chdir(f"{repo_root}/asf-dist/dev/airflow")
+ run_command(["svn", "mkdir", f"{version}"], dry_run_override=DRY_RUN, check=True)
+ run_command(["mv", f"{repo_root}/dist/*", f"{version}/"], dry_run_override=DRY_RUN, check=True)
+ console_print("Moved artifacts to SVN:")
+ run_command(["ls"], dry_run_override=DRY_RUN)
+
+
+def push_artifacts_to_asf_repo(version, repo_root):
+ if confirm_action("Do you want to push artifacts to ASF repo?"):
+ console_print("Files to push to svn:")
+ if not DRY_RUN:
+ os.chdir(f"{repo_root}/asf-dist/dev/airflow/{version}")
+ run_command(["ls"], dry_run_override=DRY_RUN)
+ confirm_action("Do you want to continue?", abort=True)
+ run_command(["svn", "add", "*"], dry_run_override=DRY_RUN, check=True)
+ run_command(
+ ["svn", "commit", "-m", f"Add artifacts for Airflow {version}"],
+ dry_run_override=DRY_RUN,
+ check=True,
+ )
+ console_print("Files pushed to svn")
+ os.chdir(repo_root)
+ run_command(["rm", "-rf", "asf-dist"], dry_run_override=DRY_RUN, check=True)
+
+
+def prepare_pypi_packages(version, version_suffix, repo_root):
+ if confirm_action("Prepare pypi packages?"):
+ console_print("Preparing PyPI packages")
+ os.chdir(repo_root)
+ run_command(["git", "checkout", f"{version}"], dry_run_override=DRY_RUN, check=True)
+ run_command(
+ [
+ "breeze",
+ "release-management",
+ "prepare-airflow-package",
+ "--version-suffix-for-pypi",
+ f"{version_suffix}",
+ "--package-format",
+ "both",
+ ],
+ check=True,
+ )
+ run_command(["twine", "check", "dist/*"], check=True)
+ console_print("PyPI packages prepared")
+
+
+def push_packages_to_test_pypi():
+ if confirm_action("Do you want to push packages to test PyPI?"):
+ run_command(["twine", "upload", "-r", "pypitest", "dist/*"], dry_run_override=DRY_RUN, check=True)
+ console_print("Packages pushed to test PyPI")
+ console_print(
+ "Verify that the test package looks good by downloading it and installing it into a virtual "
+ "environment. The package download link is available at: "
+ "https://test.pypi.org/project/apache-airflow/#files"
+ )
+
+
+def push_packages_to_pypi():
+ if confirm_action("Do you want to push packages to production PyPI?"):
+ confirm_action(
+ "I have tested the package I uploaded to test PyPI. "
+ "I installed and ran a DAG with it and there's no issue. Do you agree to the above?",
+ abort=True,
+ )
+ run_command(["twine", "upload", "-r", "pypi", "dist/*"], dry_run_override=DRY_RUN, check=True)
+ console_print("Packages pushed to production PyPI")
+ console_print(
+ "Again, confirm that the package is available here: https://pypi.python.org/pypi/apache-airflow"
+ )
+ console_print(
+ """
+ It is important to stress that this snapshot should not be named "release", and it
+ is not supposed to be used by and advertised to the end-users who do not read the devlist.
+ """
+ )
+
+
+def push_release_candidate_to_github(version):
+ if confirm_action("Do you want to push release candidate to GitHub?"):
+ console_print(
+ """
+ This step should only be done now and not before, because it triggers an automated
+ build of the production docker image, using the packages that are currently released
+ in PyPI (both airflow and latest provider packages).
+ """
+ )
+ confirm_action(f"Confirm that {version} is pushed to PyPI(not PyPI test). Is it pushed?", abort=True)
+ run_command(["git", "push", "origin", "tag", f"{version}"], dry_run_override=DRY_RUN, check=True)
+ console_print("Release candidate pushed to GitHub")
+
+
+def create_issue_for_testing(version, previous_version, github_token):
+ if confirm_action("Do you want to create issue for testing? Only applicable for patch release"):
+ console_print()
+ console_print("Create issue in github for testing the release using this subject:")
+ console_print()
+ console_print(f"Status of testing of Apache Airflow {version}")
+ console_print()
+ if CI:
+ run_command(["git", "fetch"], check=True)
+ if confirm_action("Print the issue body?"):
+ run_command(
+ [
+ "./dev/prepare_release_issue.py",
+ "generate-issue-content",
+ "--previous-release",
+ f"{previous_version}",
+ "--current-release",
+ f"{version}",
+ "--github-token",
+ f"{github_token}",
+ ],
+ check=True,
+ )
+
+
+@click.command(
+ name="start-rc-process",
+ short_help="Start RC process",
+ help="Start the process for releasing a new RC.",
+ hidden=True,
+)
+@click.option("--version", required=True, help="The release candidate version e.g. 2.4.3rc1")
+@click.option("--previous-version", required=True, help="Previous version released e.g. 2.4.2")
+@click.option(
+ "--github-token", help="GitHub token to use in generating issue for testing of release candidate"
+)
+@option_answer
+def publish_release_candidate(version, previous_version, github_token):
+ if "rc" not in version:
+ exit("Version must contain 'rc'")
+ if "rc" in previous_version:
+ exit("Previous version must not contain 'rc'")
+ if not github_token:
+ github_token = os.environ.get("GITHUB_TOKEN")
+ if not github_token:
+ console_print("GITHUB_TOKEN is not set! Issue generation will fail.")
+ confirm_action("Do you want to continue?", abort=True)
+ version_suffix = version[-3:]
+ version_branch = version[:3].replace(".", "-")
+ version_without_rc = version[:-3]
+ os.chdir(AIRFLOW_SOURCES_ROOT)
+ airflow_repo_root = os.getcwd()
+
+ # List the above variables and ask for confirmation
+ console_print()
+ console_print(f"Previous version: {previous_version}")
+ console_print(f"version: {version}")
+ console_print(f"version_suffix: {version_suffix}")
+ console_print(f"version_branch: {version_branch}")
+ console_print(f"version_without_rc: {version_without_rc}")
+ console_print(f"airflow_repo_root: {airflow_repo_root}")
+ console_print()
+ console_print("Below are your git remotes. We will push to origin:")
+ run_command(["git", "remote", "-v"], dry_run_override=DRY_RUN)
+ console_print()
+ confirm_action("Verify that the above information is correct. Do you want to continue?", abort=True)
+ # Final confirmation
+ confirm_action("Pushes will be made to origin. Do you want to continue?", abort=True)
+ # Merge the sync PR
+ merge_pr(version_branch)
+
+ # Tag & clean the repo
+ git_tag(version)
+ git_clean()
+ # Build the latest image
+ if confirm_action("Build latest breeze image?"):
+ run_command(["breeze", "ci-image", "build", "--python", "3.7"], dry_run_override=DRY_RUN, check=True)
+ # Create the tarball
+ tarball_release(version, version_without_rc)
+ # Create the artifacts
+ if confirm_action("Use breeze to create artifacts?"):
+ create_artifacts_with_breeze()
+ elif confirm_action("Use setup.py to create artifacts?"):
+ create_artifacts_with_sdist()
+ # Sign the release
+ sign_the_release(airflow_repo_root)
+ # Tag and push constraints
+ tag_and_push_constraints(version, version_branch)
+ # Clone the asf repo
+ clone_asf_repo(version, airflow_repo_root)
+ # Move artifacts to SVN
+ move_artifacts_to_svn(version, airflow_repo_root)
+ # Push the artifacts to the asf repo
+ push_artifacts_to_asf_repo(version, airflow_repo_root)
+ # Prepare the pypi packages
+ prepare_pypi_packages(version, version_suffix, airflow_repo_root)
+ # Push the packages to test pypi
+ push_packages_to_test_pypi()
+
+ # Push the packages to pypi
+ push_packages_to_pypi()
+ # Push the release candidate to gitHub
+
+ push_release_candidate_to_github(version)
+ # Create issue for testing
+ os.chdir(airflow_repo_root)
+ create_issue_for_testing(version, previous_version, github_token)
+ console_print()
+ console_print("Done!")
diff --git a/dev/breeze/src/airflow_breeze/commands/release_command.py b/dev/breeze/src/airflow_breeze/commands/release_command.py
new file mode 100644
index 0000000000..73af6cae38
--- /dev/null
+++ b/dev/breeze/src/airflow_breeze/commands/release_command.py
@@ -0,0 +1,291 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+
+import click
+
+from airflow_breeze.utils.common_options import option_answer
+from airflow_breeze.utils.confirm import confirm_action
+from airflow_breeze.utils.console import console_print
+from airflow_breeze.utils.path_utils import AIRFLOW_SOURCES_ROOT
+from airflow_breeze.utils.run_utils import run_command
+
+CI = os.environ.get("CI")
+DRY_RUN = True if CI else False
+
+
+def clone_asf_repo(working_dir):
+ if confirm_action("Clone ASF repo?"):
+ run_command(["rm", "-rf", f"{working_dir}/asf-dist"], check=True)
+ run_command(
+ ["svn", "checkout", "--depth=immediates", "https://dist.apache.org/repos/dist", "asf-dist"],
+ check=True,
+ )
+ dev_dir = f"{working_dir}/asf-dist/dev/airflow"
+ release_dir = f"{working_dir}/asf-dist/release/airflow"
+ run_command(["svn", "update", "--set-depth", "infinity", dev_dir], check=True)
+ run_command(["svn", "update", "--set-depth", "infinity", release_dir], check=True)
+
+
+def create_version_dir(version):
+ if confirm_action(f"Create SVN version directory for {version}?"):
+ run_command(["svn", "mkdir", f"{version}"], dry_run_override=DRY_RUN, check=True)
+ console_print(f"{version} directory created")
+
+
+def copy_artifacts_to_svn(rc, svn_dev_repo):
+ if confirm_action(f"Copy artifacts to SVN for {rc}?"):
+ run_command(
+ [
+ "for",
+ "f",
+ "in",
+ f"{svn_dev_repo}/{rc}/*",
+ ";",
+ "do",
+ "svn",
+ "cp",
+ "$f",
+ "${$(basename $f)/}",
+ "done",
+ ],
+ dry_run_override=DRY_RUN,
+ check=True,
+ )
+ console_print("Artifacts copied to SVN:")
+ run_command(["ls"], dry_run_override=DRY_RUN)
+
+
+def commit_release(version, rc, svn_release_version_dir):
+ if confirm_action(f"Commit release {version} to SVN?"):
+ run_command(
+ ["svn", "commit", "-m", f"Release Airflow {version} from {rc}"],
+ dry_run_override=DRY_RUN,
+ check=True,
+ )
+
+
+def remove_old_release(previous_release):
+ if confirm_action(f"Remove old release {previous_release}?"):
+ run_command(["svn", "rm", f"{previous_release}"], dry_run_override=DRY_RUN, check=True)
+ run_command(
+ ["svn", "commit", "-m", f"Remove old release: {previous_release}"],
+ dry_run_override=DRY_RUN,
+ check=True,
+ )
+ confirm_action(
+ "Verify that the packages appear in "
+ "[airflow](https://dist.apache.org/repos/dist/release/airflow/). Continue?",
+ abort=True,
+ )
+
+
+def verify_pypi_package(version):
+ if confirm_action("Verify PyPI package?"):
+ run_command(["twine", "check", "*.whl", f"*{version}.tar.gz"], check=True)
+
+
+def upload_to_pypi_test(version):
+ if confirm_action("Upload to PyPI test?"):
+ run_command(
+ ["twine", "upload", "-r", "pypitest", "*.whl", f"*{version}.tar.gz"],
+ dry_run_override=DRY_RUN,
+ check=True,
+ )
+ console_print("Packages pushed to test PyPI")
+ console_print(
+ "Verify that the test package looks good by downloading it and installing it into a virtual "
+ "environment. The package download link is available at: "
+ "https://test.pypi.org/project/apache-airflow/#files"
+ )
+
+
+def upload_to_pypi(version):
+ if confirm_action("Upload to PyPI?"):
+ confirm_action(
+ "I have tested the package I uploaded to test PyPI. "
+ "I installed and ran a DAG with it and there's no issue. Do you agree to the above?",
+ abort=True,
+ )
+ run_command(
+ ["twine", "upload", "-r", "pypi", "*.whl", f"*{version}.tar.gz"],
+ dry_run_override=DRY_RUN,
+ check=True,
+ )
+ console_print("Packages pushed to production PyPI")
+ console_print(
+ "Again, confirm that the package is available here: https://pypi.python.org/pypi/apache-airflow"
+ )
+
+
+def retag_constraints(release_candidate, version):
+ if confirm_action(f"Retag constraints for {release_candidate} as {version}?"):
+ run_command(
+ ["git", "checkout", f"constraints-{release_candidate}"], dry_run_override=DRY_RUN, check=True
+ )
+ run_command(
+ [
+ "git",
+ "tag",
+ "-s",
+ f"constraints-{version}",
+ "-m",
+ f"Constraints for Apache Airflow {version}",
+ ],
+ dry_run_override=DRY_RUN,
+ check=True,
+ )
+ if confirm_action("Push latest constraints tag to GitHub?"):
+ run_command(
+ ["git", "push", "origin", "tag", f"constraints-{version}"], dry_run_override=DRY_RUN, check=True
+ )
+
+
+def tag_and_push_latest_constraint(version):
+ console_print("In case you release 'latest stable' version, also update 'latest' constraints")
+ if confirm_action("Tag latest constraint?"):
+ run_command(
+ [
+ "git",
+ "tag",
+ "-f",
+ "-s",
+ "constraints-latest",
+ "-m",
+ f"Latest constraints set to Apache Airflow {version}",
+ ],
+ dry_run_override=DRY_RUN,
+ check=True,
+ )
+ if confirm_action("Push latest constraints tag to GitHub?"):
+ run_command(
+ ["git", "push", "origin", "tag", "constraints-latest"], dry_run_override=DRY_RUN, check=True
+ )
+
+
+def push_tag_for_final_version(version, release_candidate):
+
+ if confirm_action(f"Push tag for final version {version}?"):
+ console_print(
+ """
+ This step should only be done now and not before, because it triggers an automated
+ build of the production docker image, using the packages that are currently released
+ in PyPI (both airflow and latest provider packages).
+ """
+ )
+ confirm_action(f"Confirm that {version} is pushed to PyPI(not PyPI test). Is it pushed?", abort=True)
+
+ run_command(["git", "checkout", f"{release_candidate}"], dry_run_override=DRY_RUN, check=True)
+ run_command(
+ ["git", "tag", "-s", f"{version}", "-m", f"Apache Airflow {version}"],
+ dry_run_override=DRY_RUN,
+ check=True,
+ )
+ run_command(["git", "push", "origin", "tag", f"{version}"], dry_run_override=DRY_RUN, check=True)
+
+
+@click.command(
+ name="start-release",
+ short_help="Start Airflow release process",
+ help="Start the process of releasing an Airflow version. "
+ "This command will guide you through the release process. ",
+ hidden=True,
+)
+@click.option("--release-candidate", required=True)
+@click.option("--previous-release", required=True)
+@option_answer
+def airflow_release(release_candidate, previous_release):
+ if "rc" not in release_candidate:
+ exit("Release candidate must contain 'rc'")
+ if "rc" in previous_release:
+ exit("Previous release must not contain 'rc'")
+ version = release_candidate[:-3]
+ os.chdir(AIRFLOW_SOURCES_ROOT)
+ airflow_repo_root = os.getcwd()
+ console_print()
+ console_print("Release candidate:", release_candidate)
+ console_print("Release Version:", version)
+ console_print("Previous release:", previous_release)
+ console_print("Airflow repo root:", airflow_repo_root)
+ console_print()
+ console_print("Below are your git remotes. We will push to origin:")
+ run_command(["git", "remote", "-v"], check=True)
+ console_print()
+ confirm_action("Verify that the above information is correct. Do you want to continue?", abort=True)
+ # Final confirmation
+ confirm_action("Pushes will be made to origin. Do you want to continue?", abort=True)
+
+ # Clone the asf repo
+ os.chdir("..")
+ working_dir = os.getcwd()
+ clone_asf_repo(working_dir)
+ svn_dev_repo = f"{working_dir}/asf-dist/dev/airflow"
+ svn_release_repo = f"{working_dir}/asf-dist/release/airflow"
+ console_print("SVN dev repo root:", svn_dev_repo)
+ console_print("SVN release repo root:", svn_release_repo)
+
+ # Create the version directory
+ confirm_action("Confirm that the above repo exists. Continue?", abort=True)
+
+ # Change to the svn release repo
+ os.chdir(svn_release_repo)
+
+ # Create the version directory
+ create_version_dir(version)
+ svn_release_version_dir = f"{svn_release_repo}/{version}"
+ console_print("SVN Release version dir:", svn_release_version_dir)
+
+ # Change directory to the version directory
+ if os.path.exists(svn_release_version_dir):
+ os.chdir(svn_release_version_dir)
+ else:
+ confirm_action("Version directory does not exist. Do you want to Continue?", abort=True)
+
+ # Copy artifacts to the version directory
+ copy_artifacts_to_svn(release_candidate, svn_dev_repo)
+
+ # Commit the release to svn
+ commit_release(version, release_candidate, svn_release_version_dir)
+
+ # Remove old release
+ if os.path.exists(svn_release_version_dir):
+ os.chdir("..")
+ remove_old_release(previous_release)
+
+ # Verify pypi package
+ if os.path.exists(svn_release_version_dir):
+ os.chdir(svn_release_version_dir)
+ verify_pypi_package(version)
+
+ # Upload to pypi test
+ upload_to_pypi_test(version)
+ # Upload to pypi
+ upload_to_pypi(version)
+
+ # Change Directory to airflow
+ os.chdir(airflow_repo_root)
+
+ # Retag and push the constraint file
+ retag_constraints(release_candidate, version)
+ tag_and_push_latest_constraint(version)
+
+ # Push tag for final version
+ push_tag_for_final_version(version, release_candidate)
+
+ console_print("Done!")
diff --git a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
index 0534ea8cf1..287ab93a43 100644
--- a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
@@ -34,6 +34,9 @@ from rich.progress import Progress
from rich.syntax import Syntax
from airflow_breeze.commands.ci_image_commands import rebuild_or_pull_ci_image_if_needed
+from airflow_breeze.commands.minor_release_command import create_minor_version_branch
+from airflow_breeze.commands.release_candidate_command import publish_release_candidate
+from airflow_breeze.commands.release_command import airflow_release
from airflow_breeze.global_constants import (
ALLOWED_PLATFORMS,
APACHE_AIRFLOW_GITHUB_REPOSITORY,
@@ -887,3 +890,9 @@ def generate_issue_content(
users.add("@" + pr.user.login)
get_console().print("All users involved in the PRs:")
get_console().print(" ".join(users))
+
+
+# AIRFLOW RELEASE COMMANDS
+release_management.add_command(publish_release_candidate)
+release_management.add_command(airflow_release)
+release_management.add_command(create_minor_version_branch)
diff --git a/dev/breeze/src/airflow_breeze/utils/confirm.py b/dev/breeze/src/airflow_breeze/utils/confirm.py
index 1bb95d8080..0b64697c19 100644
--- a/dev/breeze/src/airflow_breeze/utils/confirm.py
+++ b/dev/breeze/src/airflow_breeze/utils/confirm.py
@@ -98,3 +98,20 @@ def user_confirm(
if quit_allowed:
return Answer.QUIT
sys.exit(1)
+
+
+def confirm_action(
+ message: str,
+ timeout: float | None = None,
+ default_answer: Answer | None = Answer.NO,
+ quit_allowed: bool = True,
+ abort: bool = False,
+) -> bool:
+ answer = user_confirm(message, timeout, default_answer, quit_allowed)
+ if answer == Answer.YES:
+ return True
+ elif abort:
+ sys.exit(1)
+ elif answer == Answer.QUIT:
+ sys.exit(1)
+ return False
diff --git a/dev/breeze/src/airflow_breeze/utils/console.py b/dev/breeze/src/airflow_breeze/utils/console.py
index 72d3cd2f30..cf357e48b8 100644
--- a/dev/breeze/src/airflow_breeze/utils/console.py
+++ b/dev/breeze/src/airflow_breeze/utils/console.py
@@ -105,3 +105,7 @@ def get_stderr_console(output: Output | None = None) -> Console:
theme=get_theme(),
record=True if recording_file else False,
)
+
+
+def console_print(*message) -> None:
+ return get_console().print(*message)
diff --git a/images/breeze/output-commands-hash.txt b/images/breeze/output-commands-hash.txt
index 219c440000..5fa6522464 100644
--- a/images/breeze/output-commands-hash.txt
+++ b/images/breeze/output-commands-hash.txt
@@ -34,21 +34,25 @@ k8s:456db86a3571397c44f9647d3c06864a
prod-image:build:9cfa194bc24c8636416db1f21b086b1e
prod-image:pull:e3c89dd908fc44adf6e159c2950ebdd0
prod-image:verify:31bc5efada1d70a0a31990025db1a093
-prod-image:a4013428dc7f71a1defc3778d2efe3dc
+prod-image:4f98deab35e53ebddbdc9950a50555a4
+release-management:create-minor-branch:6a01066dce15e09fb269a8385626657c
release-management:generate-constraints:ae30d6ad49a1b2c15b61cb29080fd957
release-management:generate-issue-content:24218438f9e85e7c92258aadebbb19de
release-management:prepare-airflow-package:3ac14ea6d2b09614959c0ec4fd564789
release-management:prepare-provider-documentation:3fe5ead9887c518d1b397d1103dc0025
release-management:prepare-provider-packages:40144cb01afc56f6a4f92d9e117e546e
release-management:release-prod-images:c9bc40938e0efad49e51ef66e83f9527
+release-management:start-rc-process:6aafbaceabd7b67b9a1af4c2f59abc4c
+release-management:start-release:acb384d86e02ff5fde1bf971897be17c
release-management:verify-provider-packages:88bd609aff6d09d52ab8d80d6e055e7b
-release-management:eafd13da512a5a7c0fb1499cf7ff1d63
+release-management:ce37a79b83b964ecf575b5aef0515462
setup:autocomplete:03343478bf1d0cf9c101d454cdb63b68
+setup:check-all-params-in-groups:5b3dc1f6e630510c8f52c3d0687b3b2a
setup:config:3ffcd35dd24b486ddf1d08b797e3d017
-setup:regenerate-command-images:ab2d83c339fa3a42b0c819b6b6cc88ae
+setup:regenerate-command-images:10d5d83f294cd86245cd51d4185504bb
setup:self-upgrade:d02f70c7a230eae3463ceec2056b63fa
setup:version:123b462a421884dc2320ffc5e54b2478
-setup:fbabee281b69f818091d780b24bd815a
+setup:f383b9236f6141f95276136ccd9217f5
shell:76e0f530b7af514a2aad3032b6516c46
start-airflow:06d4aeb5f1b65f6b975f3f915558d0b3
static-checks:f45ad432bdc47a2256fdb0277b19d816
diff --git a/images/breeze/output-commands.svg b/images/breeze/output-commands.svg
index c8763961ad..22001729e6 100644
--- a/images/breeze/output-commands.svg
+++ b/images/breeze/output-commands.svg
@@ -35,8 +35,8 @@
.breeze-help-r1 { fill: #c5c8c6;font-weight: bold }
.breeze-help-r2 { fill: #c5c8c6 }
.breeze-help-r3 { fill: #d0b344;font-weight: bold }
-.breeze-help-r4 { fill: #868887 }
-.breeze-help-r5 { fill: #68a0b3;font-weight: bold }
+.breeze-help-r4 { fill: #68a0b3;font-weight: bold }
+.breeze-help-r5 { fill: #868887 }
.breeze-help-r6 { fill: #98a84b;font-weight: bold }
.breeze-help-r7 { fill: #8d7b39 }
</style>
@@ -187,49 +187,49 @@
<g class="breeze-help-matrix">
<text class="breeze-help-r2" x="1464" y="20" textLength="12.2" clip-path="url(#breeze-help-line-0)">
-</text><text class="breeze-help-r3" x="12.2" y="44.4" textLength="85.4" clip-path="url(#breeze-help-line-1)">Usage: </text><text class="breeze-help-r1" x="97.6" y="44.4" textLength="414.8" clip-path="url(#breeze-help-line-1)">breeze [OPTIONS] COMMAND [ARGS]...</text><text class="breeze-help-r2" x="1464" y="44.4" textLength="12.2" clip-path="url(#breeze-help-line-1)">
+</text><text class="breeze-help-r3" x="12.2" y="44.4" textLength="85.4" clip-path="url(#breeze-help-line-1)">Usage: </text><text class="breeze-help-r1" x="97.6" y="44.4" textLength="97.6" clip-path="url(#breeze-help-line-1)">breeze [</text><text class="breeze-help-r4" x="195.2" y="44.4" textLength="85.4" clip-path="url(#breeze-help-line-1)">OPTIONS</text><text class="breeze-help-r1" x="280.6" y="44.4" textLength="24.4" clip-path="url(#breeze-help-line-1)">] </text><text cl [...]
</text><text class="breeze-help-r2" x="1464" y="68.8" textLength="12.2" clip-path="url(#breeze-help-line-2)">
-</text><text class="breeze-help-r4" x="0" y="93.2" textLength="24.4" clip-path="url(#breeze-help-line-3)">╭─</text><text class="breeze-help-r4" x="24.4" y="93.2" textLength="158.6" clip-path="url(#breeze-help-line-3)"> Basic flags </text><text class="breeze-help-r4" x="183" y="93.2" textLength="1256.6" clip-path="url(#breeze-help-line-3)">───────────────────────────────────────────────────────────────────────────────────────────────────────</text><text class="breeze-help-r [...]
-</text><text class="breeze-help-r4" x="0" y="117.6" textLength="12.2" clip-path="url(#breeze-help-line-4)">│</text><text class="breeze-help-r5" x="24.4" y="117.6" textLength="12.2" clip-path="url(#breeze-help-line-4)">-</text><text class="breeze-help-r5" x="36.6" y="117.6" textLength="85.4" clip-path="url(#breeze-help-line-4)">-python</text><text class="breeze-help-r6" x="305" y="117.6" textLength="24.4" clip-path="url(#breeze-help-line-4)">-p</text><text class="breeze-help-r2" x="353.8" [...]
-</text><text class="breeze-help-r4" x="0" y="142" textLength="12.2" clip-path="url(#breeze-help-line-5)">│</text><text class="breeze-help-r4" x="353.8" y="142" textLength="732" clip-path="url(#breeze-help-line-5)">[default: 3.7]                                           & [...]
-</text><text class="breeze-help-r4" x="0" y="166.4" textLength="12.2" clip-path="url(#breeze-help-line-6)">│</text><text class="breeze-help-r5" x="24.4" y="166.4" textLength="12.2" clip-path="url(#breeze-help-line-6)">-</text><text class="breeze-help-r5" x="36.6" y="166.4" textLength="97.6" clip-path="url(#breeze-help-line-6)">-backend</text><text class="breeze-help-r6" x="305" y="166.4" textLength="24.4" clip-path="url(#breeze-help-line-6)">-b</text><text class="breeze-help-r2" x="353.8 [...]
-</text><text class="breeze-help-r4" x="0" y="190.8" textLength="12.2" clip-path="url(#breeze-help-line-7)">│</text><text class="breeze-help-r5" x="24.4" y="190.8" textLength="12.2" clip-path="url(#breeze-help-line-7)">-</text><text class="breeze-help-r5" x="36.6" y="190.8" textLength="109.8" clip-path="url(#breeze-help-line-7)">-postgres</text><text class="breeze-help-r5" x="146.4" y="190.8" textLength="97.6" clip-path="url(#breeze-help-line-7)">-version</text><text class="breeze-help-r6 [...]
-</text><text class="breeze-help-r4" x="0" y="215.2" textLength="12.2" clip-path="url(#breeze-help-line-8)">│</text><text class="breeze-help-r5" x="24.4" y="215.2" textLength="12.2" clip-path="url(#breeze-help-line-8)">-</text><text class="breeze-help-r5" x="36.6" y="215.2" textLength="73.2" clip-path="url(#breeze-help-line-8)">-mysql</text><text class="breeze-help-r5" x="109.8" y="215.2" textLength="97.6" clip-path="url(#breeze-help-line-8)">-version</text><text class="breeze-help-r6" x= [...]
-</text><text class="breeze-help-r4" x="0" y="239.6" textLength="12.2" clip-path="url(#breeze-help-line-9)">│</text><text class="breeze-help-r5" x="24.4" y="239.6" textLength="12.2" clip-path="url(#breeze-help-line-9)">-</text><text class="breeze-help-r5" x="36.6" y="239.6" textLength="73.2" clip-path="url(#breeze-help-line-9)">-mssql</text><text class="breeze-help-r5" x="109.8" y="239.6" textLength="97.6" clip-path="url(#breeze-help-line-9)">-version</text><text class="breeze-help-r6" x= [...]
-</text><text class="breeze-help-r4" x="0" y="264" textLength="12.2" clip-path="url(#breeze-help-line-10)">│</text><text class="breeze-help-r5" x="24.4" y="264" textLength="12.2" clip-path="url(#breeze-help-line-10)">-</text><text class="breeze-help-r5" x="36.6" y="264" textLength="146.4" clip-path="url(#breeze-help-line-10)">-integration</text><text class="breeze-help-r2" x="353.8" y="264" textLength="744.2" clip-path="url(#breeze-help-line-10)">Integration(s) to enable wh [...]
-</text><text class="breeze-help-r4" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-help-line-11)">│</text><text class="breeze-help-r7" x="353.8" y="288.4" textLength="744.2" clip-path="url(#breeze-help-line-11)">(cassandra | kerberos | mongo | pinot | celery | trino | all)</text><text class="breeze-help-r4" x="1451.8" y="288.4" textLength="12.2" clip-path="url(#breeze-help-line-11)">│</text><text class="breeze-help-r2" [...]
-</text><text class="breeze-help-r4" x="0" y="312.8" textLength="12.2" clip-path="url(#breeze-help-line-12)">│</text><text class="breeze-help-r5" x="24.4" y="312.8" textLength="12.2" clip-path="url(#breeze-help-line-12)">-</text><text class="breeze-help-r5" x="36.6" y="312.8" textLength="97.6" clip-path="url(#breeze-help-line-12)">-forward</text><text class="breeze-help-r5" x="134.2" y="312.8" textLength="146.4" clip-path="url(#breeze-help-line-12)">-credentials</text><text class="breeze- [...]
-</text><text class="breeze-help-r4" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-help-line-13)">│</text><text class="breeze-help-r5" x="24.4" y="337.2" textLength="12.2" clip-path="url(#breeze-help-line-13)">-</text><text class="breeze-help-r5" x="36.6" y="337.2" textLength="36.6" clip-path="url(#breeze-help-line-13)">-db</text><text class="breeze-help-r5" x="73.2" y="337.2" textLength="73.2" clip-path="url(#breeze-help-line-13)">-reset</text><text class="breeze-help-r6" x="3 [...]
-</text><text class="breeze-help-r4" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-help-line-14)">│</text><text class="breeze-help-r5" x="24.4" y="361.6" textLength="12.2" clip-path="url(#breeze-help-line-14)">-</text><text class="breeze-help-r5" x="36.6" y="361.6" textLength="48.8" clip-path="url(#breeze-help-line-14)">-max</text><text class="breeze-help-r5" x="85.4" y="361.6" textLength="61" clip-path="url(#breeze-help-line-14)">-time</text><text class="breeze-help-r2" x="353 [...]
-</text><text class="breeze-help-r4" x="0" y="386" textLength="12.2" clip-path="url(#breeze-help-line-15)">│</text><text class="breeze-help-r7" x="353.8" y="386" textLength="1049.2" clip-path="url(#breeze-help-line-15)">(INTEGER RANGE)                                          & [...]
-</text><text class="breeze-help-r4" x="0" y="410.4" textLength="12.2" clip-path="url(#breeze-help-line-16)">│</text><text class="breeze-help-r5" x="24.4" y="410.4" textLength="12.2" clip-path="url(#breeze-help-line-16)">-</text><text class="breeze-help-r5" x="36.6" y="410.4" textLength="85.4" clip-path="url(#breeze-help-line-16)">-github</text><text class="breeze-help-r5" x="122" y="410.4" textLength="134.2" clip-path="url(#breeze-help-line-16)">-repository</text><text class="breeze-help [...]
-</text><text class="breeze-help-r4" x="0" y="434.8" textLength="1464" clip-path="url(#breeze-help-line-17)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-help-r2" x="1464" y="434.8" textLength="12.2" clip-path="url(#breeze-help-line-17)">
-</text><text class="breeze-help-r4" x="0" y="459.2" textLength="24.4" clip-path="url(#breeze-help-line-18)">╭─</text><text class="breeze-help-r4" x="24.4" y="459.2" textLength="195.2" clip-path="url(#breeze-help-line-18)"> Common options </text><text class="breeze-help-r4" x="219.6" y="459.2" textLength="1220" clip-path="url(#breeze-help-line-18)">────────────────────────────────────────────────────────────────────────────────────────────────────</text><text class="breeze- [...]
-</text><text class="breeze-help-r4" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-help-line-19)">│</text><text class="breeze-help-r5" x="24.4" y="483.6" textLength="12.2" clip-path="url(#breeze-help-line-19)">-</text><text class="breeze-help-r5" x="36.6" y="483.6" textLength="97.6" clip-path="url(#breeze-help-line-19)">-verbose</text><text class="breeze-help-r6" x="158.6" y="483.6" textLength="24.4" clip-path="url(#breeze-help-line-19)">-v</text><text class="breeze-help-r2" x= [...]
-</text><text class="breeze-help-r4" x="0" y="508" textLength="12.2" clip-path="url(#breeze-help-line-20)">│</text><text class="breeze-help-r5" x="24.4" y="508" textLength="12.2" clip-path="url(#breeze-help-line-20)">-</text><text class="breeze-help-r5" x="36.6" y="508" textLength="48.8" clip-path="url(#breeze-help-line-20)">-dry</text><text class="breeze-help-r5" x="85.4" y="508" textLength="48.8" clip-path="url(#breeze-help-line-20)">-run</text><text class="breeze-help-r6" x="158.6" y=" [...]
-</text><text class="breeze-help-r4" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-help-line-21)">│</text><text class="breeze-help-r5" x="24.4" y="532.4" textLength="12.2" clip-path="url(#breeze-help-line-21)">-</text><text class="breeze-help-r5" x="36.6" y="532.4" textLength="85.4" clip-path="url(#breeze-help-line-21)">-answer</text><text class="breeze-help-r6" x="158.6" y="532.4" textLength="24.4" clip-path="url(#breeze-help-line-21)">-a</text><text class="breeze-help-r2" x=" [...]
-</text><text class="breeze-help-r4" x="0" y="556.8" textLength="12.2" clip-path="url(#breeze-help-line-22)">│</text><text class="breeze-help-r5" x="24.4" y="556.8" textLength="12.2" clip-path="url(#breeze-help-line-22)">-</text><text class="breeze-help-r5" x="36.6" y="556.8" textLength="61" clip-path="url(#breeze-help-line-22)">-help</text><text class="breeze-help-r6" x="158.6" y="556.8" textLength="24.4" clip-path="url(#breeze-help-line-22)">-h</text><text class="breeze-help-r2" x="207. [...]
-</text><text class="breeze-help-r4" x="0" y="581.2" textLength="1464" clip-path="url(#breeze-help-line-23)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-help-r2" x="1464" y="581.2" textLength="12.2" clip-path="url(#breeze-help-line-23)">
-</text><text class="breeze-help-r4" x="0" y="605.6" textLength="24.4" clip-path="url(#breeze-help-line-24)">╭─</text><text class="breeze-help-r4" x="24.4" y="605.6" textLength="317.2" clip-path="url(#breeze-help-line-24)"> Basic developer commands </text><text class="breeze-help-r4" x="341.6" y="605.6" textLength="1098" clip-path="url(#breeze-help-line-24)">──────────────────────────────────────────────────────────────────────────────────────────</text><text class="br [...]
-</text><text class="breeze-help-r4" x="0" y="630" textLength="12.2" clip-path="url(#breeze-help-line-25)">│</text><text class="breeze-help-r5" x="24.4" y="630" textLength="219.6" clip-path="url(#breeze-help-line-25)">start-airflow     </text><text class="breeze-help-r2" x="268.4" y="630" textLength="1171.2" clip-path="url(#breeze-help-line-25)">Enter breeze environment and starts all Airflow components in the tmux [...]
-</text><text class="breeze-help-r4" x="0" y="654.4" textLength="12.2" clip-path="url(#breeze-help-line-26)">│</text><text class="breeze-help-r2" x="268.4" y="654.4" textLength="1171.2" clip-path="url(#breeze-help-line-26)">if contents of www directory changed.                                   [...]
-</text><text class="breeze-help-r4" x="0" y="678.8" textLength="12.2" clip-path="url(#breeze-help-line-27)">│</text><text class="breeze-help-r5" x="24.4" y="678.8" textLength="219.6" clip-path="url(#breeze-help-line-27)">static-checks     </text><text class="breeze-help-r2" x="268.4" y="678.8" textLength="1171.2" clip-path="url(#breeze-help-line-27)">Run static checks.              &# [...]
-</text><text class="breeze-help-r4" x="0" y="703.2" textLength="12.2" clip-path="url(#breeze-help-line-28)">│</text><text class="breeze-help-r5" x="24.4" y="703.2" textLength="219.6" clip-path="url(#breeze-help-line-28)">build-docs        </text><text class="breeze-help-r2" x="268.4" y="703.2" textLength="1171.2" clip-path="url(#breeze-help-line-28)">Build documentation in the container.        [...]
-</text><text class="breeze-help-r4" x="0" y="727.6" textLength="12.2" clip-path="url(#breeze-help-line-29)">│</text><text class="breeze-help-r5" x="24.4" y="727.6" textLength="219.6" clip-path="url(#breeze-help-line-29)">stop              </text><text class="breeze-help-r2" x="268.4" y="727.6" textLength="1171.2" clip-path="url(#breeze-help-line-29)">Stop running breeze environment.    [...]
-</text><text class="breeze-help-r4" x="0" y="752" textLength="12.2" clip-path="url(#breeze-help-line-30)">│</text><text class="breeze-help-r5" x="24.4" y="752" textLength="219.6" clip-path="url(#breeze-help-line-30)">shell             </text><text class="breeze-help-r2" x="268.4" y="752" textLength="1171.2" clip-path="url(#breeze-help-line-30)">Enter breeze environment. this is the default  [...]
-</text><text class="breeze-help-r4" x="0" y="776.4" textLength="12.2" clip-path="url(#breeze-help-line-31)">│</text><text class="breeze-help-r5" x="24.4" y="776.4" textLength="219.6" clip-path="url(#breeze-help-line-31)">exec              </text><text class="breeze-help-r2" x="268.4" y="776.4" textLength="1171.2" clip-path="url(#breeze-help-line-31)">Joins the interactive shell of running  [...]
-</text><text class="breeze-help-r4" x="0" y="800.8" textLength="12.2" clip-path="url(#breeze-help-line-32)">│</text><text class="breeze-help-r5" x="24.4" y="800.8" textLength="219.6" clip-path="url(#breeze-help-line-32)">compile-www-assets</text><text class="breeze-help-r2" x="268.4" y="800.8" textLength="1171.2" clip-path="url(#breeze-help-line-32)">Compiles www assets.                  & [...]
-</text><text class="breeze-help-r4" x="0" y="825.2" textLength="12.2" clip-path="url(#breeze-help-line-33)">│</text><text class="breeze-help-r5" x="24.4" y="825.2" textLength="219.6" clip-path="url(#breeze-help-line-33)">cleanup           </text><text class="breeze-help-r2" x="268.4" y="825.2" textLength="1171.2" clip-path="url(#breeze-help-line-33)">Cleans the cache of parameters, docker cache and& [...]
-</text><text class="breeze-help-r4" x="0" y="849.6" textLength="1464" clip-path="url(#breeze-help-line-34)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-help-r2" x="1464" y="849.6" textLength="12.2" clip-path="url(#breeze-help-line-34)">
-</text><text class="breeze-help-r4" x="0" y="874" textLength="24.4" clip-path="url(#breeze-help-line-35)">╭─</text><text class="breeze-help-r4" x="24.4" y="874" textLength="305" clip-path="url(#breeze-help-line-35)"> Advanced command groups </text><text class="breeze-help-r4" x="329.4" y="874" textLength="1110.2" clip-path="url(#breeze-help-line-35)">───────────────────────────────────────────────────────────────────────────────────────────</text><text class="breeze-h [...]
-</text><text class="breeze-help-r4" x="0" y="898.4" textLength="12.2" clip-path="url(#breeze-help-line-36)">│</text><text class="breeze-help-r5" x="24.4" y="898.4" textLength="280.6" clip-path="url(#breeze-help-line-36)">testing                </text><text class="breeze-help-r2" x="329.4" y="898.4" textLength="1110.2" clip-path="url(#breeze-help-line-36)">Tools that developers can use  [...]
-</text><text class="breeze-help-r4" x="0" y="922.8" textLength="12.2" clip-path="url(#breeze-help-line-37)">│</text><text class="breeze-help-r5" x="24.4" y="922.8" textLength="280.6" clip-path="url(#breeze-help-line-37)">ci-image               </text><text class="breeze-help-r2" x="329.4" y="922.8" textLength="1110.2" clip-path="url(#breeze-help-line-37)">Tools that developers can use to&# [...]
-</text><text class="breeze-help-r4" x="0" y="947.2" textLength="12.2" clip-path="url(#breeze-help-line-38)">│</text><text class="breeze-help-r5" x="24.4" y="947.2" textLength="280.6" clip-path="url(#breeze-help-line-38)">k8s                    </text><text class="breeze-help-r2" x="329.4" y="947.2" textLength="1110.2" clip-path="url(#breeze-help-line-38)">Tools that developers [...]
-</text><text class="breeze-help-r4" x="0" y="971.6" textLength="12.2" clip-path="url(#breeze-help-line-39)">│</text><text class="breeze-help-r5" x="24.4" y="971.6" textLength="280.6" clip-path="url(#breeze-help-line-39)">prod-image             </text><text class="breeze-help-r2" x="329.4" y="971.6" textLength="1110.2" clip-path="url(#breeze-help-line-39)">Tools that developers can use to manual [...]
-</text><text class="breeze-help-r4" x="0" y="996" textLength="12.2" clip-path="url(#breeze-help-line-40)">│</text><text class="breeze-help-r5" x="24.4" y="996" textLength="280.6" clip-path="url(#breeze-help-line-40)">setup                  </text><text class="breeze-help-r2" x="329.4" y="996" textLength="1110.2" clip-path="url(#breeze-help-line-40)">Tools that developers can use& [...]
-</text><text class="breeze-help-r4" x="0" y="1020.4" textLength="12.2" clip-path="url(#breeze-help-line-41)">│</text><text class="breeze-help-r5" x="24.4" y="1020.4" textLength="280.6" clip-path="url(#breeze-help-line-41)">release-management     </text><text class="breeze-help-r2" x="329.4" y="1020.4" textLength="1110.2" clip-path="url(#breeze-help-line-41)">Tools that release managers can use to prepare and manage [...]
-</text><text class="breeze-help-r4" x="0" y="1044.8" textLength="12.2" clip-path="url(#breeze-help-line-42)">│</text><text class="breeze-help-r5" x="24.4" y="1044.8" textLength="280.6" clip-path="url(#breeze-help-line-42)">ci                     </text><text class="breeze-help-r2" x="329.4" y="1044.8" textLength="1110.2" clip-path="url(#breeze-help-line-42)">Tools that CI [...]
-</text><text class="breeze-help-r4" x="0" y="1069.2" textLength="1464" clip-path="url(#breeze-help-line-43)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-help-r2" x="1464" y="1069.2" textLength="12.2" clip-path="url(#breeze-help-line-43)">
+</text><text class="breeze-help-r5" x="0" y="93.2" textLength="24.4" clip-path="url(#breeze-help-line-3)">╭─</text><text class="breeze-help-r5" x="24.4" y="93.2" textLength="158.6" clip-path="url(#breeze-help-line-3)"> Basic flags </text><text class="breeze-help-r5" x="183" y="93.2" textLength="1256.6" clip-path="url(#breeze-help-line-3)">───────────────────────────────────────────────────────────────────────────────────────────────────────</text><text class="breeze-help-r [...]
+</text><text class="breeze-help-r5" x="0" y="117.6" textLength="12.2" clip-path="url(#breeze-help-line-4)">│</text><text class="breeze-help-r4" x="24.4" y="117.6" textLength="12.2" clip-path="url(#breeze-help-line-4)">-</text><text class="breeze-help-r4" x="36.6" y="117.6" textLength="85.4" clip-path="url(#breeze-help-line-4)">-python</text><text class="breeze-help-r6" x="305" y="117.6" textLength="24.4" clip-path="url(#breeze-help-line-4)">-p</text><text class="breeze-help-r2" x="353.8" [...]
+</text><text class="breeze-help-r5" x="0" y="142" textLength="12.2" clip-path="url(#breeze-help-line-5)">│</text><text class="breeze-help-r5" x="353.8" y="142" textLength="732" clip-path="url(#breeze-help-line-5)">[default: 3.7]                                           & [...]
+</text><text class="breeze-help-r5" x="0" y="166.4" textLength="12.2" clip-path="url(#breeze-help-line-6)">│</text><text class="breeze-help-r4" x="24.4" y="166.4" textLength="12.2" clip-path="url(#breeze-help-line-6)">-</text><text class="breeze-help-r4" x="36.6" y="166.4" textLength="97.6" clip-path="url(#breeze-help-line-6)">-backend</text><text class="breeze-help-r6" x="305" y="166.4" textLength="24.4" clip-path="url(#breeze-help-line-6)">-b</text><text class="breeze-help-r2" x="353.8 [...]
+</text><text class="breeze-help-r5" x="0" y="190.8" textLength="12.2" clip-path="url(#breeze-help-line-7)">│</text><text class="breeze-help-r4" x="24.4" y="190.8" textLength="12.2" clip-path="url(#breeze-help-line-7)">-</text><text class="breeze-help-r4" x="36.6" y="190.8" textLength="109.8" clip-path="url(#breeze-help-line-7)">-postgres</text><text class="breeze-help-r4" x="146.4" y="190.8" textLength="97.6" clip-path="url(#breeze-help-line-7)">-version</text><text class="breeze-help-r6 [...]
+</text><text class="breeze-help-r5" x="0" y="215.2" textLength="12.2" clip-path="url(#breeze-help-line-8)">│</text><text class="breeze-help-r4" x="24.4" y="215.2" textLength="12.2" clip-path="url(#breeze-help-line-8)">-</text><text class="breeze-help-r4" x="36.6" y="215.2" textLength="73.2" clip-path="url(#breeze-help-line-8)">-mysql</text><text class="breeze-help-r4" x="109.8" y="215.2" textLength="97.6" clip-path="url(#breeze-help-line-8)">-version</text><text class="breeze-help-r6" x= [...]
+</text><text class="breeze-help-r5" x="0" y="239.6" textLength="12.2" clip-path="url(#breeze-help-line-9)">│</text><text class="breeze-help-r4" x="24.4" y="239.6" textLength="12.2" clip-path="url(#breeze-help-line-9)">-</text><text class="breeze-help-r4" x="36.6" y="239.6" textLength="73.2" clip-path="url(#breeze-help-line-9)">-mssql</text><text class="breeze-help-r4" x="109.8" y="239.6" textLength="97.6" clip-path="url(#breeze-help-line-9)">-version</text><text class="breeze-help-r6" x= [...]
+</text><text class="breeze-help-r5" x="0" y="264" textLength="12.2" clip-path="url(#breeze-help-line-10)">│</text><text class="breeze-help-r4" x="24.4" y="264" textLength="12.2" clip-path="url(#breeze-help-line-10)">-</text><text class="breeze-help-r4" x="36.6" y="264" textLength="146.4" clip-path="url(#breeze-help-line-10)">-integration</text><text class="breeze-help-r2" x="353.8" y="264" textLength="744.2" clip-path="url(#breeze-help-line-10)">Integration(s) to enable wh [...]
+</text><text class="breeze-help-r5" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-help-line-11)">│</text><text class="breeze-help-r7" x="353.8" y="288.4" textLength="744.2" clip-path="url(#breeze-help-line-11)">(cassandra | kerberos | mongo | pinot | celery | trino | all)</text><text class="breeze-help-r5" x="1451.8" y="288.4" textLength="12.2" clip-path="url(#breeze-help-line-11)">│</text><text class="breeze-help-r2" [...]
+</text><text class="breeze-help-r5" x="0" y="312.8" textLength="12.2" clip-path="url(#breeze-help-line-12)">│</text><text class="breeze-help-r4" x="24.4" y="312.8" textLength="12.2" clip-path="url(#breeze-help-line-12)">-</text><text class="breeze-help-r4" x="36.6" y="312.8" textLength="97.6" clip-path="url(#breeze-help-line-12)">-forward</text><text class="breeze-help-r4" x="134.2" y="312.8" textLength="146.4" clip-path="url(#breeze-help-line-12)">-credentials</text><text class="breeze- [...]
+</text><text class="breeze-help-r5" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-help-line-13)">│</text><text class="breeze-help-r4" x="24.4" y="337.2" textLength="12.2" clip-path="url(#breeze-help-line-13)">-</text><text class="breeze-help-r4" x="36.6" y="337.2" textLength="36.6" clip-path="url(#breeze-help-line-13)">-db</text><text class="breeze-help-r4" x="73.2" y="337.2" textLength="73.2" clip-path="url(#breeze-help-line-13)">-reset</text><text class="breeze-help-r6" x="3 [...]
+</text><text class="breeze-help-r5" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-help-line-14)">│</text><text class="breeze-help-r4" x="24.4" y="361.6" textLength="12.2" clip-path="url(#breeze-help-line-14)">-</text><text class="breeze-help-r4" x="36.6" y="361.6" textLength="48.8" clip-path="url(#breeze-help-line-14)">-max</text><text class="breeze-help-r4" x="85.4" y="361.6" textLength="61" clip-path="url(#breeze-help-line-14)">-time</text><text class="breeze-help-r2" x="353 [...]
+</text><text class="breeze-help-r5" x="0" y="386" textLength="12.2" clip-path="url(#breeze-help-line-15)">│</text><text class="breeze-help-r7" x="353.8" y="386" textLength="1049.2" clip-path="url(#breeze-help-line-15)">(INTEGER RANGE)                                          & [...]
+</text><text class="breeze-help-r5" x="0" y="410.4" textLength="12.2" clip-path="url(#breeze-help-line-16)">│</text><text class="breeze-help-r4" x="24.4" y="410.4" textLength="12.2" clip-path="url(#breeze-help-line-16)">-</text><text class="breeze-help-r4" x="36.6" y="410.4" textLength="85.4" clip-path="url(#breeze-help-line-16)">-github</text><text class="breeze-help-r4" x="122" y="410.4" textLength="134.2" clip-path="url(#breeze-help-line-16)">-repository</text><text class="breeze-help [...]
+</text><text class="breeze-help-r5" x="0" y="434.8" textLength="1464" clip-path="url(#breeze-help-line-17)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-help-r2" x="1464" y="434.8" textLength="12.2" clip-path="url(#breeze-help-line-17)">
+</text><text class="breeze-help-r5" x="0" y="459.2" textLength="24.4" clip-path="url(#breeze-help-line-18)">╭─</text><text class="breeze-help-r5" x="24.4" y="459.2" textLength="195.2" clip-path="url(#breeze-help-line-18)"> Common options </text><text class="breeze-help-r5" x="219.6" y="459.2" textLength="1220" clip-path="url(#breeze-help-line-18)">────────────────────────────────────────────────────────────────────────────────────────────────────</text><text class="breeze- [...]
+</text><text class="breeze-help-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-help-line-19)">│</text><text class="breeze-help-r4" x="24.4" y="483.6" textLength="12.2" clip-path="url(#breeze-help-line-19)">-</text><text class="breeze-help-r4" x="36.6" y="483.6" textLength="97.6" clip-path="url(#breeze-help-line-19)">-verbose</text><text class="breeze-help-r6" x="158.6" y="483.6" textLength="24.4" clip-path="url(#breeze-help-line-19)">-v</text><text class="breeze-help-r2" x= [...]
+</text><text class="breeze-help-r5" x="0" y="508" textLength="12.2" clip-path="url(#breeze-help-line-20)">│</text><text class="breeze-help-r4" x="24.4" y="508" textLength="12.2" clip-path="url(#breeze-help-line-20)">-</text><text class="breeze-help-r4" x="36.6" y="508" textLength="48.8" clip-path="url(#breeze-help-line-20)">-dry</text><text class="breeze-help-r4" x="85.4" y="508" textLength="48.8" clip-path="url(#breeze-help-line-20)">-run</text><text class="breeze-help-r6" x="158.6" y=" [...]
+</text><text class="breeze-help-r5" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-help-line-21)">│</text><text class="breeze-help-r4" x="24.4" y="532.4" textLength="12.2" clip-path="url(#breeze-help-line-21)">-</text><text class="breeze-help-r4" x="36.6" y="532.4" textLength="85.4" clip-path="url(#breeze-help-line-21)">-answer</text><text class="breeze-help-r6" x="158.6" y="532.4" textLength="24.4" clip-path="url(#breeze-help-line-21)">-a</text><text class="breeze-help-r2" x=" [...]
+</text><text class="breeze-help-r5" x="0" y="556.8" textLength="12.2" clip-path="url(#breeze-help-line-22)">│</text><text class="breeze-help-r4" x="24.4" y="556.8" textLength="12.2" clip-path="url(#breeze-help-line-22)">-</text><text class="breeze-help-r4" x="36.6" y="556.8" textLength="61" clip-path="url(#breeze-help-line-22)">-help</text><text class="breeze-help-r6" x="158.6" y="556.8" textLength="24.4" clip-path="url(#breeze-help-line-22)">-h</text><text class="breeze-help-r2" x="207. [...]
+</text><text class="breeze-help-r5" x="0" y="581.2" textLength="1464" clip-path="url(#breeze-help-line-23)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-help-r2" x="1464" y="581.2" textLength="12.2" clip-path="url(#breeze-help-line-23)">
+</text><text class="breeze-help-r5" x="0" y="605.6" textLength="24.4" clip-path="url(#breeze-help-line-24)">╭─</text><text class="breeze-help-r5" x="24.4" y="605.6" textLength="317.2" clip-path="url(#breeze-help-line-24)"> Basic developer commands </text><text class="breeze-help-r5" x="341.6" y="605.6" textLength="1098" clip-path="url(#breeze-help-line-24)">──────────────────────────────────────────────────────────────────────────────────────────</text><text class="br [...]
+</text><text class="breeze-help-r5" x="0" y="630" textLength="12.2" clip-path="url(#breeze-help-line-25)">│</text><text class="breeze-help-r4" x="24.4" y="630" textLength="219.6" clip-path="url(#breeze-help-line-25)">start-airflow     </text><text class="breeze-help-r2" x="268.4" y="630" textLength="1171.2" clip-path="url(#breeze-help-line-25)">Enter breeze environment and starts all Airflow components in the tmux [...]
+</text><text class="breeze-help-r5" x="0" y="654.4" textLength="12.2" clip-path="url(#breeze-help-line-26)">│</text><text class="breeze-help-r2" x="268.4" y="654.4" textLength="1171.2" clip-path="url(#breeze-help-line-26)">if contents of www directory changed.                                   [...]
+</text><text class="breeze-help-r5" x="0" y="678.8" textLength="12.2" clip-path="url(#breeze-help-line-27)">│</text><text class="breeze-help-r4" x="24.4" y="678.8" textLength="219.6" clip-path="url(#breeze-help-line-27)">static-checks     </text><text class="breeze-help-r2" x="268.4" y="678.8" textLength="1171.2" clip-path="url(#breeze-help-line-27)">Run static checks.              &# [...]
+</text><text class="breeze-help-r5" x="0" y="703.2" textLength="12.2" clip-path="url(#breeze-help-line-28)">│</text><text class="breeze-help-r4" x="24.4" y="703.2" textLength="219.6" clip-path="url(#breeze-help-line-28)">build-docs        </text><text class="breeze-help-r2" x="268.4" y="703.2" textLength="1171.2" clip-path="url(#breeze-help-line-28)">Build documentation in the container.        [...]
+</text><text class="breeze-help-r5" x="0" y="727.6" textLength="12.2" clip-path="url(#breeze-help-line-29)">│</text><text class="breeze-help-r4" x="24.4" y="727.6" textLength="219.6" clip-path="url(#breeze-help-line-29)">stop              </text><text class="breeze-help-r2" x="268.4" y="727.6" textLength="1171.2" clip-path="url(#breeze-help-line-29)">Stop running breeze environment.    [...]
+</text><text class="breeze-help-r5" x="0" y="752" textLength="12.2" clip-path="url(#breeze-help-line-30)">│</text><text class="breeze-help-r4" x="24.4" y="752" textLength="219.6" clip-path="url(#breeze-help-line-30)">shell             </text><text class="breeze-help-r2" x="268.4" y="752" textLength="1171.2" clip-path="url(#breeze-help-line-30)">Enter breeze environment. this is the default  [...]
+</text><text class="breeze-help-r5" x="0" y="776.4" textLength="12.2" clip-path="url(#breeze-help-line-31)">│</text><text class="breeze-help-r4" x="24.4" y="776.4" textLength="219.6" clip-path="url(#breeze-help-line-31)">exec              </text><text class="breeze-help-r2" x="268.4" y="776.4" textLength="1171.2" clip-path="url(#breeze-help-line-31)">Joins the interactive shell of running  [...]
+</text><text class="breeze-help-r5" x="0" y="800.8" textLength="12.2" clip-path="url(#breeze-help-line-32)">│</text><text class="breeze-help-r4" x="24.4" y="800.8" textLength="219.6" clip-path="url(#breeze-help-line-32)">compile-www-assets</text><text class="breeze-help-r2" x="268.4" y="800.8" textLength="1171.2" clip-path="url(#breeze-help-line-32)">Compiles www assets.                  & [...]
+</text><text class="breeze-help-r5" x="0" y="825.2" textLength="12.2" clip-path="url(#breeze-help-line-33)">│</text><text class="breeze-help-r4" x="24.4" y="825.2" textLength="219.6" clip-path="url(#breeze-help-line-33)">cleanup           </text><text class="breeze-help-r2" x="268.4" y="825.2" textLength="805.2" clip-path="url(#breeze-help-line-33)">Cleans the cache of parameters, docker cache and&# [...]
+</text><text class="breeze-help-r5" x="0" y="849.6" textLength="1464" clip-path="url(#breeze-help-line-34)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-help-r2" x="1464" y="849.6" textLength="12.2" clip-path="url(#breeze-help-line-34)">
+</text><text class="breeze-help-r5" x="0" y="874" textLength="24.4" clip-path="url(#breeze-help-line-35)">╭─</text><text class="breeze-help-r5" x="24.4" y="874" textLength="305" clip-path="url(#breeze-help-line-35)"> Advanced command groups </text><text class="breeze-help-r5" x="329.4" y="874" textLength="1110.2" clip-path="url(#breeze-help-line-35)">───────────────────────────────────────────────────────────────────────────────────────────</text><text class="breeze-h [...]
+</text><text class="breeze-help-r5" x="0" y="898.4" textLength="12.2" clip-path="url(#breeze-help-line-36)">│</text><text class="breeze-help-r4" x="24.4" y="898.4" textLength="280.6" clip-path="url(#breeze-help-line-36)">testing                </text><text class="breeze-help-r2" x="329.4" y="898.4" textLength="1110.2" clip-path="url(#breeze-help-line-36)">Tools that developers can use  [...]
+</text><text class="breeze-help-r5" x="0" y="922.8" textLength="12.2" clip-path="url(#breeze-help-line-37)">│</text><text class="breeze-help-r4" x="24.4" y="922.8" textLength="280.6" clip-path="url(#breeze-help-line-37)">ci-image               </text><text class="breeze-help-r2" x="329.4" y="922.8" textLength="597.8" clip-path="url(#breeze-help-line-37)">Tools that developers can use to [...]
+</text><text class="breeze-help-r5" x="0" y="947.2" textLength="12.2" clip-path="url(#breeze-help-line-38)">│</text><text class="breeze-help-r4" x="24.4" y="947.2" textLength="280.6" clip-path="url(#breeze-help-line-38)">k8s                    </text><text class="breeze-help-r2" x="329.4" y="947.2" textLength="1110.2" clip-path="url(#breeze-help-line-38)">Tools that developers [...]
+</text><text class="breeze-help-r5" x="0" y="971.6" textLength="12.2" clip-path="url(#breeze-help-line-39)">│</text><text class="breeze-help-r4" x="24.4" y="971.6" textLength="280.6" clip-path="url(#breeze-help-line-39)">prod-image             </text><text class="breeze-help-r2" x="329.4" y="971.6" textLength="597.8" clip-path="url(#breeze-help-line-39)">Tools that developers can use to manuall [...]
+</text><text class="breeze-help-r5" x="0" y="996" textLength="12.2" clip-path="url(#breeze-help-line-40)">│</text><text class="breeze-help-r4" x="24.4" y="996" textLength="280.6" clip-path="url(#breeze-help-line-40)">setup                  </text><text class="breeze-help-r2" x="329.4" y="996" textLength="1110.2" clip-path="url(#breeze-help-line-40)">Tools that developers can use& [...]
+</text><text class="breeze-help-r5" x="0" y="1020.4" textLength="12.2" clip-path="url(#breeze-help-line-41)">│</text><text class="breeze-help-r4" x="24.4" y="1020.4" textLength="280.6" clip-path="url(#breeze-help-line-41)">release-management     </text><text class="breeze-help-r2" x="329.4" y="1020.4" textLength="1110.2" clip-path="url(#breeze-help-line-41)">Tools that release managers can use to prepare and manage [...]
+</text><text class="breeze-help-r5" x="0" y="1044.8" textLength="12.2" clip-path="url(#breeze-help-line-42)">│</text><text class="breeze-help-r4" x="24.4" y="1044.8" textLength="280.6" clip-path="url(#breeze-help-line-42)">ci                     </text><text class="breeze-help-r2" x="329.4" y="1044.8" textLength="134.2" clip-path="url(#breeze-help-line-42)">Tools that </text [...]
+</text><text class="breeze-help-r5" x="0" y="1069.2" textLength="1464" clip-path="url(#breeze-help-line-43)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-help-r2" x="1464" y="1069.2" textLength="12.2" clip-path="url(#breeze-help-line-43)">
</text>
</g>
</g>
diff --git a/images/breeze/output_ci-image.svg b/images/breeze/output_ci-image.svg
index 53e01649c2..a7efb69b26 100644
--- a/images/breeze/output_ci-image.svg
+++ b/images/breeze/output_ci-image.svg
@@ -32,12 +32,12 @@
font-family: arial;
}
- .terminal-985073313-r1 { fill: #c5c8c6;font-weight: bold }
-.terminal-985073313-r2 { fill: #c5c8c6 }
-.terminal-985073313-r3 { fill: #d0b344;font-weight: bold }
-.terminal-985073313-r4 { fill: #68a0b3;font-weight: bold }
-.terminal-985073313-r5 { fill: #868887 }
-.terminal-985073313-r6 { fill: #98a84b;font-weight: bold }
+ .breeze-ci-image-r1 { fill: #c5c8c6;font-weight: bold }
+.breeze-ci-image-r2 { fill: #c5c8c6 }
+.breeze-ci-image-r3 { fill: #d0b344;font-weight: bold }
+.breeze-ci-image-r4 { fill: #68a0b3;font-weight: bold }
+.breeze-ci-image-r5 { fill: #868887 }
+.breeze-ci-image-r6 { fill: #98a84b;font-weight: bold }
</style>
<defs>
@@ -91,20 +91,20 @@
<g transform="translate(9, 41)" clip-path="url(#terminal-985073313-clip-terminal)">
- <g class="terminal-985073313-matrix">
- <text class="terminal-985073313-r2" x="1464" y="20" textLength="12.2" clip-path="url(#terminal-985073313-line-0)">
-</text><text class="terminal-985073313-r3" x="12.2" y="44.4" textLength="85.4" clip-path="url(#terminal-985073313-line-1)">Usage: </text><text class="terminal-985073313-r1" x="97.6" y="44.4" textLength="207.4" clip-path="url(#terminal-985073313-line-1)">breeze ci-image [</text><text class="terminal-985073313-r4" x="305" y="44.4" textLength="85.4" clip-path="url(#terminal-985073313-line-1)">OPTIONS</text><text class="terminal-985073313-r1" x="390.4" y="44.4" textLength="24. [...]
-</text><text class="terminal-985073313-r2" x="1464" y="68.8" textLength="12.2" clip-path="url(#terminal-985073313-line-2)">
-</text><text class="terminal-985073313-r2" x="12.2" y="93.2" textLength="597.8" clip-path="url(#terminal-985073313-line-3)">Tools that developers can use to manually manage </text><text class="terminal-985073313-r4" x="610" y="93.2" textLength="24.4" clip-path="url(#terminal-985073313-line-3)">CI</text><text class="terminal-985073313-r2" x="634.4" y="93.2" textLength="85.4" clip-path="url(#terminal-985073313-line-3)"> images</text><text class= [...]
-</text><text class="terminal-985073313-r2" x="1464" y="117.6" textLength="12.2" clip-path="url(#terminal-985073313-line-4)">
-</text><text class="terminal-985073313-r5" x="0" y="142" textLength="24.4" clip-path="url(#terminal-985073313-line-5)">╭─</text><text class="terminal-985073313-r5" x="24.4" y="142" textLength="195.2" clip-path="url(#terminal-985073313-line-5)"> Common options </text><text class="terminal-985073313-r5" x="219.6" y="142" textLength="1220" clip-path="url(#terminal-985073313-line-5)">────────────────────────────────────────────────────────────────────────────────────────────── [...]
-</text><text class="terminal-985073313-r5" x="0" y="166.4" textLength="12.2" clip-path="url(#terminal-985073313-line-6)">│</text><text class="terminal-985073313-r4" x="24.4" y="166.4" textLength="12.2" clip-path="url(#terminal-985073313-line-6)">-</text><text class="terminal-985073313-r4" x="36.6" y="166.4" textLength="61" clip-path="url(#terminal-985073313-line-6)">-help</text><text class="terminal-985073313-r6" x="122" y="166.4" textLength="24.4" clip-path="url(#terminal-985073313-line [...]
-</text><text class="terminal-985073313-r5" x="0" y="190.8" textLength="1464" clip-path="url(#terminal-985073313-line-7)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="terminal-985073313-r2" x="1464" y="190.8" textLength="12.2" clip-path="url(#terminal-985073313-line-7)">
-</text><text class="terminal-985073313-r5" x="0" y="215.2" textLength="24.4" clip-path="url(#terminal-985073313-line-8)">╭─</text><text class="terminal-985073313-r5" x="24.4" y="215.2" textLength="195.2" clip-path="url(#terminal-985073313-line-8)"> CI Image tools </text><text class="terminal-985073313-r5" x="219.6" y="215.2" textLength="1220" clip-path="url(#terminal-985073313-line-8)">─────────────────────────────────────────────────────────────────────────────────── [...]
-</text><text class="terminal-985073313-r5" x="0" y="239.6" textLength="12.2" clip-path="url(#terminal-985073313-line-9)">│</text><text class="terminal-985073313-r4" x="24.4" y="239.6" textLength="97.6" clip-path="url(#terminal-985073313-line-9)">build   </text><text class="terminal-985073313-r2" x="146.4" y="239.6" textLength="73.2" clip-path="url(#terminal-985073313-line-9)">Build </text><text class="terminal-985073313-r4" x="219.6" y="239.6" textLength="24.4" clip-p [...]
-</text><text class="terminal-985073313-r5" x="0" y="264" textLength="12.2" clip-path="url(#terminal-985073313-line-10)">│</text><text class="terminal-985073313-r4" x="24.4" y="264" textLength="97.6" clip-path="url(#terminal-985073313-line-10)">pull    </text><text class="terminal-985073313-r2" x="146.4" y="264" textLength="329.4" clip-path="url(#terminal-985073313-line-10)">Pull and optionally verify </text><text class="terminal-985073313-r4" x="47 [...]
-</text><text class="terminal-985073313-r5" x="0" y="288.4" textLength="12.2" clip-path="url(#terminal-985073313-line-11)">│</text><text class="terminal-985073313-r4" x="24.4" y="288.4" textLength="97.6" clip-path="url(#terminal-985073313-line-11)">verify  </text><text class="terminal-985073313-r2" x="146.4" y="288.4" textLength="85.4" clip-path="url(#terminal-985073313-line-11)">Verify </text><text class="terminal-985073313-r4" x="231.8" y="288.4" textLength="24.4" clip-pa [...]
-</text><text class="terminal-985073313-r5" x="0" y="312.8" textLength="1464" clip-path="url(#terminal-985073313-line-12)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="terminal-985073313-r2" x="1464" y="312.8" textLength="12.2" clip-path="url(#terminal-985073313-line-12)">
+ <g class="breeze-ci-image-matrix">
+ <text class="breeze-ci-image-r2" x="1464" y="20" textLength="12.2" clip-path="url(#breeze-ci-image-line-0)">
+</text><text class="breeze-ci-image-r3" x="12.2" y="44.4" textLength="85.4" clip-path="url(#breeze-ci-image-line-1)">Usage: </text><text class="breeze-ci-image-r1" x="97.6" y="44.4" textLength="207.4" clip-path="url(#breeze-ci-image-line-1)">breeze ci-image [</text><text class="breeze-ci-image-r4" x="305" y="44.4" textLength="85.4" clip-path="url(#breeze-ci-image-line-1)">OPTIONS</text><text class="breeze-ci-image-r1" x="390.4" y="44.4" textLength="24.4" clip-path="url(#br [...]
+</text><text class="breeze-ci-image-r2" x="1464" y="68.8" textLength="12.2" clip-path="url(#breeze-ci-image-line-2)">
+</text><text class="breeze-ci-image-r2" x="12.2" y="93.2" textLength="597.8" clip-path="url(#breeze-ci-image-line-3)">Tools that developers can use to manually manage </text><text class="breeze-ci-image-r4" x="610" y="93.2" textLength="24.4" clip-path="url(#breeze-ci-image-line-3)">CI</text><text class="breeze-ci-image-r2" x="634.4" y="93.2" textLength="85.4" clip-path="url(#breeze-ci-image-line-3)"> images</text><text class="breeze-ci-image-r [...]
+</text><text class="breeze-ci-image-r2" x="1464" y="117.6" textLength="12.2" clip-path="url(#breeze-ci-image-line-4)">
+</text><text class="breeze-ci-image-r5" x="0" y="142" textLength="24.4" clip-path="url(#breeze-ci-image-line-5)">╭─</text><text class="breeze-ci-image-r5" x="24.4" y="142" textLength="195.2" clip-path="url(#breeze-ci-image-line-5)"> Common options </text><text class="breeze-ci-image-r5" x="219.6" y="142" textLength="1220" clip-path="url(#breeze-ci-image-line-5)">────────────────────────────────────────────────────────────────────────────────────────────────────</text><text [...]
+</text><text class="breeze-ci-image-r5" x="0" y="166.4" textLength="12.2" clip-path="url(#breeze-ci-image-line-6)">│</text><text class="breeze-ci-image-r4" x="24.4" y="166.4" textLength="12.2" clip-path="url(#breeze-ci-image-line-6)">-</text><text class="breeze-ci-image-r4" x="36.6" y="166.4" textLength="61" clip-path="url(#breeze-ci-image-line-6)">-help</text><text class="breeze-ci-image-r6" x="122" y="166.4" textLength="24.4" clip-path="url(#breeze-ci-image-line-6)">-h</text><text clas [...]
+</text><text class="breeze-ci-image-r5" x="0" y="190.8" textLength="1464" clip-path="url(#breeze-ci-image-line-7)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-r2" x="1464" y="190.8" textLength="12.2" clip-path="url(#breeze-ci-image-line-7)">
+</text><text class="breeze-ci-image-r5" x="0" y="215.2" textLength="24.4" clip-path="url(#breeze-ci-image-line-8)">╭─</text><text class="breeze-ci-image-r5" x="24.4" y="215.2" textLength="195.2" clip-path="url(#breeze-ci-image-line-8)"> CI Image tools </text><text class="breeze-ci-image-r5" x="219.6" y="215.2" textLength="1220" clip-path="url(#breeze-ci-image-line-8)">────────────────────────────────────────────────────────────────────────────────────────────────────< [...]
+</text><text class="breeze-ci-image-r5" x="0" y="239.6" textLength="12.2" clip-path="url(#breeze-ci-image-line-9)">│</text><text class="breeze-ci-image-r4" x="24.4" y="239.6" textLength="97.6" clip-path="url(#breeze-ci-image-line-9)">build   </text><text class="breeze-ci-image-r2" x="146.4" y="239.6" textLength="73.2" clip-path="url(#breeze-ci-image-line-9)">Build </text><text class="breeze-ci-image-r4" x="219.6" y="239.6" textLength="24.4" clip-path="url(#breeze-ci-i [...]
+</text><text class="breeze-ci-image-r5" x="0" y="264" textLength="12.2" clip-path="url(#breeze-ci-image-line-10)">│</text><text class="breeze-ci-image-r4" x="24.4" y="264" textLength="97.6" clip-path="url(#breeze-ci-image-line-10)">pull    </text><text class="breeze-ci-image-r2" x="146.4" y="264" textLength="329.4" clip-path="url(#breeze-ci-image-line-10)">Pull and optionally verify </text><text class="breeze-ci-image-r4" x="475.8" y="264" textLeng [...]
+</text><text class="breeze-ci-image-r5" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-ci-image-line-11)">│</text><text class="breeze-ci-image-r4" x="24.4" y="288.4" textLength="97.6" clip-path="url(#breeze-ci-image-line-11)">verify  </text><text class="breeze-ci-image-r2" x="146.4" y="288.4" textLength="85.4" clip-path="url(#breeze-ci-image-line-11)">Verify </text><text class="breeze-ci-image-r4" x="231.8" y="288.4" textLength="24.4" clip-path="url(#breeze-ci-im [...]
+</text><text class="breeze-ci-image-r5" x="0" y="312.8" textLength="1464" clip-path="url(#breeze-ci-image-line-12)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-image-r2" x="1464" y="312.8" textLength="12.2" clip-path="url(#breeze-ci-image-line-12)">
</text>
</g>
</g>
diff --git a/images/breeze/output_release-management.svg b/images/breeze/output_release-management.svg
index 1a70516162..b1119c5deb 100644
--- a/images/breeze/output_release-management.svg
+++ b/images/breeze/output_release-management.svg
@@ -35,8 +35,8 @@
.breeze-release-management-r1 { fill: #c5c8c6;font-weight: bold }
.breeze-release-management-r2 { fill: #c5c8c6 }
.breeze-release-management-r3 { fill: #d0b344;font-weight: bold }
-.breeze-release-management-r4 { fill: #868887 }
-.breeze-release-management-r5 { fill: #68a0b3;font-weight: bold }
+.breeze-release-management-r4 { fill: #68a0b3;font-weight: bold }
+.breeze-release-management-r5 { fill: #868887 }
.breeze-release-management-r6 { fill: #98a84b;font-weight: bold }
</style>
@@ -105,22 +105,22 @@
<g class="breeze-release-management-matrix">
<text class="breeze-release-management-r2" x="1464" y="20" textLength="12.2" clip-path="url(#breeze-release-management-line-0)">
-</text><text class="breeze-release-management-r3" x="12.2" y="44.4" textLength="85.4" clip-path="url(#breeze-release-management-line-1)">Usage: </text><text class="breeze-release-management-r1" x="97.6" y="44.4" textLength="646.6" clip-path="url(#breeze-release-management-line-1)">breeze release-management [OPTIONS] COMMAND [ARGS]...</text><text class="breeze-release-management-r2" x="1464" y="44.4" textLength="12.2" clip-path="url(#breeze-release-management-line-1)">
+</text><text class="breeze-release-management-r3" x="12.2" y="44.4" textLength="85.4" clip-path="url(#breeze-release-management-line-1)">Usage: </text><text class="breeze-release-management-r1" x="97.6" y="44.4" textLength="329.4" clip-path="url(#breeze-release-management-line-1)">breeze release-management [</text><text class="breeze-release-management-r4" x="427" y="44.4" textLength="85.4" clip-path="url(#breeze-release-management-line-1)">OPTIONS</text><text class="breez [...]
</text><text class="breeze-release-management-r2" x="1464" y="68.8" textLength="12.2" clip-path="url(#breeze-release-management-line-2)">
</text><text class="breeze-release-management-r2" x="12.2" y="93.2" textLength="902.8" clip-path="url(#breeze-release-management-line-3)">Tools that release managers can use to prepare and manage Airflow releases</text><text class="breeze-release-management-r2" x="1464" y="93.2" textLength="12.2" clip-path="url(#breeze-release-management-line-3)">
</text><text class="breeze-release-management-r2" x="1464" y="117.6" textLength="12.2" clip-path="url(#breeze-release-management-line-4)">
-</text><text class="breeze-release-management-r4" x="0" y="142" textLength="24.4" clip-path="url(#breeze-release-management-line-5)">╭─</text><text class="breeze-release-management-r4" x="24.4" y="142" textLength="195.2" clip-path="url(#breeze-release-management-line-5)"> Common options </text><text class="breeze-release-management-r4" x="219.6" y="142" textLength="1220" clip-path="url(#breeze-release-management-line-5)">──────────────────────────────────────────────────── [...]
-</text><text class="breeze-release-management-r4" x="0" y="166.4" textLength="12.2" clip-path="url(#breeze-release-management-line-6)">│</text><text class="breeze-release-management-r5" x="24.4" y="166.4" textLength="12.2" clip-path="url(#breeze-release-management-line-6)">-</text><text class="breeze-release-management-r5" x="36.6" y="166.4" textLength="61" clip-path="url(#breeze-release-management-line-6)">-help</text><text class="breeze-release-management-r6" x="122" y="166.4" textLeng [...]
-</text><text class="breeze-release-management-r4" x="0" y="190.8" textLength="1464" clip-path="url(#breeze-release-management-line-7)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-release-management-r2" x="1464" y="190.8" textLength="12.2" clip-path="url(#breeze-release-management-line-7)">
-</text><text class="breeze-release-management-r4" x="0" y="215.2" textLength="24.4" clip-path="url(#breeze-release-management-line-8)">╭─</text><text class="breeze-release-management-r4" x="24.4" y="215.2" textLength="122" clip-path="url(#breeze-release-management-line-8)"> Commands </text><text class="breeze-release-management-r4" x="146.4" y="215.2" textLength="1293.2" clip-path="url(#breeze-release-management-line-8)">───────────────────────────────────────────────────────── [...]
-</text><text class="breeze-release-management-r4" x="0" y="239.6" textLength="12.2" clip-path="url(#breeze-release-management-line-9)">│</text><text class="breeze-release-management-r5" x="24.4" y="239.6" textLength="402.6" clip-path="url(#breeze-release-management-line-9)">generate-constraints             </text><text class="breeze-release-management-r2" x="451.4" y="239.6" textLength="988.2" clip-path="url(#breeze-release [...]
-</text><text class="breeze-release-management-r4" x="0" y="264" textLength="12.2" clip-path="url(#breeze-release-management-line-10)">│</text><text class="breeze-release-management-r5" x="24.4" y="264" textLength="402.6" clip-path="url(#breeze-release-management-line-10)">generate-issue-content           </text><text class="breeze-release-management-r2" x="451.4" y="264" textLength="988.2" clip-path="url(#breeze-release-management-li [...]
-</text><text class="breeze-release-management-r4" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-release-management-line-11)">│</text><text class="breeze-release-management-r5" x="24.4" y="288.4" textLength="402.6" clip-path="url(#breeze-release-management-line-11)">prepare-airflow-package          </text><text class="breeze-release-management-r2" x="451.4" y="288.4" textLength="988.2" clip-path="url(#breeze-release-management-l [...]
-</text><text class="breeze-release-management-r4" x="0" y="312.8" textLength="12.2" clip-path="url(#breeze-release-management-line-12)">│</text><text class="breeze-release-management-r5" x="24.4" y="312.8" textLength="402.6" clip-path="url(#breeze-release-management-line-12)">prepare-provider-documentation   </text><text class="breeze-release-management-r2" x="451.4" y="312.8" textLength="988.2" clip-path="url(#breeze-release-management-line-12)">Prepare CHANGELOG, [...]
-</text><text class="breeze-release-management-r4" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-release-management-line-13)">│</text><text class="breeze-release-management-r5" x="24.4" y="337.2" textLength="402.6" clip-path="url(#breeze-release-management-line-13)">prepare-provider-packages        </text><text class="breeze-release-management-r2" x="451.4" y="337.2" textLength="988.2" clip-path="url(#breeze-release-management-line-13)">P [...]
-</text><text class="breeze-release-management-r4" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-release-management-line-14)">│</text><text class="breeze-release-management-r5" x="24.4" y="361.6" textLength="402.6" clip-path="url(#breeze-release-management-line-14)">release-prod-images              </text><text class="breeze-release-management-r2" x="451.4" y="361.6" textLength="988.2" clip-path="url(#breeze- [...]
-</text><text class="breeze-release-management-r4" x="0" y="386" textLength="12.2" clip-path="url(#breeze-release-management-line-15)">│</text><text class="breeze-release-management-r5" x="24.4" y="386" textLength="402.6" clip-path="url(#breeze-release-management-line-15)">verify-provider-packages         </text><text class="breeze-release-management-r2" x="451.4" y="386" textLength="988.2" clip-path="url(#breeze-release-management-line-15)">Ve [...]
-</text><text class="breeze-release-management-r4" x="0" y="410.4" textLength="1464" clip-path="url(#breeze-release-management-line-16)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-release-management-r2" x="1464" y="410.4" textLength="12.2" clip-path="url(#breeze-release-management-line-16)">
+</text><text class="breeze-release-management-r5" x="0" y="142" textLength="24.4" clip-path="url(#breeze-release-management-line-5)">╭─</text><text class="breeze-release-management-r5" x="24.4" y="142" textLength="195.2" clip-path="url(#breeze-release-management-line-5)"> Common options </text><text class="breeze-release-management-r5" x="219.6" y="142" textLength="1220" clip-path="url(#breeze-release-management-line-5)">──────────────────────────────────────────────────── [...]
+</text><text class="breeze-release-management-r5" x="0" y="166.4" textLength="12.2" clip-path="url(#breeze-release-management-line-6)">│</text><text class="breeze-release-management-r4" x="24.4" y="166.4" textLength="12.2" clip-path="url(#breeze-release-management-line-6)">-</text><text class="breeze-release-management-r4" x="36.6" y="166.4" textLength="61" clip-path="url(#breeze-release-management-line-6)">-help</text><text class="breeze-release-management-r6" x="122" y="166.4" textLeng [...]
+</text><text class="breeze-release-management-r5" x="0" y="190.8" textLength="1464" clip-path="url(#breeze-release-management-line-7)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-release-management-r2" x="1464" y="190.8" textLength="12.2" clip-path="url(#breeze-release-management-line-7)">
+</text><text class="breeze-release-management-r5" x="0" y="215.2" textLength="24.4" clip-path="url(#breeze-release-management-line-8)">╭─</text><text class="breeze-release-management-r5" x="24.4" y="215.2" textLength="122" clip-path="url(#breeze-release-management-line-8)"> Commands </text><text class="breeze-release-management-r5" x="146.4" y="215.2" textLength="1293.2" clip-path="url(#breeze-release-management-line-8)">───────────────────────────────────────────────────────── [...]
+</text><text class="breeze-release-management-r5" x="0" y="239.6" textLength="12.2" clip-path="url(#breeze-release-management-line-9)">│</text><text class="breeze-release-management-r4" x="24.4" y="239.6" textLength="402.6" clip-path="url(#breeze-release-management-line-9)">generate-constraints             </text><text class="breeze-release-management-r2" x="451.4" y="239.6" textLength="988.2" clip-path="url(#breeze-release [...]
+</text><text class="breeze-release-management-r5" x="0" y="264" textLength="12.2" clip-path="url(#breeze-release-management-line-10)">│</text><text class="breeze-release-management-r4" x="24.4" y="264" textLength="402.6" clip-path="url(#breeze-release-management-line-10)">generate-issue-content           </text><text class="breeze-release-management-r2" x="451.4" y="264" textLength="988.2" clip-path="url(#breeze-release-management-li [...]
+</text><text class="breeze-release-management-r5" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-release-management-line-11)">│</text><text class="breeze-release-management-r4" x="24.4" y="288.4" textLength="402.6" clip-path="url(#breeze-release-management-line-11)">prepare-airflow-package          </text><text class="breeze-release-management-r2" x="451.4" y="288.4" textLength="988.2" clip-path="url(#breeze-release-management-l [...]
+</text><text class="breeze-release-management-r5" x="0" y="312.8" textLength="12.2" clip-path="url(#breeze-release-management-line-12)">│</text><text class="breeze-release-management-r4" x="24.4" y="312.8" textLength="402.6" clip-path="url(#breeze-release-management-line-12)">prepare-provider-documentation   </text><text class="breeze-release-management-r2" x="451.4" y="312.8" textLength="97.6" clip-path="url(#breeze-release-management-line-12)">Prepare </text><text c [...]
+</text><text class="breeze-release-management-r5" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-release-management-line-13)">│</text><text class="breeze-release-management-r4" x="24.4" y="337.2" textLength="402.6" clip-path="url(#breeze-release-management-line-13)">prepare-provider-packages        </text><text class="breeze-release-management-r2" x="451.4" y="337.2" textLength="988.2" clip-path="url(#breeze-release-management-line-13)">P [...]
+</text><text class="breeze-release-management-r5" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-release-management-line-14)">│</text><text class="breeze-release-management-r4" x="24.4" y="361.6" textLength="402.6" clip-path="url(#breeze-release-management-line-14)">release-prod-images              </text><text class="breeze-release-management-r2" x="451.4" y="361.6" textLength="988.2" clip-path="url(#breeze- [...]
+</text><text class="breeze-release-management-r5" x="0" y="386" textLength="12.2" clip-path="url(#breeze-release-management-line-15)">│</text><text class="breeze-release-management-r4" x="24.4" y="386" textLength="402.6" clip-path="url(#breeze-release-management-line-15)">verify-provider-packages         </text><text class="breeze-release-management-r2" x="451.4" y="386" textLength="988.2" clip-path="url(#breeze-release-management-line-15)">Ve [...]
+</text><text class="breeze-release-management-r5" x="0" y="410.4" textLength="1464" clip-path="url(#breeze-release-management-line-16)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-release-management-r2" x="1464" y="410.4" textLength="12.2" clip-path="url(#breeze-release-management-line-16)">
</text>
</g>
</g>
diff --git a/images/breeze/output_setup.svg b/images/breeze/output_setup.svg
index 91df1eb737..9a0d176ddb 100644
--- a/images/breeze/output_setup.svg
+++ b/images/breeze/output_setup.svg
@@ -35,8 +35,8 @@
.breeze-setup-r1 { fill: #c5c8c6;font-weight: bold }
.breeze-setup-r2 { fill: #c5c8c6 }
.breeze-setup-r3 { fill: #d0b344;font-weight: bold }
-.breeze-setup-r4 { fill: #868887 }
-.breeze-setup-r5 { fill: #68a0b3;font-weight: bold }
+.breeze-setup-r4 { fill: #68a0b3;font-weight: bold }
+.breeze-setup-r5 { fill: #868887 }
.breeze-setup-r6 { fill: #98a84b;font-weight: bold }
</style>
@@ -99,20 +99,21 @@
<g class="breeze-setup-matrix">
<text class="breeze-setup-r2" x="1464" y="20" textLength="12.2" clip-path="url(#breeze-setup-line-0)">
-</text><text class="breeze-setup-r3" x="12.2" y="44.4" textLength="85.4" clip-path="url(#breeze-setup-line-1)">Usage: </text><text class="breeze-setup-r1" x="97.6" y="44.4" textLength="488" clip-path="url(#breeze-setup-line-1)">breeze setup [OPTIONS] COMMAND [ARGS]...</text><text class="breeze-setup-r2" x="1464" y="44.4" textLength="12.2" clip-path="url(#breeze-setup-line-1)">
+</text><text class="breeze-setup-r3" x="12.2" y="44.4" textLength="85.4" clip-path="url(#breeze-setup-line-1)">Usage: </text><text class="breeze-setup-r1" x="97.6" y="44.4" textLength="170.8" clip-path="url(#breeze-setup-line-1)">breeze setup [</text><text class="breeze-setup-r4" x="268.4" y="44.4" textLength="85.4" clip-path="url(#breeze-setup-line-1)">OPTIONS</text><text class="breeze-setup-r1" x="353.8" y="44.4" textLength="24.4" clip-path="url(#breeze-setup-line-1)">]& [...]
</text><text class="breeze-setup-r2" x="1464" y="68.8" textLength="12.2" clip-path="url(#breeze-setup-line-2)">
</text><text class="breeze-setup-r2" x="12.2" y="93.2" textLength="597.8" clip-path="url(#breeze-setup-line-3)">Tools that developers can use to configure Breeze</text><text class="breeze-setup-r2" x="1464" y="93.2" textLength="12.2" clip-path="url(#breeze-setup-line-3)">
</text><text class="breeze-setup-r2" x="1464" y="117.6" textLength="12.2" clip-path="url(#breeze-setup-line-4)">
-</text><text class="breeze-setup-r4" x="0" y="142" textLength="24.4" clip-path="url(#breeze-setup-line-5)">╭─</text><text class="breeze-setup-r4" x="24.4" y="142" textLength="195.2" clip-path="url(#breeze-setup-line-5)"> Common options </text><text class="breeze-setup-r4" x="219.6" y="142" textLength="1220" clip-path="url(#breeze-setup-line-5)">────────────────────────────────────────────────────────────────────────────────────────────────────</text><text class="breeze-set [...]
-</text><text class="breeze-setup-r4" x="0" y="166.4" textLength="12.2" clip-path="url(#breeze-setup-line-6)">│</text><text class="breeze-setup-r5" x="24.4" y="166.4" textLength="12.2" clip-path="url(#breeze-setup-line-6)">-</text><text class="breeze-setup-r5" x="36.6" y="166.4" textLength="61" clip-path="url(#breeze-setup-line-6)">-help</text><text class="breeze-setup-r6" x="122" y="166.4" textLength="24.4" clip-path="url(#breeze-setup-line-6)">-h</text><text class="breeze-setup-r2" x="1 [...]
-</text><text class="breeze-setup-r4" x="0" y="190.8" textLength="1464" clip-path="url(#breeze-setup-line-7)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-setup-r2" x="1464" y="190.8" textLength="12.2" clip-path="url(#breeze-setup-line-7)">
-</text><text class="breeze-setup-r4" x="0" y="215.2" textLength="24.4" clip-path="url(#breeze-setup-line-8)">╭─</text><text class="breeze-setup-r4" x="24.4" y="215.2" textLength="122" clip-path="url(#breeze-setup-line-8)"> Commands </text><text class="breeze-setup-r4" x="146.4" y="215.2" textLength="1293.2" clip-path="url(#breeze-setup-line-8)">──────────────────────────────────────────────────────────────────────────────────────────────────────────</text><text class="breeze-se [...]
-</text><text class="breeze-setup-r4" x="0" y="239.6" textLength="12.2" clip-path="url(#breeze-setup-line-9)">│</text><text class="breeze-setup-r5" x="24.4" y="239.6" textLength="390.4" clip-path="url(#breeze-setup-line-9)">autocomplete                    </text><text class="breeze-setup-r2" x="439.2" y="239.6" textLength="1000.4" clip-path="url(#breeze-setup-line-9)">Enables autocompl [...]
-</text><text class="breeze-setup-r4" x="0" y="264" textLength="12.2" clip-path="url(#breeze-setup-line-10)">│</text><text class="breeze-setup-r5" x="24.4" y="264" textLength="390.4" clip-path="url(#breeze-setup-line-10)">config                          </text><text class="breeze-setup-r2" x="439.2" y="264" textLength="1000.4" clip-path="url(#breeze-setup-line- [...]
-</text><text class="breeze-setup-r4" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-setup-line-11)">│</text><text class="breeze-setup-r5" x="24.4" y="288.4" textLength="390.4" clip-path="url(#breeze-setup-line-11)">regenerate-command-images       </text><text class="breeze-setup-r2" x="439.2" y="288.4" textLength="1000.4" clip-path="url(#breeze-setup-line-11)">Regenerate breeze command images.       [...]
-</text><text class="breeze-setup-r4" x="0" y="312.8" textLength="12.2" clip-path="url(#breeze-setup-line-12)">│</text><text class="breeze-setup-r5" x="24.4" y="312.8" textLength="390.4" clip-path="url(#breeze-setup-line-12)">self-upgrade                    </text><text class="breeze-setup-r2" x="439.2" y="312.8" textLength="1000.4" clip-path="url(#breeze-setup-line-12)">Self upgrade&# [...]
-</text><text class="breeze-setup-r4" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-setup-line-13)">│</text><text class="breeze-setup-r5" x="24.4" y="337.2" textLength="390.4" clip-path="url(#breeze-setup-line-13)">version                         </text><text class="breeze-setup-r2" x="439.2" y="337.2" textLength="1000.4" clip-path="url(#breeze-setup-line [...]
-</text><text class="breeze-setup-r4" x="0" y="361.6" textLength="1464" clip-path="url(#breeze-setup-line-14)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-setup-r2" x="1464" y="361.6" textLength="12.2" clip-path="url(#breeze-setup-line-14)">
+</text><text class="breeze-setup-r5" x="0" y="142" textLength="24.4" clip-path="url(#breeze-setup-line-5)">╭─</text><text class="breeze-setup-r5" x="24.4" y="142" textLength="195.2" clip-path="url(#breeze-setup-line-5)"> Common options </text><text class="breeze-setup-r5" x="219.6" y="142" textLength="1220" clip-path="url(#breeze-setup-line-5)">────────────────────────────────────────────────────────────────────────────────────────────────────</text><text class="breeze-set [...]
+</text><text class="breeze-setup-r5" x="0" y="166.4" textLength="12.2" clip-path="url(#breeze-setup-line-6)">│</text><text class="breeze-setup-r4" x="24.4" y="166.4" textLength="12.2" clip-path="url(#breeze-setup-line-6)">-</text><text class="breeze-setup-r4" x="36.6" y="166.4" textLength="61" clip-path="url(#breeze-setup-line-6)">-help</text><text class="breeze-setup-r6" x="122" y="166.4" textLength="24.4" clip-path="url(#breeze-setup-line-6)">-h</text><text class="breeze-setup-r2" x="1 [...]
+</text><text class="breeze-setup-r5" x="0" y="190.8" textLength="1464" clip-path="url(#breeze-setup-line-7)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-setup-r2" x="1464" y="190.8" textLength="12.2" clip-path="url(#breeze-setup-line-7)">
+</text><text class="breeze-setup-r5" x="0" y="215.2" textLength="24.4" clip-path="url(#breeze-setup-line-8)">╭─</text><text class="breeze-setup-r5" x="24.4" y="215.2" textLength="122" clip-path="url(#breeze-setup-line-8)"> Commands </text><text class="breeze-setup-r5" x="146.4" y="215.2" textLength="1293.2" clip-path="url(#breeze-setup-line-8)">──────────────────────────────────────────────────────────────────────────────────────────────────────────</text><text class="breeze-se [...]
+</text><text class="breeze-setup-r5" x="0" y="239.6" textLength="12.2" clip-path="url(#breeze-setup-line-9)">│</text><text class="breeze-setup-r4" x="24.4" y="239.6" textLength="402.6" clip-path="url(#breeze-setup-line-9)">autocomplete                     </text><text class="breeze-setup-r2" x="451.4" y="239.6" textLength="988.2" clip-path="url(#breeze-setup-line-9)">Enables auto [...]
+</text><text class="breeze-setup-r5" x="0" y="264" textLength="12.2" clip-path="url(#breeze-setup-line-10)">│</text><text class="breeze-setup-r4" x="24.4" y="264" textLength="402.6" clip-path="url(#breeze-setup-line-10)">check-all-params-in-groups       </text><text class="breeze-setup-r2" x="451.4" y="264" textLength="988.2" clip-path="url(#breeze-setup-line-10)">Check that all parameters are put in groups.   [...]
+</text><text class="breeze-setup-r5" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-setup-line-11)">│</text><text class="breeze-setup-r4" x="24.4" y="288.4" textLength="402.6" clip-path="url(#breeze-setup-line-11)">config                           </text><text class="breeze-setup-r2" x="451.4" y="288.4" textLength="683.2" clip-path="url(#breeze- [...]
+</text><text class="breeze-setup-r5" x="0" y="312.8" textLength="12.2" clip-path="url(#breeze-setup-line-12)">│</text><text class="breeze-setup-r4" x="24.4" y="312.8" textLength="402.6" clip-path="url(#breeze-setup-line-12)">regenerate-command-images        </text><text class="breeze-setup-r2" x="451.4" y="312.8" textLength="988.2" clip-path="url(#breeze-setup-line-12)">Regenerate breeze command images.     & [...]
+</text><text class="breeze-setup-r5" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-setup-line-13)">│</text><text class="breeze-setup-r4" x="24.4" y="337.2" textLength="402.6" clip-path="url(#breeze-setup-line-13)">self-upgrade                     </text><text class="breeze-setup-r2" x="451.4" y="337.2" textLength="988.2" clip-path="url(#breeze-setup-line-13)">Self upgr [...]
+</text><text class="breeze-setup-r5" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-setup-line-14)">│</text><text class="breeze-setup-r4" x="24.4" y="361.6" textLength="402.6" clip-path="url(#breeze-setup-line-14)">version                          </text><text class="breeze-setup-r2" x="451.4" y="361.6" textLength="988.2" clip-path="url(#breeze-setup [...]
+</text><text class="breeze-setup-r5" x="0" y="386" textLength="1464" clip-path="url(#breeze-setup-line-15)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-setup-r2" x="1464" y="386" textLength="12.2" clip-path="url(#breeze-setup-line-15)">
</text>
</g>
</g>
diff --git a/images/breeze/output_setup_check-all-params-in-groups.svg b/images/breeze/output_setup_check-all-params-in-groups.svg
new file mode 100644
index 0000000000..85c23ac5e3
--- /dev/null
+++ b/images/breeze/output_setup_check-all-params-in-groups.svg
@@ -0,0 +1,172 @@
+<svg class="rich-terminal" viewBox="0 0 1482 733.1999999999999" xmlns="http://www.w3.org/2000/svg">
+ <!-- Generated with Rich https://www.textualize.io -->
+ <style>
+
+ @font-face {
+ font-family: "Fira Code";
+ src: local("FiraCode-Regular"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff2/FiraCode-Regular.woff2") format("woff2"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff/FiraCode-Regular.woff") format("woff");
+ font-style: normal;
+ font-weight: 400;
+ }
+ @font-face {
+ font-family: "Fira Code";
+ src: local("FiraCode-Bold"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff2/FiraCode-Bold.woff2") format("woff2"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff/FiraCode-Bold.woff") format("woff");
+ font-style: bold;
+ font-weight: 700;
+ }
+
+ .breeze-setup-check-all-params-in-groups-matrix {
+ font-family: Fira Code, monospace;
+ font-size: 20px;
+ line-height: 24.4px;
+ font-variant-east-asian: full-width;
+ }
+
+ .breeze-setup-check-all-params-in-groups-title {
+ font-size: 18px;
+ font-weight: bold;
+ font-family: arial;
+ }
+
+ .breeze-setup-check-all-params-in-groups-r1 { fill: #c5c8c6;font-weight: bold }
+.breeze-setup-check-all-params-in-groups-r2 { fill: #c5c8c6 }
+.breeze-setup-check-all-params-in-groups-r3 { fill: #d0b344;font-weight: bold }
+.breeze-setup-check-all-params-in-groups-r4 { fill: #68a0b3;font-weight: bold }
+.breeze-setup-check-all-params-in-groups-r5 { fill: #868887 }
+.breeze-setup-check-all-params-in-groups-r6 { fill: #8d7b39 }
+.breeze-setup-check-all-params-in-groups-r7 { fill: #98a84b;font-weight: bold }
+ </style>
+
+ <defs>
+ <clipPath id="breeze-setup-check-all-params-in-groups-clip-terminal">
+ <rect x="0" y="0" width="1463.0" height="682.1999999999999" />
+ </clipPath>
+ <clipPath id="breeze-setup-check-all-params-in-groups-line-0">
+ <rect x="0" y="1.5" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-setup-check-all-params-in-groups-line-1">
+ <rect x="0" y="25.9" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-setup-check-all-params-in-groups-line-2">
+ <rect x="0" y="50.3" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-setup-check-all-params-in-groups-line-3">
+ <rect x="0" y="74.7" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-setup-check-all-params-in-groups-line-4">
+ <rect x="0" y="99.1" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-setup-check-all-params-in-groups-line-5">
+ <rect x="0" y="123.5" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-setup-check-all-params-in-groups-line-6">
+ <rect x="0" y="147.9" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-setup-check-all-params-in-groups-line-7">
+ <rect x="0" y="172.3" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-setup-check-all-params-in-groups-line-8">
+ <rect x="0" y="196.7" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-setup-check-all-params-in-groups-line-9">
+ <rect x="0" y="221.1" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-setup-check-all-params-in-groups-line-10">
+ <rect x="0" y="245.5" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-setup-check-all-params-in-groups-line-11">
+ <rect x="0" y="269.9" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-setup-check-all-params-in-groups-line-12">
+ <rect x="0" y="294.3" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-setup-check-all-params-in-groups-line-13">
+ <rect x="0" y="318.7" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-setup-check-all-params-in-groups-line-14">
+ <rect x="0" y="343.1" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-setup-check-all-params-in-groups-line-15">
+ <rect x="0" y="367.5" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-setup-check-all-params-in-groups-line-16">
+ <rect x="0" y="391.9" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-setup-check-all-params-in-groups-line-17">
+ <rect x="0" y="416.3" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-setup-check-all-params-in-groups-line-18">
+ <rect x="0" y="440.7" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-setup-check-all-params-in-groups-line-19">
+ <rect x="0" y="465.1" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-setup-check-all-params-in-groups-line-20">
+ <rect x="0" y="489.5" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-setup-check-all-params-in-groups-line-21">
+ <rect x="0" y="513.9" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-setup-check-all-params-in-groups-line-22">
+ <rect x="0" y="538.3" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-setup-check-all-params-in-groups-line-23">
+ <rect x="0" y="562.7" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-setup-check-all-params-in-groups-line-24">
+ <rect x="0" y="587.1" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-setup-check-all-params-in-groups-line-25">
+ <rect x="0" y="611.5" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-setup-check-all-params-in-groups-line-26">
+ <rect x="0" y="635.9" width="1464" height="24.65"/>
+ </clipPath>
+ </defs>
+
+ <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="731.2" rx="8"/><text class="breeze-setup-check-all-params-in-groups-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: setup check-all-params-in-groups</text>
+ <g transform="translate(26,22)">
+ <circle cx="0" cy="0" r="7" fill="#ff5f57"/>
+ <circle cx="22" cy="0" r="7" fill="#febc2e"/>
+ <circle cx="44" cy="0" r="7" fill="#28c840"/>
+ </g>
+
+ <g transform="translate(9, 41)" clip-path="url(#breeze-setup-check-all-params-in-groups-clip-terminal)">
+
+ <g class="breeze-setup-check-all-params-in-groups-matrix">
+ <text class="breeze-setup-check-all-params-in-groups-r2" x="1464" y="20" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-0)">
+</text><text class="breeze-setup-check-all-params-in-groups-r3" x="12.2" y="44.4" textLength="85.4" clip-path="url(#breeze-setup-check-all-params-in-groups-line-1)">Usage: </text><text class="breeze-setup-check-all-params-in-groups-r1" x="97.6" y="44.4" textLength="500.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-1)">breeze setup check-all-params-in-groups [</text><text class="breeze-setup-check-all-params-in-groups-r4" x="597.8" y="44.4" textLength [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r2" x="1464" y="68.8" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-2)">
+</text><text class="breeze-setup-check-all-params-in-groups-r2" x="12.2" y="93.2" textLength="536.8" clip-path="url(#breeze-setup-check-all-params-in-groups-line-3)">Check that all parameters are put in groups.</text><text class="breeze-setup-check-all-params-in-groups-r2" x="1464" y="93.2" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-3)">
+</text><text class="breeze-setup-check-all-params-in-groups-r2" x="1464" y="117.6" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-4)">
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="142" textLength="24.4" clip-path="url(#breeze-setup-check-all-params-in-groups-line-5)">╭─</text><text class="breeze-setup-check-all-params-in-groups-r5" x="24.4" y="142" textLength="414.8" clip-path="url(#breeze-setup-check-all-params-in-groups-line-5)"> Check all params in groups flags </text><text class="breeze-setup-check-all-params-in-groups-r5" x="439.2" y="142" textLength="1 [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="166.4" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-6)">│</text><text class="breeze-setup-check-all-params-in-groups-r4" x="24.4" y="166.4" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-6)">-</text><text class="breeze-setup-check-all-params-in-groups-r4" x="36.6" y="166.4" textLength="97.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="190.8" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-7)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="190.8" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-7)">(main | build-docs | ci:find-newer-dependencies | ci:fix-ownership | ci:free-space |     &# [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="215.2" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-8)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="215.2" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-8)">ci:get-workflow-info | ci:resource-check | ci:selective-check | ci | ci-image:build | ci-image:pull |  [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="239.6" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-9)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="239.6" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-9)">ci-image:verify | ci-image | cleanup | compile-www-assets | exec | k8s:build-k8s-image |    [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="264" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-10)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="264" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-10)">k8s:configure-cluster | k8s:create-cluster | k8s:delete-cluster | k8s:deploy-airflow | k8s:k9s |    [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-11)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="288.4" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-11)">k8s:logs | k8s:run-complete-tests | k8s:setup-env | k8s:shell | k8s:status | k8s:tests |   [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="312.8" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-12)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="312.8" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-12)">k8s:upload-k8s-image | k8s | prod-image:build | prod-image:pull | prod-image:verify | prod-image |  [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-13)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="337.2" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-13)">release-management:create-minor-branch | release-management:generate-constraints |          &# [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-14)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="361.6" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-14)">release-management:generate-issue-content | release-management:prepare-airflow-package |         &# [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="386" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-15)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="386" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-15)">release-management:prepare-provider-documentation | release-management:prepare-provider-packages |     </text><text class=" [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="410.4" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-16)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="410.4" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-16)">release-management:release-prod-images | release-management:start-rc-process |            [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="434.8" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-17)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="434.8" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-17)">release-management:start-release | release-management:verify-provider-packages | release-management |  </text><text class="b [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="459.2" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-18)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="459.2" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-18)">setup:autocomplete | setup:check-all-params-in-groups | setup:config | setup:regenerate-command-images </text><text class="b [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-19)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="483.6" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-19)">| setup:self-upgrade | setup:version | setup | shell | start-airflow | static-checks | stop [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="508" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-20)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="508" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-20)">testing:docker-compose-tests | testing:helm-tests | testing:integration-tests | testing:tests |      [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-21)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="183" y="532.4" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-21)">testing)                         & [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="556.8" textLength="1464" clip-path="url(#breeze-setup-check-all-params-in-groups-line-22)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-setup-check-all-params-in-groups-r2" x="1464" y="556.8" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-22)">
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="581.2" textLength="24.4" clip-path="url(#breeze-setup-check-all-params-in-groups-line-23)">╭─</text><text class="breeze-setup-check-all-params-in-groups-r5" x="24.4" y="581.2" textLength="195.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-23)"> Common options </text><text class="breeze-setup-check-all-params-in-groups-r5" x="219.6" y="581.2" textLength="1220" clip-path="url(#breeze-se [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="605.6" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-24)">│</text><text class="breeze-setup-check-all-params-in-groups-r4" x="24.4" y="605.6" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-24)">-</text><text class="breeze-setup-check-all-params-in-groups-r4" x="36.6" y="605.6" textLength="97.6" clip-path="url(#breeze-setup-check-all-params-in-groups-li [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="630" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-25)">│</text><text class="breeze-setup-check-all-params-in-groups-r4" x="24.4" y="630" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-25)">-</text><text class="breeze-setup-check-all-params-in-groups-r4" x="36.6" y="630" textLength="48.8" clip-path="url(#breeze-setup-check-all-params-in-groups-line-25) [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="654.4" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-26)">│</text><text class="breeze-setup-check-all-params-in-groups-r4" x="24.4" y="654.4" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-26)">-</text><text class="breeze-setup-check-all-params-in-groups-r4" x="36.6" y="654.4" textLength="61" clip-path="url(#breeze-setup-check-all-params-in-groups-line [...]
+</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="678.8" textLength="1464" clip-path="url(#breeze-setup-check-all-params-in-groups-line-27)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-setup-check-all-params-in-groups-r2" x="1464" y="678.8" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-27)">
+</text>
+ </g>
+ </g>
+</svg>
diff --git a/images/breeze/output_setup_regenerate-command-images.svg b/images/breeze/output_setup_regenerate-command-images.svg
index 3ecbb459e5..c0f2199538 100644
--- a/images/breeze/output_setup_regenerate-command-images.svg
+++ b/images/breeze/output_setup_regenerate-command-images.svg
@@ -1,4 +1,4 @@
-<svg class="rich-terminal" viewBox="0 0 1482 757.5999999999999" xmlns="http://www.w3.org/2000/svg">
+<svg class="rich-terminal" viewBox="0 0 1482 806.4" xmlns="http://www.w3.org/2000/svg">
<!-- Generated with Rich https://www.textualize.io -->
<style>
@@ -35,15 +35,15 @@
.breeze-setup-regenerate-command-images-r1 { fill: #c5c8c6;font-weight: bold }
.breeze-setup-regenerate-command-images-r2 { fill: #c5c8c6 }
.breeze-setup-regenerate-command-images-r3 { fill: #d0b344;font-weight: bold }
-.breeze-setup-regenerate-command-images-r4 { fill: #868887 }
-.breeze-setup-regenerate-command-images-r5 { fill: #68a0b3;font-weight: bold }
+.breeze-setup-regenerate-command-images-r4 { fill: #68a0b3;font-weight: bold }
+.breeze-setup-regenerate-command-images-r5 { fill: #868887 }
.breeze-setup-regenerate-command-images-r6 { fill: #8d7b39 }
.breeze-setup-regenerate-command-images-r7 { fill: #98a84b;font-weight: bold }
</style>
<defs>
<clipPath id="breeze-setup-regenerate-command-images-clip-terminal">
- <rect x="0" y="0" width="1463.0" height="706.5999999999999" />
+ <rect x="0" y="0" width="1463.0" height="755.4" />
</clipPath>
<clipPath id="breeze-setup-regenerate-command-images-line-0">
<rect x="0" y="1.5" width="1464" height="24.65"/>
@@ -129,9 +129,15 @@
<clipPath id="breeze-setup-regenerate-command-images-line-27">
<rect x="0" y="660.3" width="1464" height="24.65"/>
</clipPath>
+<clipPath id="breeze-setup-regenerate-command-images-line-28">
+ <rect x="0" y="684.7" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-setup-regenerate-command-images-line-29">
+ <rect x="0" y="709.1" width="1464" height="24.65"/>
+ </clipPath>
</defs>
- <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="755.6" rx="8"/><text class="breeze-setup-regenerate-command-images-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: setup regenerate-command-images</text>
+ <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="804.4" rx="8"/><text class="breeze-setup-regenerate-command-images-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: setup regenerate-command-images</text>
<g transform="translate(26,22)">
<circle cx="0" cy="0" r="7" fill="#ff5f57"/>
<circle cx="22" cy="0" r="7" fill="#febc2e"/>
@@ -142,34 +148,36 @@
<g class="breeze-setup-regenerate-command-images-matrix">
<text class="breeze-setup-regenerate-command-images-r2" x="1464" y="20" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-0)">
-</text><text class="breeze-setup-regenerate-command-images-r3" x="12.2" y="44.4" textLength="85.4" clip-path="url(#breeze-setup-regenerate-command-images-line-1)">Usage: </text><text class="breeze-setup-regenerate-command-images-r1" x="97.6" y="44.4" textLength="585.6" clip-path="url(#breeze-setup-regenerate-command-images-line-1)">breeze setup regenerate-command-images [OPTIONS]</text><text class="breeze-setup-regenerate-command-images-r2" x="1464" y="44.4" textLengt [...]
+</text><text class="breeze-setup-regenerate-command-images-r3" x="12.2" y="44.4" textLength="85.4" clip-path="url(#breeze-setup-regenerate-command-images-line-1)">Usage: </text><text class="breeze-setup-regenerate-command-images-r1" x="97.6" y="44.4" textLength="488" clip-path="url(#breeze-setup-regenerate-command-images-line-1)">breeze setup regenerate-command-images [</text><text class="breeze-setup-regenerate-command-images-r4" x="585.6" y="44.4" textLength="85.4" [...]
</text><text class="breeze-setup-regenerate-command-images-r2" x="1464" y="68.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-2)">
</text><text class="breeze-setup-regenerate-command-images-r2" x="12.2" y="93.2" textLength="402.6" clip-path="url(#breeze-setup-regenerate-command-images-line-3)">Regenerate breeze command images.</text><text class="breeze-setup-regenerate-command-images-r2" x="1464" y="93.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-3)">
</text><text class="breeze-setup-regenerate-command-images-r2" x="1464" y="117.6" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-4)">
-</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="142" textLength="24.4" clip-path="url(#breeze-setup-regenerate-command-images-line-5)">╭─</text><text class="breeze-setup-regenerate-command-images-r4" x="24.4" y="142" textLength="329.4" clip-path="url(#breeze-setup-regenerate-command-images-line-5)"> Image regeneration option </text><text class="breeze-setup-regenerate-command-images-r4" x="353.8" y="142" textLength="1085.8" clip-path="url(#bree [...]
-</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="166.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-6)">│</text><text class="breeze-setup-regenerate-command-images-r5" x="24.4" y="166.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-6)">-</text><text class="breeze-setup-regenerate-command-images-r5" x="36.6" y="166.4" textLength="73.2" clip-path="url(#breeze-setup-regenerate-command-images-line-6)">- [...]
-</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="190.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-7)">│</text><text class="breeze-setup-regenerate-command-images-r5" x="24.4" y="190.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-7)">-</text><text class="breeze-setup-regenerate-command-images-r5" x="36.6" y="190.8" textLength="97.6" clip-path="url(#breeze-setup-regenerate-command-images-line-7)">- [...]
-</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="215.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-8)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="215.2" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-8)">(main | build-docs | ci:find-newer-dependencies | ci:fix-ownership | ci:free-space |       [...]
-</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="239.6" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-9)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="239.6" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-9)">ci:get-workflow-info | ci:resource-check | ci:selective-check | ci | ci-image:build | ci-image:pull </text><te [...]
-</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="264" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-10)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="264" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-10)">| ci-image:verify | ci-image | cleanup | compile-www-assets | exec | k8s:build-k8s-image |    [...]
-</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-11)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="288.4" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-11)">k8s:configure-cluster | k8s:create-cluster | k8s:delete-cluster | k8s:deploy-airflow | k8s:k9s |    [...]
-</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="312.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-12)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="312.8" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-12)">k8s:logs | k8s:run-complete-tests | k8s:setup-env | k8s:shell | k8s:status | k8s:tests |   &# [...]
-</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-13)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="337.2" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-13)">k8s:upload-k8s-image | k8s | prod-image:build | prod-image:pull | prod-image:verify | prod-image |  [...]
-</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-14)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="361.6" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-14)">release-management:generate-constraints | release-management:generate-issue-content |           [...]
-</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="386" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-15)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="386" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-15)">release-management:prepare-airflow-package | release-management:prepare-provider-documentation |    </text><text class="breeze-setup [...]
-</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="410.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-16)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="410.4" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-16)">release-management:prepare-provider-packages | release-management:release-prod-images |          & [...]
-</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="434.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-17)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="434.8" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-17)">release-management:verify-provider-packages | release-management | setup:autocomplete | setup:config</text><text class="breeze-setup- [...]
-</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="459.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-18)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="459.2" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-18)">| setup:regenerate-command-images | setup:self-upgrade | setup:version | setup | shell |    & [...]
-</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-19)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="483.6" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-19)">start-airflow | static-checks | stop | testing:docker-compose-tests | testing:helm-tests |     [...]
-</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="508" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-20)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="508" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-20)">testing:integration-tests | testing:tests | testing)                 [...]
-</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-21)">│</text><text class="breeze-setup-regenerate-command-images-r5" x="24.4" y="532.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-21)">-</text><text class="breeze-setup-regenerate-command-images-r5" x="36.6" y="532.4" textLength="73.2" clip-path="url(#breeze-setup-regenerate-command-images-line-21) [...]
-</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="556.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-22)">│</text><text class="breeze-setup-regenerate-command-images-r2" x="219.6" y="556.8" textLength="170.8" clip-path="url(#breeze-setup-regenerate-command-images-line-22)">together with </text><text class="breeze-setup-regenerate-command-images-r5" x="390.4" y="556.8" textLength="12.2" clip-path="url(#breeze-setup-regenera [...]
-</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="581.2" textLength="1464" clip-path="url(#breeze-setup-regenerate-command-images-line-23)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-setup-regenerate-command-images-r2" x="1464" y="581.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-23)">
-</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="605.6" textLength="24.4" clip-path="url(#breeze-setup-regenerate-command-images-line-24)">╭─</text><text class="breeze-setup-regenerate-command-images-r4" x="24.4" y="605.6" textLength="195.2" clip-path="url(#breeze-setup-regenerate-command-images-line-24)"> Common options </text><text class="breeze-setup-regenerate-command-images-r4" x="219.6" y="605.6" textLength="1220" clip-path="url(#breeze-setup-r [...]
-</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="630" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-25)">│</text><text class="breeze-setup-regenerate-command-images-r5" x="24.4" y="630" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-25)">-</text><text class="breeze-setup-regenerate-command-images-r5" x="36.6" y="630" textLength="97.6" clip-path="url(#breeze-setup-regenerate-command-images-line-25)">-ver [...]
-</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="654.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-26)">│</text><text class="breeze-setup-regenerate-command-images-r5" x="24.4" y="654.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-26)">-</text><text class="breeze-setup-regenerate-command-images-r5" x="36.6" y="654.4" textLength="48.8" clip-path="url(#breeze-setup-regenerate-command-images-line-26) [...]
-</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="678.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-27)">│</text><text class="breeze-setup-regenerate-command-images-r5" x="24.4" y="678.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-27)">-</text><text class="breeze-setup-regenerate-command-images-r5" x="36.6" y="678.8" textLength="61" clip-path="url(#breeze-setup-regenerate-command-images-line-27)"> [...]
-</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="703.2" textLength="1464" clip-path="url(#breeze-setup-regenerate-command-images-line-28)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-setup-regenerate-command-images-r2" x="1464" y="703.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-28)">
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="142" textLength="24.4" clip-path="url(#breeze-setup-regenerate-command-images-line-5)">╭─</text><text class="breeze-setup-regenerate-command-images-r5" x="24.4" y="142" textLength="329.4" clip-path="url(#breeze-setup-regenerate-command-images-line-5)"> Image regeneration option </text><text class="breeze-setup-regenerate-command-images-r5" x="353.8" y="142" textLength="1085.8" clip-path="url(#bree [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="166.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-6)">│</text><text class="breeze-setup-regenerate-command-images-r4" x="24.4" y="166.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-6)">-</text><text class="breeze-setup-regenerate-command-images-r4" x="36.6" y="166.4" textLength="73.2" clip-path="url(#breeze-setup-regenerate-command-images-line-6)">- [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="190.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-7)">│</text><text class="breeze-setup-regenerate-command-images-r4" x="24.4" y="190.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-7)">-</text><text class="breeze-setup-regenerate-command-images-r4" x="36.6" y="190.8" textLength="97.6" clip-path="url(#breeze-setup-regenerate-command-images-line-7)">- [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="215.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-8)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="215.2" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-8)">(main | build-docs | ci:find-newer-dependencies | ci:fix-ownership | ci:free-space |       [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="239.6" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-9)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="239.6" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-9)">ci:get-workflow-info | ci:resource-check | ci:selective-check | ci | ci-image:build | ci-image:pull </text><te [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="264" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-10)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="264" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-10)">| ci-image:verify | ci-image | cleanup | compile-www-assets | exec | k8s:build-k8s-image |    [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-11)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="288.4" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-11)">k8s:configure-cluster | k8s:create-cluster | k8s:delete-cluster | k8s:deploy-airflow | k8s:k9s |    [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="312.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-12)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="312.8" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-12)">k8s:logs | k8s:run-complete-tests | k8s:setup-env | k8s:shell | k8s:status | k8s:tests |   &# [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-13)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="337.2" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-13)">k8s:upload-k8s-image | k8s | prod-image:build | prod-image:pull | prod-image:verify | prod-image |  [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-14)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="361.6" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-14)">release-management:create-minor-branch | release-management:generate-constraints |            [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="386" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-15)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="386" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-15)">release-management:generate-issue-content | release-management:prepare-airflow-package |           [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="410.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-16)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="410.4" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-16)">release-management:prepare-provider-documentation | release-management:prepare-provider-packages |  </text><text class="breeze-setup-regen [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="434.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-17)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="434.8" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-17)">release-management:release-prod-images | release-management:start-rc-process |            [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="459.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-18)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="459.2" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-18)">release-management:start-release | release-management:verify-provider-packages | release-management </text><text class="breeze-setup-regen [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-19)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="483.6" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-19)">| setup:autocomplete | setup:check-all-params-in-groups | setup:config |          & [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="508" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-20)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="508" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-20)">setup:regenerate-command-images | setup:self-upgrade | setup:version | setup | shell | start-airflow</text><text clas [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-21)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="532.4" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-21)">| static-checks | stop | testing:docker-compose-tests | testing:helm-tests |         [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="556.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-22)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="556.8" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-22)">testing:integration-tests | testing:tests | testing)               & [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="581.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-23)">│</text><text class="breeze-setup-regenerate-command-images-r4" x="24.4" y="581.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-23)">-</text><text class="breeze-setup-regenerate-command-images-r4" x="36.6" y="581.2" textLength="73.2" clip-path="url(#breeze-setup-regenerate-command-images-line-23) [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="605.6" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-24)">│</text><text class="breeze-setup-regenerate-command-images-r2" x="219.6" y="605.6" textLength="170.8" clip-path="url(#breeze-setup-regenerate-command-images-line-24)">together with </text><text class="breeze-setup-regenerate-command-images-r4" x="390.4" y="605.6" textLength="12.2" clip-path="url(#breeze-setup-regenera [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="630" textLength="1464" clip-path="url(#breeze-setup-regenerate-command-images-line-25)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-setup-regenerate-command-images-r2" x="1464" y="630" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-25)">
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="654.4" textLength="24.4" clip-path="url(#breeze-setup-regenerate-command-images-line-26)">╭─</text><text class="breeze-setup-regenerate-command-images-r5" x="24.4" y="654.4" textLength="195.2" clip-path="url(#breeze-setup-regenerate-command-images-line-26)"> Common options </text><text class="breeze-setup-regenerate-command-images-r5" x="219.6" y="654.4" textLength="1220" clip-path="url(#breeze-setup-r [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="678.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-27)">│</text><text class="breeze-setup-regenerate-command-images-r4" x="24.4" y="678.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-27)">-</text><text class="breeze-setup-regenerate-command-images-r4" x="36.6" y="678.8" textLength="97.6" clip-path="url(#breeze-setup-regenerate-command-images-line-27) [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="703.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-28)">│</text><text class="breeze-setup-regenerate-command-images-r4" x="24.4" y="703.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-28)">-</text><text class="breeze-setup-regenerate-command-images-r4" x="36.6" y="703.2" textLength="48.8" clip-path="url(#breeze-setup-regenerate-command-images-line-28) [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="727.6" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-29)">│</text><text class="breeze-setup-regenerate-command-images-r4" x="24.4" y="727.6" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-29)">-</text><text class="breeze-setup-regenerate-command-images-r4" x="36.6" y="727.6" textLength="61" clip-path="url(#breeze-setup-regenerate-command-images-line-29)"> [...]
+</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="752" textLength="1464" clip-path="url(#breeze-setup-regenerate-command-images-line-30)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-setup-regenerate-command-images-r2" x="1464" y="752" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-30)">
</text>
</g>
</g>
[airflow] 11/37: More robust cleanup of executors in test_kubernetes_executor (#28281)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 389c074b82758ff56c0785ad19ee06ed1b9e860a
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sat Dec 10 17:09:06 2022 +0100
More robust cleanup of executors in test_kubernetes_executor (#28281)
As a follow up after #28047, this PR will make the test cleanup
more robust and resilient to any errors that might have caused
kubernetes_executors left behind.
wrapping start()/end() in try/finally will make the tests
completely resilient to cases where the asserts start to fail -
without those, any failure in tests would cause the same resource
leakage as we initially had when #28407 was iterated on.
(cherry picked from commit 3b203bcb676853bd642a01121988b1cbe929307d)
---
tests/executors/test_kubernetes_executor.py | 348 +++++++++++++++-------------
1 file changed, 193 insertions(+), 155 deletions(-)
diff --git a/tests/executors/test_kubernetes_executor.py b/tests/executors/test_kubernetes_executor.py
index 97619225e6..d1210765c0 100644
--- a/tests/executors/test_kubernetes_executor.py
+++ b/tests/executors/test_kubernetes_executor.py
@@ -162,9 +162,11 @@ class TestAirflowKubernetesScheduler:
kube_executor = KubernetesExecutor()
kube_executor.job_id = 1
kube_executor.start()
- kube_executor.kube_scheduler.delete_pod(pod_id, namespace)
-
- mock_delete_namespace.assert_called_with(pod_id, namespace, body=mock_client.V1DeleteOptions())
+ try:
+ kube_executor.kube_scheduler.delete_pod(pod_id, namespace)
+ mock_delete_namespace.assert_called_with(pod_id, namespace, body=mock_client.V1DeleteOptions())
+ finally:
+ kube_executor.end()
@unittest.skipIf(AirflowKubernetesScheduler is None, "kubernetes python package is not installed")
@mock.patch("airflow.executors.kubernetes_executor.get_kube_client")
@@ -203,9 +205,11 @@ class TestAirflowKubernetesScheduler:
kube_executor = KubernetesExecutor()
kube_executor.job_id = 1
kube_executor.start()
-
- kube_executor.kube_scheduler.delete_pod(pod_id, namespace)
- mock_delete_namespace.assert_called_with(pod_id, namespace, body=mock_client.V1DeleteOptions())
+ try:
+ kube_executor.kube_scheduler.delete_pod(pod_id, namespace)
+ mock_delete_namespace.assert_called_with(pod_id, namespace, body=mock_client.V1DeleteOptions())
+ finally:
+ kube_executor.end()
class TestKubernetesExecutor:
@@ -266,32 +270,35 @@ class TestKubernetesExecutor:
with conf_vars(config):
kubernetes_executor = self.kubernetes_executor
kubernetes_executor.start()
- # Execute a task while the Api Throws errors
- try_number = 1
- task_instance_key = TaskInstanceKey("dag", "task", "run_id", try_number)
- kubernetes_executor.execute_async(
- key=task_instance_key,
- queue=None,
- command=["airflow", "tasks", "run", "true", "some_parameter"],
- )
- kubernetes_executor.sync()
+ try:
+ # Execute a task while the Api Throws errors
+ try_number = 1
+ task_instance_key = TaskInstanceKey("dag", "task", "run_id", try_number)
+ kubernetes_executor.execute_async(
+ key=task_instance_key,
+ queue=None,
+ command=["airflow", "tasks", "run", "true", "some_parameter"],
+ )
+ kubernetes_executor.sync()
- assert mock_kube_client.create_namespaced_pod.call_count == 1
+ assert mock_kube_client.create_namespaced_pod.call_count == 1
- if should_requeue:
- assert not kubernetes_executor.task_queue.empty()
+ if should_requeue:
+ assert not kubernetes_executor.task_queue.empty()
- # Disable the ApiException
- mock_kube_client.create_namespaced_pod.side_effect = None
+ # Disable the ApiException
+ mock_kube_client.create_namespaced_pod.side_effect = None
- # Execute the task without errors should empty the queue
- mock_kube_client.create_namespaced_pod.reset_mock()
- kubernetes_executor.sync()
- assert mock_kube_client.create_namespaced_pod.called
- assert kubernetes_executor.task_queue.empty()
- else:
- assert kubernetes_executor.task_queue.empty()
- assert kubernetes_executor.event_buffer[task_instance_key][0] == State.FAILED
+ # Execute the task without errors should empty the queue
+ mock_kube_client.create_namespaced_pod.reset_mock()
+ kubernetes_executor.sync()
+ assert mock_kube_client.create_namespaced_pod.called
+ assert kubernetes_executor.task_queue.empty()
+ else:
+ assert kubernetes_executor.task_queue.empty()
+ assert kubernetes_executor.event_buffer[task_instance_key][0] == State.FAILED
+ finally:
+ kubernetes_executor.end()
@pytest.mark.skipif(
AirflowKubernetesScheduler is None, reason="kubernetes python package is not installed"
@@ -311,23 +318,26 @@ class TestKubernetesExecutor:
kubernetes_executor = self.kubernetes_executor
kubernetes_executor.start()
- try_number = 1
- task_instance_key = TaskInstanceKey("dag", "task", "run_id", try_number)
- kubernetes_executor.execute_async(
- key=task_instance_key,
- queue=None,
- command=["airflow", "tasks", "run", "true", "some_parameter"],
- )
- kubernetes_executor.sync()
+ try:
+ try_number = 1
+ task_instance_key = TaskInstanceKey("dag", "task", "run_id", try_number)
+ kubernetes_executor.execute_async(
+ key=task_instance_key,
+ queue=None,
+ command=["airflow", "tasks", "run", "true", "some_parameter"],
+ )
+ kubernetes_executor.sync()
- # The pod_mutation_hook should have been called once.
- assert mock_pmh.call_count == 1
- # There should be no pod creation request sent
- assert mock_kube_client.create_namespaced_pod.call_count == 0
- # The task is not re-queued and there is the failed record in event_buffer
- assert kubernetes_executor.task_queue.empty()
- assert kubernetes_executor.event_buffer[task_instance_key][0] == State.FAILED
- assert kubernetes_executor.event_buffer[task_instance_key][1].args[0] == exception_in_pmh
+ # The pod_mutation_hook should have been called once.
+ assert mock_pmh.call_count == 1
+ # There should be no pod creation request sent
+ assert mock_kube_client.create_namespaced_pod.call_count == 0
+ # The task is not re-queued and there is the failed record in event_buffer
+ assert kubernetes_executor.task_queue.empty()
+ assert kubernetes_executor.event_buffer[task_instance_key][0] == State.FAILED
+ assert kubernetes_executor.event_buffer[task_instance_key][1].args[0] == exception_in_pmh
+ finally:
+ kubernetes_executor.end()
@pytest.mark.skipif(
AirflowKubernetesScheduler is None, reason="kubernetes python package is not installed"
@@ -351,19 +361,22 @@ class TestKubernetesExecutor:
with conf_vars(config):
kubernetes_executor = self.kubernetes_executor
kubernetes_executor.start()
- # Execute a task while the Api Throws errors
- try_number = 1
- task_instance_key = TaskInstanceKey("dag", "task", "run_id", try_number)
- kubernetes_executor.execute_async(
- key=task_instance_key,
- queue=None,
- command=["airflow", "tasks", "run", "true", "some_parameter"],
- )
- kubernetes_executor.sync()
+ try:
+ # Execute a task while the Api Throws errors
+ try_number = 1
+ task_instance_key = TaskInstanceKey("dag", "task", "run_id", try_number)
+ kubernetes_executor.execute_async(
+ key=task_instance_key,
+ queue=None,
+ command=["airflow", "tasks", "run", "true", "some_parameter"],
+ )
+ kubernetes_executor.sync()
- assert kubernetes_executor.task_queue.empty()
- assert kubernetes_executor.event_buffer[task_instance_key][0] == State.FAILED
- assert kubernetes_executor.event_buffer[task_instance_key][1].args[0] == fail_msg
+ assert kubernetes_executor.task_queue.empty()
+ assert kubernetes_executor.event_buffer[task_instance_key][0] == State.FAILED
+ assert kubernetes_executor.event_buffer[task_instance_key][1].args[0] == fail_msg
+ finally:
+ kubernetes_executor.end()
@mock.patch("airflow.executors.kubernetes_executor.KubeConfig")
@mock.patch("airflow.executors.kubernetes_executor.KubernetesExecutor.sync")
@@ -384,20 +397,22 @@ class TestKubernetesExecutor:
def test_invalid_executor_config(self, mock_get_kube_client, mock_kubernetes_job_watcher):
executor = self.kubernetes_executor
executor.start()
+ try:
+ assert executor.event_buffer == {}
+ executor.execute_async(
+ key=("dag", "task", datetime.utcnow(), 1),
+ queue=None,
+ command=["airflow", "tasks", "run", "true", "some_parameter"],
+ executor_config=k8s.V1Pod(
+ spec=k8s.V1PodSpec(
+ containers=[k8s.V1Container(name="base", image="myimage", image_pull_policy="Always")]
+ )
+ ),
+ )
- assert executor.event_buffer == {}
- executor.execute_async(
- key=("dag", "task", datetime.utcnow(), 1),
- queue=None,
- command=["airflow", "tasks", "run", "true", "some_parameter"],
- executor_config=k8s.V1Pod(
- spec=k8s.V1PodSpec(
- containers=[k8s.V1Container(name="base", image="myimage", image_pull_policy="Always")]
- )
- ),
- )
-
- assert list(executor.event_buffer.values())[0][1] == "Invalid executor_config passed"
+ assert list(executor.event_buffer.values())[0][1] == "Invalid executor_config passed"
+ finally:
+ executor.end()
@pytest.mark.execution_timeout(10)
@pytest.mark.skipif(
@@ -417,83 +432,88 @@ class TestKubernetesExecutor:
with conf_vars({("kubernetes", "pod_template_file"): ""}):
executor = self.kubernetes_executor
executor.start()
+ try:
+ assert executor.event_buffer == {}
+ assert executor.task_queue.empty()
+
+ executor.execute_async(
+ key=TaskInstanceKey("dag", "task", "run_id", 1),
+ queue=None,
+ command=["airflow", "tasks", "run", "true", "some_parameter"],
+ executor_config={
+ "pod_template_file": template_file,
+ "pod_override": k8s.V1Pod(
+ metadata=k8s.V1ObjectMeta(labels={"release": "stable"}),
+ spec=k8s.V1PodSpec(
+ containers=[k8s.V1Container(name="base", image="airflow:3.6")],
+ ),
+ ),
+ },
+ )
- assert executor.event_buffer == {}
- assert executor.task_queue.empty()
-
- executor.execute_async(
- key=TaskInstanceKey("dag", "task", "run_id", 1),
- queue=None,
- command=["airflow", "tasks", "run", "true", "some_parameter"],
- executor_config={
- "pod_template_file": template_file,
- "pod_override": k8s.V1Pod(
- metadata=k8s.V1ObjectMeta(labels={"release": "stable"}),
+ assert not executor.task_queue.empty()
+ task = executor.task_queue.get_nowait()
+ _, _, expected_executor_config, expected_pod_template_file = task
+ executor.task_queue.task_done()
+ # Test that the correct values have been put to queue
+ assert expected_executor_config.metadata.labels == {"release": "stable"}
+ assert expected_pod_template_file == template_file
+
+ self.kubernetes_executor.kube_scheduler.run_next(task)
+ mock_run_pod_async.assert_called_once_with(
+ k8s.V1Pod(
+ api_version="v1",
+ kind="Pod",
+ metadata=k8s.V1ObjectMeta(
+ name=mock.ANY,
+ namespace="default",
+ annotations={
+ "dag_id": "dag",
+ "run_id": "run_id",
+ "task_id": "task",
+ "try_number": "1",
+ },
+ labels={
+ "airflow-worker": "5",
+ "airflow_version": mock.ANY,
+ "dag_id": "dag",
+ "run_id": "run_id",
+ "kubernetes_executor": "True",
+ "mylabel": "foo",
+ "release": "stable",
+ "task_id": "task",
+ "try_number": "1",
+ },
+ ),
spec=k8s.V1PodSpec(
- containers=[k8s.V1Container(name="base", image="airflow:3.6")],
+ containers=[
+ k8s.V1Container(
+ name="base",
+ image="airflow:3.6",
+ args=["airflow", "tasks", "run", "true", "some_parameter"],
+ env=[k8s.V1EnvVar(name="AIRFLOW_IS_K8S_EXECUTOR_POD", value="True")],
+ )
+ ],
+ image_pull_secrets=[k8s.V1LocalObjectReference(name="airflow-registry")],
+ scheduler_name="default-scheduler",
+ security_context=k8s.V1PodSecurityContext(fs_group=50000, run_as_user=50000),
),
- ),
- },
- )
-
- assert not executor.task_queue.empty()
- task = executor.task_queue.get_nowait()
- _, _, expected_executor_config, expected_pod_template_file = task
-
- # Test that the correct values have been put to queue
- assert expected_executor_config.metadata.labels == {"release": "stable"}
- assert expected_pod_template_file == template_file
-
- self.kubernetes_executor.kube_scheduler.run_next(task)
- mock_run_pod_async.assert_called_once_with(
- k8s.V1Pod(
- api_version="v1",
- kind="Pod",
- metadata=k8s.V1ObjectMeta(
- name=mock.ANY,
- namespace="default",
- annotations={
- "dag_id": "dag",
- "run_id": "run_id",
- "task_id": "task",
- "try_number": "1",
- },
- labels={
- "airflow-worker": "5",
- "airflow_version": mock.ANY,
- "dag_id": "dag",
- "run_id": "run_id",
- "kubernetes_executor": "True",
- "mylabel": "foo",
- "release": "stable",
- "task_id": "task",
- "try_number": "1",
- },
- ),
- spec=k8s.V1PodSpec(
- containers=[
- k8s.V1Container(
- name="base",
- image="airflow:3.6",
- args=["airflow", "tasks", "run", "true", "some_parameter"],
- env=[k8s.V1EnvVar(name="AIRFLOW_IS_K8S_EXECUTOR_POD", value="True")],
- )
- ],
- image_pull_secrets=[k8s.V1LocalObjectReference(name="airflow-registry")],
- scheduler_name="default-scheduler",
- security_context=k8s.V1PodSecurityContext(fs_group=50000, run_as_user=50000),
- ),
+ )
)
- )
+ finally:
+ executor.end()
@mock.patch("airflow.executors.kubernetes_executor.KubernetesJobWatcher")
@mock.patch("airflow.executors.kubernetes_executor.get_kube_client")
def test_change_state_running(self, mock_get_kube_client, mock_kubernetes_job_watcher):
executor = self.kubernetes_executor
executor.start()
- key = ("dag_id", "task_id", "run_id", "try_number1")
- executor._change_state(key, State.RUNNING, "pod_id", "default")
- assert executor.event_buffer[key][0] == State.RUNNING
+ try:
+ key = ("dag_id", "task_id", "run_id", "try_number1")
+ executor._change_state(key, State.RUNNING, "pod_id", "default")
+ assert executor.event_buffer[key][0] == State.RUNNING
+ finally:
+ executor.end()
@mock.patch("airflow.executors.kubernetes_executor.KubernetesJobWatcher")
@mock.patch("airflow.executors.kubernetes_executor.get_kube_client")
@@ -501,10 +521,13 @@ class TestKubernetesExecutor:
def test_change_state_success(self, mock_delete_pod, mock_get_kube_client, mock_kubernetes_job_watcher):
executor = self.kubernetes_executor
executor.start()
- key = ("dag_id", "task_id", "run_id", "try_number2")
- executor._change_state(key, State.SUCCESS, "pod_id", "default")
- assert executor.event_buffer[key][0] == State.SUCCESS
- mock_delete_pod.assert_called_once_with("pod_id", "default")
+ try:
+ key = ("dag_id", "task_id", "run_id", "try_number2")
+ executor._change_state(key, State.SUCCESS, "pod_id", "default")
+ assert executor.event_buffer[key][0] == State.SUCCESS
+ mock_delete_pod.assert_called_once_with("pod_id", "default")
+ finally:
+ executor.end()
@mock.patch("airflow.executors.kubernetes_executor.KubernetesJobWatcher")
@mock.patch("airflow.executors.kubernetes_executor.get_kube_client")
@@ -516,10 +539,13 @@ class TestKubernetesExecutor:
executor.kube_config.delete_worker_pods = False
executor.kube_config.delete_worker_pods_on_failure = False
executor.start()
- key = ("dag_id", "task_id", "run_id", "try_number3")
- executor._change_state(key, State.FAILED, "pod_id", "default")
- assert executor.event_buffer[key][0] == State.FAILED
- mock_delete_pod.assert_not_called()
+ try:
+ key = ("dag_id", "task_id", "run_id", "try_number3")
+ executor._change_state(key, State.FAILED, "pod_id", "default")
+ assert executor.event_buffer[key][0] == State.FAILED
+ mock_delete_pod.assert_not_called()
+ finally:
+ executor.end()
@mock.patch("airflow.executors.kubernetes_executor.KubernetesJobWatcher")
@mock.patch("airflow.executors.kubernetes_executor.get_kube_client")
@@ -532,10 +558,13 @@ class TestKubernetesExecutor:
executor.kube_config.delete_worker_pods_on_failure = False
executor.start()
- key = ("dag_id", "task_id", "run_id", "try_number2")
- executor._change_state(key, State.SUCCESS, "pod_id", "default")
- assert executor.event_buffer[key][0] == State.SUCCESS
- mock_delete_pod.assert_not_called()
+ try:
+ key = ("dag_id", "task_id", "run_id", "try_number2")
+ executor._change_state(key, State.SUCCESS, "pod_id", "default")
+ assert executor.event_buffer[key][0] == State.SUCCESS
+ mock_delete_pod.assert_not_called()
+ finally:
+ executor.end()
@mock.patch("airflow.executors.kubernetes_executor.KubernetesJobWatcher")
@mock.patch("airflow.executors.kubernetes_executor.get_kube_client")
@@ -547,10 +576,13 @@ class TestKubernetesExecutor:
executor.kube_config.delete_worker_pods_on_failure = True
executor.start()
- key = ("dag_id", "task_id", "run_id", "try_number2")
- executor._change_state(key, State.FAILED, "pod_id", "test-namespace")
- assert executor.event_buffer[key][0] == State.FAILED
- mock_delete_pod.assert_called_once_with("pod_id", "test-namespace")
+ try:
+ key = ("dag_id", "task_id", "run_id", "try_number2")
+ executor._change_state(key, State.FAILED, "pod_id", "test-namespace")
+ assert executor.event_buffer[key][0] == State.FAILED
+ mock_delete_pod.assert_called_once_with("pod_id", "test-namespace")
+ finally:
+ executor.end()
@mock.patch("airflow.executors.kubernetes_executor.KubernetesExecutor.adopt_launched_task")
@mock.patch("airflow.executors.kubernetes_executor.KubernetesExecutor._adopt_completed_pods")
@@ -789,8 +821,11 @@ class TestKubernetesExecutor:
executor = KubernetesExecutor()
executor.job_id = 123
executor.start()
- assert 2 == len(executor.event_scheduler.queue)
- executor._check_worker_pods_pending_timeout()
+ try:
+ assert 2 == len(executor.event_scheduler.queue)
+ executor._check_worker_pods_pending_timeout()
+ finally:
+ executor.end()
mock_kube_client.list_namespaced_pod.assert_called_once_with(
"mynamespace",
@@ -831,7 +866,10 @@ class TestKubernetesExecutor:
executor = KubernetesExecutor()
executor.job_id = 123
executor.start()
- executor._check_worker_pods_pending_timeout()
+ try:
+ executor._check_worker_pods_pending_timeout()
+ finally:
+ executor.end()
mock_kube_client.list_pod_for_all_namespaces.assert_called_once_with(
field_selector="status.phase=Pending",
[airflow] 14/37: clarify that the total_entries property isn't impact by pagination (#28867)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 934e169ed14dc8538bca73577172ca19d0664726
Author: Steve Zhang <86...@users.noreply.github.com>
AuthorDate: Thu Jan 12 08:34:07 2023 -0800
clarify that the total_entries property isn't impact by pagination (#28867)
Co-authored-by: Ash Berlin-Taylor <as...@apache.org>
(cherry picked from commit 952952602161d353effa877fbb091cbfcf56068b)
---
airflow/api_connexion/openapi/v1.yaml | 4 +++-
airflow/www/static/js/types/api-generated.ts | 5 ++++-
2 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/airflow/api_connexion/openapi/v1.yaml b/airflow/api_connexion/openapi/v1.yaml
index 3f25b54640..9e4ec7e1d4 100644
--- a/airflow/api_connexion/openapi/v1.yaml
+++ b/airflow/api_connexion/openapi/v1.yaml
@@ -4492,7 +4492,9 @@ components:
properties:
total_entries:
type: integer
- description: Count of objects in the current result set.
+ description: |
+ Count of total objects in the current result set before pagination parameters
+ (limit, offset) are applied.
# Enums
TaskState:
diff --git a/airflow/www/static/js/types/api-generated.ts b/airflow/www/static/js/types/api-generated.ts
index 1ec47bd7d1..8841cd7225 100644
--- a/airflow/www/static/js/types/api-generated.ts
+++ b/airflow/www/static/js/types/api-generated.ts
@@ -2040,7 +2040,10 @@ export interface components {
};
/** @description Metadata about collection. */
CollectionInfo: {
- /** @description Count of objects in the current result set. */
+ /**
+ * @description Count of total objects in the current result set before pagination parameters
+ * (limit, offset) are applied.
+ */
total_entries?: number;
};
/**
[airflow] 29/37: Resolve all variables in pickled XCom iterator (#28982)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit e2cf93305803027d462373ca356bd5badee1128e
Author: Tzu-ping Chung <ur...@gmail.com>
AuthorDate: Wed Jan 18 18:19:35 2023 +0800
Resolve all variables in pickled XCom iterator (#28982)
(cherry picked from commit ccf53e167ea57716c76ec7ab5bd1223f0c0d47d3)
---
airflow/models/xcom.py | 7 +++++-
tests/conftest.py | 2 +-
tests/models/test_taskinstance.py | 49 ++++++++++++++++++++++++++++++++++++++-
3 files changed, 55 insertions(+), 3 deletions(-)
diff --git a/airflow/models/xcom.py b/airflow/models/xcom.py
index 3b43618424..6294fa3d7f 100644
--- a/airflow/models/xcom.py
+++ b/airflow/models/xcom.py
@@ -731,7 +731,12 @@ class LazyXComAccess(collections.abc.Sequence):
# do the same for count(), but I think it should be performant enough to
# calculate only that eagerly.
with self._get_bound_query() as query:
- statement = query.statement.compile(query.session.get_bind())
+ statement = query.statement.compile(
+ query.session.get_bind(),
+ # This inlines all the values into the SQL string to simplify
+ # cross-process commuinication as much as possible.
+ compile_kwargs={"literal_binds": True},
+ )
return (str(statement), query.count())
def __setstate__(self, state: Any) -> None:
diff --git a/tests/conftest.py b/tests/conftest.py
index d71d8eb0f0..3fb6d83489 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -291,7 +291,7 @@ def skip_if_not_marked_with_backend(selected_backend, item):
if selected_backend in backend_names:
return
pytest.skip(
- f"The test is skipped because it does not have the right backend marker "
+ f"The test is skipped because it does not have the right backend marker. "
f"Only tests marked with pytest.mark.backend('{selected_backend}') are run: {item}"
)
diff --git a/tests/models/test_taskinstance.py b/tests/models/test_taskinstance.py
index 17ce74178d..849358cc69 100644
--- a/tests/models/test_taskinstance.py
+++ b/tests/models/test_taskinstance.py
@@ -3592,6 +3592,40 @@ class TestMappedTaskInstanceReceiveValue:
assert out_lines == ["hello FOO", "goodbye FOO", "hello BAR", "goodbye BAR"]
+def _get_lazy_xcom_access_expected_sql_lines() -> list[str]:
+ backend = os.environ.get("BACKEND")
+ if backend == "mssql":
+ return [
+ "SELECT xcom.value",
+ "FROM xcom",
+ "WHERE xcom.dag_id = 'test_dag' AND xcom.run_id = 'test' "
+ "AND xcom.task_id = 't' AND xcom.map_index = -1 AND xcom.[key] = 'xxx'",
+ ]
+ elif backend == "mysql":
+ return [
+ "SELECT xcom.value",
+ "FROM xcom",
+ "WHERE xcom.dag_id = 'test_dag' AND xcom.run_id = 'test' "
+ "AND xcom.task_id = 't' AND xcom.map_index = -1 AND xcom.`key` = 'xxx'",
+ ]
+ elif backend == "postgres":
+ return [
+ "SELECT xcom.value",
+ "FROM xcom",
+ "WHERE xcom.dag_id = 'test_dag' AND xcom.run_id = 'test' "
+ "AND xcom.task_id = 't' AND xcom.map_index = -1 AND xcom.key = 'xxx'",
+ ]
+ elif backend == "sqlite":
+ return [
+ "SELECT xcom.value",
+ "FROM xcom",
+ "WHERE xcom.dag_id = 'test_dag' AND xcom.run_id = 'test' "
+ "AND xcom.task_id = 't' AND xcom.map_index = -1 AND xcom.\"key\" = 'xxx'",
+ ]
+ else:
+ raise RuntimeError(f"unknown backend {backend!r}")
+
+
def test_lazy_xcom_access_does_not_pickle_session(dag_maker, session):
with dag_maker(session=session):
EmptyOperator(task_id="t")
@@ -3599,9 +3633,22 @@ def test_lazy_xcom_access_does_not_pickle_session(dag_maker, session):
run: DagRun = dag_maker.create_dagrun()
run.get_task_instance("t", session=session).xcom_push("xxx", 123, session=session)
- original = LazyXComAccess.build_from_xcom_query(session.query(XCom))
+ query = session.query(XCom.value).filter_by(
+ dag_id=run.dag_id,
+ run_id=run.run_id,
+ task_id="t",
+ map_index=-1,
+ key="xxx",
+ )
+
+ original = LazyXComAccess.build_from_xcom_query(query)
processed = pickle.loads(pickle.dumps(original))
+ # After the object went through pickling, the underlying ORM query should be
+ # replaced by one backed by a literal SQL string with all variables binded.
+ sql_lines = [line.strip() for line in str(processed._query.statement.compile(None)).splitlines()]
+ assert sql_lines == _get_lazy_xcom_access_expected_sql_lines()
+
assert len(processed) == 1
assert list(processed) == [123]
[airflow] 17/37: Remove horizontal lines in TI logs (#28876)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit c22c42dbd43b0b34b2e42a195e124e8786ceae41
Author: Daniel Standish <15...@users.noreply.github.com>
AuthorDate: Wed Jan 11 23:40:17 2023 -0800
Remove horizontal lines in TI logs (#28876)
These are not helpful and certainly provide no meaningful information.
(cherry picked from commit c7f0aca52509a528460bda2c91bce761f09ee92c)
---
airflow/models/taskinstance.py | 8 --------
1 file changed, 8 deletions(-)
diff --git a/airflow/models/taskinstance.py b/airflow/models/taskinstance.py
index f237fe35e3..615a98f6cb 100644
--- a/airflow/models/taskinstance.py
+++ b/airflow/models/taskinstance.py
@@ -1218,9 +1218,6 @@ class TaskInstance(Base, LoggingMixin):
if not ignore_all_deps and not ignore_ti_state and self.state == State.SUCCESS:
Stats.incr("previously_succeeded", 1, 1)
- # TODO: Logging needs cleanup, not clear what is being printed
- hr_line_break = "\n" + ("-" * 80) # Line break
-
if not mark_success:
# Firstly find non-runnable and non-requeueable tis.
# Since mark_success is not set, we do nothing.
@@ -1263,7 +1260,6 @@ class TaskInstance(Base, LoggingMixin):
)
if not self.are_dependencies_met(dep_context=dep_context, session=session, verbose=True):
self.state = State.NONE
- self.log.warning(hr_line_break)
self.log.warning(
"Rescheduling due to concurrency limits reached "
"at task runtime. Attempt %s of "
@@ -1271,16 +1267,12 @@ class TaskInstance(Base, LoggingMixin):
self.try_number,
self.max_tries + 1,
)
- self.log.warning(hr_line_break)
self.queued_dttm = timezone.utcnow()
session.merge(self)
session.commit()
return False
- # print status message
- self.log.info(hr_line_break)
self.log.info("Starting attempt %s of %s", self.try_number, self.max_tries + 1)
- self.log.info(hr_line_break)
self._try_number += 1
if not test_mode:
[airflow] 25/37: Move project and license docs down in menu to start with developer-focused docs (#28956)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 9bf17670dcda219a03bab9174c04585d3798efc3
Author: Bas Harenslak <Ba...@users.noreply.github.com>
AuthorDate: Fri Jan 20 01:40:45 2023 +0100
Move project and license docs down in menu to start with developer-focused docs (#28956)
(cherry picked from commit d03b9a76914babef23b84acdb6061b0b93bdc2e3)
---
docs/apache-airflow/index.rst | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/docs/apache-airflow/index.rst b/docs/apache-airflow/index.rst
index eac8ef6cde..2381c7119a 100644
--- a/docs/apache-airflow/index.rst
+++ b/docs/apache-airflow/index.rst
@@ -131,8 +131,6 @@ so coding will always be required.
:caption: Content
Overview <self>
- project
- license
start
installation/index
tutorial/index
@@ -148,6 +146,8 @@ so coding will always be required.
Release Policies <release-process>
release_notes
privacy_notice
+ project
+ license
.. toctree::
:hidden:
[airflow] 28/37: Update go client gen command (#28967)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 20e6ee0d32899ec10bee2e321bbc27f7dd2315b8
Author: Pierre Jeambrun <pi...@gmail.com>
AuthorDate: Mon Jan 16 11:37:13 2023 +0100
Update go client gen command (#28967)
(cherry picked from commit 2eb4e974d561db01d0160148b3e372998409d377)
---
clients/gen/go.sh | 1 +
1 file changed, 1 insertion(+)
diff --git a/clients/gen/go.sh b/clients/gen/go.sh
index a7a820e3f6..e8e0f26b48 100755
--- a/clients/gen/go.sh
+++ b/clients/gen/go.sh
@@ -31,6 +31,7 @@ readonly VERSION
go_config=(
"packageVersion=${VERSION}"
"enumClassPrefix=true"
+ "structPrefix=true"
)
validate_input "$@"
[airflow] 02/37: Move provider issue generation to breeze (#28352)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit b42ace20df86f331fb1c56d69a7795a6c21bab63
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Wed Dec 14 09:20:27 2022 +0100
Move provider issue generation to breeze (#28352)
The exercise where another PMC performed a release process had
proven useful to show the value of Breeze in release management
commands - there were no problems with running the Breeze commands,
but for scripts put elsewhere (docs building, dev) it was a bit
problematic because each required own virtualenv.
This is the first PR to simplify that - one less venv for the
issue generation process - now it is moved to Breeze so no more
setup is needed.
(cherry picked from commit fd5846d256b6d269b160deb8df67cd3d914188e0)
---
BREEZE.rst | 9 +
dev/README_RELEASE_PROVIDER_PACKAGES.md | 4 +-
dev/breeze/README.md | 2 +-
dev/breeze/setup.cfg | 2 +
.../commands/release_management_commands.py | 199 ++++++++++++++++++++-
.../provider_issue_TEMPLATE.md.jinja2} | 14 +-
dev/provider_packages/prepare_provider_packages.py | 164 +----------------
images/breeze/output-commands-hash.txt | 15 +-
images/breeze/output-commands.svg | 88 ++++-----
images/breeze/output_release-management.svg | 58 +++---
...t_release-management_generate-issue-content.svg | 176 ++++++++++++++++++
images/breeze/output_setup.svg | 26 +--
.../output_setup_regenerate-command-images.svg | 54 +++---
.../run_prepare_provider_documentation.sh | 5 -
14 files changed, 518 insertions(+), 298 deletions(-)
diff --git a/BREEZE.rst b/BREEZE.rst
index 341f8b8dbe..d1528485fe 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -1589,6 +1589,15 @@ All the command parameters are here:
:width: 100%
:alt: Breeze verify-provider-packages
+Generating Provider Issue
+.........................
+
+You can use Breeze to generate a provider issue when you release new providers.
+
+.. image:: ./images/breeze/output_release-management_generate-issue-content.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/images/breeze/output_release-management_generate-issue-content.svg
+ :width: 100%
+ :alt: Breeze generate-issue-content
Preparing airflow packages
..........................
diff --git a/dev/README_RELEASE_PROVIDER_PACKAGES.md b/dev/README_RELEASE_PROVIDER_PACKAGES.md
index e484be18f8..a1accffdc4 100644
--- a/dev/README_RELEASE_PROVIDER_PACKAGES.md
+++ b/dev/README_RELEASE_PROVIDER_PACKAGES.md
@@ -391,7 +391,7 @@ set as your environment variable.
You can also pass the token as `--github-token` option in the script.
```shell script
-./dev/provider_packages/prepare_provider_packages.py generate-issue-content --only-available-in-dist
+breeze release-management generate-issue-content --only-available-in-dist
```
You can also generate the token by following
@@ -400,7 +400,7 @@ You can also generate the token by following
If you are preparing release for RC2/RC3 candidates, you should add `--suffix` parameter:
```shell script
-./dev/provider_packages/prepare_provider_packages.py generate-issue-content --only-available-in-dist --suffix rc2
+breeze release-management generate-issue-content --only-available-in-dist --suffix rc2
```
diff --git a/dev/breeze/README.md b/dev/breeze/README.md
index 7dd12fbe68..0c86259305 100644
--- a/dev/breeze/README.md
+++ b/dev/breeze/README.md
@@ -52,6 +52,6 @@ PLEASE DO NOT MODIFY THE HASH BELOW! IT IS AUTOMATICALLY UPDATED BY PRE-COMMIT.
---------------------------------------------------------------------------------------------------------
-Package config hash: 670c60fceb07f18c6fabce5e1382039bc9cc5773d339b54b6957088c484d548ee99e66d11b8eb6cf6872d7467147bcf8661249dc9e64824350edc3eddd57ed5d
+Package config hash: f28f0d555b81a0f48d6b29b3cf8bba132b8c6a8f3d290a25ad4fd62019a9adbf86c0dc913c474e23ae110f3f433db0214bf46b21000f0d2bdd0884134923ae91
---------------------------------------------------------------------------------------------------------
diff --git a/dev/breeze/setup.cfg b/dev/breeze/setup.cfg
index 69d420c5ff..fe4fef0225 100644
--- a/dev/breeze/setup.cfg
+++ b/dev/breeze/setup.cfg
@@ -58,12 +58,14 @@ install_requires =
filelock
inputimeout
importlib-metadata>=4.4; python_version < "3.8"
+ jinja2
pendulum
pre-commit
psutil
pytest
pytest-xdist
pyyaml
+ PyGithub
requests
rich>=12.6.0
rich-click>=1.5
diff --git a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
index b319ebaa78..0534ea8cf1 100644
--- a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
@@ -16,14 +16,22 @@
# under the License.
from __future__ import annotations
+import json
+import os
+import re
import shlex
import sys
+import textwrap
import time
from copy import deepcopy
+from datetime import datetime
+from pathlib import Path
from re import match
-from typing import IO
+from typing import IO, NamedTuple
import click
+from rich.progress import Progress
+from rich.syntax import Syntax
from airflow_breeze.commands.ci_image_commands import rebuild_or_pull_ci_image_if_needed
from airflow_breeze.global_constants import (
@@ -75,7 +83,7 @@ from airflow_breeze.utils.parallel import (
check_async_run_results,
run_with_pool,
)
-from airflow_breeze.utils.path_utils import cleanup_python_generated_files
+from airflow_breeze.utils.path_utils import AIRFLOW_SOURCES_ROOT, cleanup_python_generated_files
from airflow_breeze.utils.python_versions import get_python_version_list
from airflow_breeze.utils.run_utils import (
RunCommandResult,
@@ -692,3 +700,190 @@ def release_prod_images(
f"{dockerhub_repo}:{airflow_version}",
f"{dockerhub_repo}:latest",
)
+
+
+def is_package_in_dist(dist_files: list[str], package: str) -> bool:
+ """Check if package has been prepared in dist folder."""
+ for file in dist_files:
+ if file.startswith(f'apache_airflow_providers_{package.replace(".","_")}') or file.startswith(
+ f'apache-airflow-providers-{package.replace(".","-")}'
+ ):
+ return True
+ return False
+
+
+def get_prs_for_package(package_id: str) -> list[int]:
+ import yaml
+
+ pr_matcher = re.compile(r".*\(#([0-9]*)\)``$")
+ changelog_path = (
+ AIRFLOW_SOURCES_ROOT / "airflow" / "providers" / package_id.replace(".", os.sep) / "CHANGELOG.rst"
+ )
+ # load yaml from file
+ provider_yaml_dict = yaml.safe_load(
+ (
+ AIRFLOW_SOURCES_ROOT
+ / "airflow"
+ / "providers"
+ / package_id.replace(r".", os.sep)
+ / "provider.yaml"
+ ).read_text()
+ )
+ current_release_version = provider_yaml_dict["versions"][0]
+ prs = []
+ with open(changelog_path) as changelog_file:
+ changelog_lines = changelog_file.readlines()
+ extract_prs = False
+ skip_line = False
+ for line in changelog_lines:
+ if skip_line:
+ # Skip first "....." header
+ skip_line = False
+ continue
+ if line.strip() == current_release_version:
+ extract_prs = True
+ skip_line = True
+ continue
+ if extract_prs:
+ if len(line) > 1 and all(c == "." for c in line.strip()):
+ # Header for next version reached
+ break
+ if line.startswith(".. Below changes are excluded from the changelog"):
+ # The reminder of PRs is not important skipping it
+ break
+ match_result = pr_matcher.match(line.strip())
+ if match_result:
+ prs.append(int(match_result.group(1)))
+ return prs
+
+
+@release_management.command(
+ name="generate-issue-content", help="Generates content for issue to test the release."
+)
+@click.option(
+ "--github-token",
+ envvar="GITHUB_TOKEN",
+ help=textwrap.dedent(
+ """
+ GitHub token used to authenticate.
+ You can set omit it if you have GITHUB_TOKEN env variable set.
+ Can be generated with:
+ https://github.com/settings/tokens/new?description=Read%20sssues&scopes=repo:status"""
+ ),
+)
+@click.option("--suffix", default="rc1", help="Suffix to add to the version prepared")
+@click.option(
+ "--only-available-in-dist",
+ is_flag=True,
+ help="Only consider package ids with packages prepared in the dist folder",
+)
+@click.option("--excluded-pr-list", type=str, help="Coma-separated list of PRs to exclude from the issue.")
+@argument_packages
+def generate_issue_content(
+ packages: list[str],
+ github_token: str,
+ suffix: str,
+ only_available_in_dist: bool,
+ excluded_pr_list: str,
+):
+ import jinja2
+ import yaml
+ from github import Github, Issue, PullRequest, UnknownObjectException
+
+ class ProviderPRInfo(NamedTuple):
+ provider_package_id: str
+ pypi_package_name: str
+ version: str
+ pr_list: list[PullRequest.PullRequest | Issue.Issue]
+
+ provider_dependencies: dict[str, dict[str, list[str]]] = json.loads(
+ (AIRFLOW_SOURCES_ROOT / "generated" / "provider_dependencies.json").read_text()
+ )
+ if not packages:
+ packages = list(provider_dependencies.keys())
+ with ci_group("Generates GitHub issue content with people who can test it"):
+ if excluded_pr_list:
+ excluded_prs = [int(pr) for pr in excluded_pr_list.split(",")]
+ else:
+ excluded_prs = []
+ all_prs: set[int] = set()
+ provider_prs: dict[str, list[int]] = {}
+ if only_available_in_dist:
+ files_in_dist = os.listdir(str(APACHE_AIRFLOW_GITHUB_REPOSITORY / "dist"))
+ prepared_package_ids = []
+ for package_id in packages:
+ if not only_available_in_dist or is_package_in_dist(files_in_dist, package_id):
+ get_console().print(f"Extracting PRs for provider {package_id}")
+ prepared_package_ids.append(package_id)
+ else:
+ get_console.print(
+ f"Skipping extracting PRs for provider {package_id} as it is missing in dist"
+ )
+ continue
+ prs = get_prs_for_package(package_id)
+ provider_prs[package_id] = list(filter(lambda pr: pr not in excluded_prs, prs))
+ all_prs.update(provider_prs[package_id])
+ g = Github(github_token)
+ repo = g.get_repo("apache/airflow")
+ pull_requests: dict[int, PullRequest.PullRequest | Issue.Issue] = {}
+ with Progress(console=get_console()) as progress:
+ task = progress.add_task(f"Retrieving {len(all_prs)} PRs ", total=len(all_prs))
+ pr_list = list(all_prs)
+ for i in range(len(pr_list)):
+ pr_number = pr_list[i]
+ progress.console.print(
+ f"Retrieving PR#{pr_number}: https://github.com/apache/airflow/pull/{pr_number}"
+ )
+ try:
+ pull_requests[pr_number] = repo.get_pull(pr_number)
+ except UnknownObjectException:
+ # Fallback to issue if PR not found
+ try:
+ pull_requests[pr_number] = repo.get_issue(pr_number) # (same fields as PR)
+ except UnknownObjectException:
+ get_console().print(f"[red]The PR #{pr_number} could not be found[/]")
+ progress.advance(task)
+ providers: dict[str, ProviderPRInfo] = {}
+ for package_id in prepared_package_ids:
+ pull_request_list = [pull_requests[pr] for pr in provider_prs[package_id] if pr in pull_requests]
+ provider_yaml_dict = yaml.safe_load(
+ (
+ AIRFLOW_SOURCES_ROOT
+ / "airflow"
+ / "providers"
+ / package_id.replace(".", os.sep)
+ / "provider.yaml"
+ ).read_text()
+ )
+ if pull_request_list:
+ providers[package_id] = ProviderPRInfo(
+ version=provider_yaml_dict["versions"][0],
+ provider_package_id=package_id,
+ pypi_package_name=provider_yaml_dict["package-name"],
+ pr_list=pull_request_list,
+ )
+ template = jinja2.Template(
+ (Path(__file__).parents[1] / "provider_issue_TEMPLATE.md.jinja2").read_text()
+ )
+ issue_content = template.render(providers=providers, date=datetime.now(), suffix=suffix)
+ get_console().print()
+ get_console().print(
+ "[green]Below you can find the issue content that you can use "
+ "to ask contributor to test providers![/]"
+ )
+ get_console().print()
+ get_console().print()
+ get_console().print(
+ "Issue title: [yellow]Status of testing Providers that were "
+ f"prepared on { datetime.now().strftime('%B %d, %Y') }[/]"
+ )
+ get_console().print()
+ syntax = Syntax(issue_content, "markdown", theme="ansi_dark")
+ get_console().print(syntax)
+ get_console().print()
+ users: set[str] = set()
+ for provider_info in providers.values():
+ for pr in provider_info.pr_list:
+ users.add("@" + pr.user.login)
+ get_console().print("All users involved in the PRs:")
+ get_console().print(" ".join(users))
diff --git a/dev/provider_packages/PROVIDER_ISSUE_TEMPLATE.md.jinja2 b/dev/breeze/src/airflow_breeze/provider_issue_TEMPLATE.md.jinja2
similarity index 57%
rename from dev/provider_packages/PROVIDER_ISSUE_TEMPLATE.md.jinja2
rename to dev/breeze/src/airflow_breeze/provider_issue_TEMPLATE.md.jinja2
index caab34d824..416d0e43a8 100644
--- a/dev/provider_packages/PROVIDER_ISSUE_TEMPLATE.md.jinja2
+++ b/dev/breeze/src/airflow_breeze/provider_issue_TEMPLATE.md.jinja2
@@ -5,9 +5,9 @@ Let us know in the comment, whether the issue is addressed.
Those are providers that require testing as there were some substantial changes introduced:
-{% for provider_id, provider_pr_info in interesting_providers.items() %}
-## Provider [{{ provider_id }}: {{ provider_pr_info.provider_details.versions[0] }}{{ suffix }}](https://pypi.org/project/{{ provider_pr_info.provider_details.pypi_package_name }}/{{ provider_pr_info.provider_details.versions[0] }}{{ suffix }})
-{%- for pr in provider_pr_info.pr_list %}
+{% for provider_id, provider_info in providers.items() %}
+## Provider [{{ provider_id }}: {{ provider_info.version }}{{ suffix }}](https://pypi.org/project/{{ provider_info.pypi_package_name }}/{{ provider_info.version }}{{ suffix }})
+{%- for pr in provider_info.pr_list %}
- [ ] [{{ pr.title }} (#{{ pr.number }})]({{ pr.html_url }}): @{{ pr.user.login }}
{%- endfor %}
{%- endfor %}
@@ -20,11 +20,11 @@ The guidelines on how to test providers can be found in
NOTE TO RELEASE MANAGER:
-Please move here the providers that have doc-only changes or for which changes are trivial and
-you could asses that they are OK. In case
+Please move here the providers that have doc-only changes or for which changes are trivial, and
+you could assess that they are OK. In case
-The providers are automatically installed on Airflow 2.1 and latest `main` during the CI, so we know they
-are installable. Also all classes within the providers are imported during the CI run so we know all
+The providers are automatically installed on Airflow 2.3 and latest `main` during the CI, so we know they
+are installable. Also, all classes within the providers are imported during the CI run so we know all
providers can be imported.
-->
diff --git a/dev/provider_packages/prepare_provider_packages.py b/dev/provider_packages/prepare_provider_packages.py
index ed1afb1e8f..f892c82774 100755
--- a/dev/provider_packages/prepare_provider_packages.py
+++ b/dev/provider_packages/prepare_provider_packages.py
@@ -41,15 +41,13 @@ from os.path import dirname, relpath
from pathlib import Path
from random import choice
from shutil import copyfile
-from typing import Any, Generator, Iterable, NamedTuple, Union
+from typing import Any, Generator, Iterable, NamedTuple
import jsonschema
import rich_click as click
import semver as semver
-from github import Github, Issue, PullRequest, UnknownObjectException
from packaging.version import Version
from rich.console import Console
-from rich.progress import Progress
from rich.syntax import Syntax
from yaml import safe_load
@@ -1870,166 +1868,6 @@ def update_changelogs(changelog_files: list[str], git_update: bool, base_branch:
_update_changelog(package_id=package_id, base_branch=base_branch, verbose=verbose)
-def get_prs_for_package(package_id: str) -> list[int]:
- pr_matcher = re.compile(r".*\(#([0-9]*)\)``$")
- verify_provider_package(package_id)
- changelog_path = verify_changelog_exists(package_id)
- provider_details = get_provider_details(package_id)
- current_release_version = provider_details.versions[0]
- prs = []
- with open(changelog_path) as changelog_file:
- changelog_lines = changelog_file.readlines()
- extract_prs = False
- skip_line = False
- for line in changelog_lines:
- if skip_line:
- # Skip first "....." header
- skip_line = False
- continue
- if line.strip() == current_release_version:
- extract_prs = True
- skip_line = True
- continue
- if extract_prs:
- if len(line) > 1 and all(c == "." for c in line.strip()):
- # Header for next version reached
- break
- if line.startswith(".. Below changes are excluded from the changelog"):
- # The reminder of PRs is not important skipping it
- break
- match_result = pr_matcher.match(line.strip())
- if match_result:
- prs.append(int(match_result.group(1)))
- return prs
-
-
-PullRequestOrIssue = Union[PullRequest.PullRequest, Issue.Issue]
-
-
-class ProviderPRInfo(NamedTuple):
- provider_details: ProviderPackageDetails
- pr_list: list[PullRequestOrIssue]
-
-
-def is_package_in_dist(dist_files: list[str], package: str) -> bool:
- """Check if package has been prepared in dist folder."""
- for file in dist_files:
- if file.startswith(f'apache_airflow_providers_{package.replace(".","_")}') or file.startswith(
- f'apache-airflow-providers-{package.replace(".","-")}'
- ):
- return True
- return False
-
-
-@cli.command()
-@click.option(
- "--github-token",
- envvar="GITHUB_TOKEN",
- help=textwrap.dedent(
- """
- GitHub token used to authenticate.
- You can set omit it if you have GITHUB_TOKEN env variable set.
- Can be generated with:
- https://github.com/settings/tokens/new?description=Read%20sssues&scopes=repo:status"""
- ),
-)
-@click.option("--suffix", default="rc1")
-@click.option(
- "--only-available-in-dist",
- is_flag=True,
- help="Only consider package ids with packages prepared in the dist folder",
-)
-@click.option("--excluded-pr-list", type=str, help="Coma-separated list of PRs to exclude from the issue.")
-@argument_package_ids
-def generate_issue_content(
- package_ids: list[str],
- github_token: str,
- suffix: str,
- only_available_in_dist: bool,
- excluded_pr_list: str,
-):
- if not package_ids:
- package_ids = get_all_providers()
- """Generates content for issue to test the release."""
- with with_group("Generates GitHub issue content with people who can test it"):
- if excluded_pr_list:
- excluded_prs = [int(pr) for pr in excluded_pr_list.split(",")]
- else:
- excluded_prs = []
- all_prs: set[int] = set()
- provider_prs: dict[str, list[int]] = {}
- if only_available_in_dist:
- files_in_dist = os.listdir(str(DIST_PATH))
- prepared_package_ids = []
- for package_id in package_ids:
- if not only_available_in_dist or is_package_in_dist(files_in_dist, package_id):
- console.print(f"Extracting PRs for provider {package_id}")
- prepared_package_ids.append(package_id)
- else:
- console.print(f"Skipping extracting PRs for provider {package_id} as it is missing in dist")
- continue
- prs = get_prs_for_package(package_id)
- provider_prs[package_id] = list(filter(lambda pr: pr not in excluded_prs, prs))
- all_prs.update(provider_prs[package_id])
- g = Github(github_token)
- repo = g.get_repo("apache/airflow")
- pull_requests: dict[int, PullRequestOrIssue] = {}
- with Progress(console=console) as progress:
- task = progress.add_task(f"Retrieving {len(all_prs)} PRs ", total=len(all_prs))
- pr_list = list(all_prs)
- for i in range(len(pr_list)):
- pr_number = pr_list[i]
- progress.console.print(
- f"Retrieving PR#{pr_number}: https://github.com/apache/airflow/pull/{pr_number}"
- )
- try:
- pull_requests[pr_number] = repo.get_pull(pr_number)
- except UnknownObjectException:
- # Fallback to issue if PR not found
- try:
- pull_requests[pr_number] = repo.get_issue(pr_number) # (same fields as PR)
- except UnknownObjectException:
- console.print(f"[red]The PR #{pr_number} could not be found[/]")
- progress.advance(task)
- interesting_providers: dict[str, ProviderPRInfo] = {}
- non_interesting_providers: dict[str, ProviderPRInfo] = {}
- for package_id in prepared_package_ids:
- pull_request_list = [pull_requests[pr] for pr in provider_prs[package_id] if pr in pull_requests]
- provider_details = get_provider_details(package_id)
- if pull_request_list:
- interesting_providers[package_id] = ProviderPRInfo(provider_details, pull_request_list)
- else:
- non_interesting_providers[package_id] = ProviderPRInfo(provider_details, pull_request_list)
- context = {
- "interesting_providers": interesting_providers,
- "date": datetime.now(),
- "suffix": suffix,
- "non_interesting_providers": non_interesting_providers,
- }
- issue_content = render_template(template_name="PROVIDER_ISSUE", context=context, extension=".md")
- console.print()
- console.print(
- "[green]Below you can find the issue content that you can use "
- "to ask contributor to test providers![/]"
- )
- console.print()
- console.print()
- console.print(
- "Issue title: [yellow]Status of testing Providers that were "
- f"prepared on { datetime.now().strftime('%B %d, %Y') }[/]"
- )
- console.print()
- syntax = Syntax(issue_content, "markdown", theme="ansi_dark")
- console.print(syntax)
- console.print()
- users: set[str] = set()
- for provider_info in interesting_providers.values():
- for pr in provider_info.pr_list:
- users.add("@" + pr.user.login)
- console.print("All users involved in the PRs:")
- console.print(" ".join(users))
-
-
if __name__ == "__main__":
# The cli exit code is:
# * 0 in case of success
diff --git a/images/breeze/output-commands-hash.txt b/images/breeze/output-commands-hash.txt
index efec015ba0..219c440000 100644
--- a/images/breeze/output-commands-hash.txt
+++ b/images/breeze/output-commands-hash.txt
@@ -36,21 +36,22 @@ prod-image:pull:e3c89dd908fc44adf6e159c2950ebdd0
prod-image:verify:31bc5efada1d70a0a31990025db1a093
prod-image:a4013428dc7f71a1defc3778d2efe3dc
release-management:generate-constraints:ae30d6ad49a1b2c15b61cb29080fd957
+release-management:generate-issue-content:24218438f9e85e7c92258aadebbb19de
release-management:prepare-airflow-package:3ac14ea6d2b09614959c0ec4fd564789
release-management:prepare-provider-documentation:3fe5ead9887c518d1b397d1103dc0025
release-management:prepare-provider-packages:40144cb01afc56f6a4f92d9e117e546e
release-management:release-prod-images:c9bc40938e0efad49e51ef66e83f9527
-release-management:verify-provider-packages:8d3c6362657d76bb3cd064fed5596e84
-release-management:c4e840ca22b11a431f5bc7b118dc061f
+release-management:verify-provider-packages:88bd609aff6d09d52ab8d80d6e055e7b
+release-management:eafd13da512a5a7c0fb1499cf7ff1d63
setup:autocomplete:03343478bf1d0cf9c101d454cdb63b68
setup:config:3ffcd35dd24b486ddf1d08b797e3d017
-setup:regenerate-command-images:255746830d7b5d1337d13b8e101f7f83
+setup:regenerate-command-images:ab2d83c339fa3a42b0c819b6b6cc88ae
setup:self-upgrade:d02f70c7a230eae3463ceec2056b63fa
setup:version:123b462a421884dc2320ffc5e54b2478
-setup:2e9e4ab1729c5420b7a2b78cbee7539a
-shell:affbf6f7f469408d0af47f75c6a38f6c
-start-airflow:109728919a0dd5c5ff5640ae86ba9e90
-static-checks:7a39e28c87fbca0a9fae0ebfe1591b71
+setup:fbabee281b69f818091d780b24bd815a
+shell:76e0f530b7af514a2aad3032b6516c46
+start-airflow:06d4aeb5f1b65f6b975f3f915558d0b3
+static-checks:f45ad432bdc47a2256fdb0277b19d816
stop:8969537ccdd799f692ccb8600a7bbed6
testing:docker-compose-tests:b86c044b24138af0659a05ed6331576c
testing:helm-tests:94a442e7f3f63b34c4831a84d165690a
diff --git a/images/breeze/output-commands.svg b/images/breeze/output-commands.svg
index 22001729e6..c8763961ad 100644
--- a/images/breeze/output-commands.svg
+++ b/images/breeze/output-commands.svg
@@ -35,8 +35,8 @@
.breeze-help-r1 { fill: #c5c8c6;font-weight: bold }
.breeze-help-r2 { fill: #c5c8c6 }
.breeze-help-r3 { fill: #d0b344;font-weight: bold }
-.breeze-help-r4 { fill: #68a0b3;font-weight: bold }
-.breeze-help-r5 { fill: #868887 }
+.breeze-help-r4 { fill: #868887 }
+.breeze-help-r5 { fill: #68a0b3;font-weight: bold }
.breeze-help-r6 { fill: #98a84b;font-weight: bold }
.breeze-help-r7 { fill: #8d7b39 }
</style>
@@ -187,49 +187,49 @@
<g class="breeze-help-matrix">
<text class="breeze-help-r2" x="1464" y="20" textLength="12.2" clip-path="url(#breeze-help-line-0)">
-</text><text class="breeze-help-r3" x="12.2" y="44.4" textLength="85.4" clip-path="url(#breeze-help-line-1)">Usage: </text><text class="breeze-help-r1" x="97.6" y="44.4" textLength="97.6" clip-path="url(#breeze-help-line-1)">breeze [</text><text class="breeze-help-r4" x="195.2" y="44.4" textLength="85.4" clip-path="url(#breeze-help-line-1)">OPTIONS</text><text class="breeze-help-r1" x="280.6" y="44.4" textLength="24.4" clip-path="url(#breeze-help-line-1)">] </text><text cl [...]
+</text><text class="breeze-help-r3" x="12.2" y="44.4" textLength="85.4" clip-path="url(#breeze-help-line-1)">Usage: </text><text class="breeze-help-r1" x="97.6" y="44.4" textLength="414.8" clip-path="url(#breeze-help-line-1)">breeze [OPTIONS] COMMAND [ARGS]...</text><text class="breeze-help-r2" x="1464" y="44.4" textLength="12.2" clip-path="url(#breeze-help-line-1)">
</text><text class="breeze-help-r2" x="1464" y="68.8" textLength="12.2" clip-path="url(#breeze-help-line-2)">
-</text><text class="breeze-help-r5" x="0" y="93.2" textLength="24.4" clip-path="url(#breeze-help-line-3)">╭─</text><text class="breeze-help-r5" x="24.4" y="93.2" textLength="158.6" clip-path="url(#breeze-help-line-3)"> Basic flags </text><text class="breeze-help-r5" x="183" y="93.2" textLength="1256.6" clip-path="url(#breeze-help-line-3)">───────────────────────────────────────────────────────────────────────────────────────────────────────</text><text class="breeze-help-r [...]
-</text><text class="breeze-help-r5" x="0" y="117.6" textLength="12.2" clip-path="url(#breeze-help-line-4)">│</text><text class="breeze-help-r4" x="24.4" y="117.6" textLength="12.2" clip-path="url(#breeze-help-line-4)">-</text><text class="breeze-help-r4" x="36.6" y="117.6" textLength="85.4" clip-path="url(#breeze-help-line-4)">-python</text><text class="breeze-help-r6" x="305" y="117.6" textLength="24.4" clip-path="url(#breeze-help-line-4)">-p</text><text class="breeze-help-r2" x="353.8" [...]
-</text><text class="breeze-help-r5" x="0" y="142" textLength="12.2" clip-path="url(#breeze-help-line-5)">│</text><text class="breeze-help-r5" x="353.8" y="142" textLength="732" clip-path="url(#breeze-help-line-5)">[default: 3.7]                                           & [...]
-</text><text class="breeze-help-r5" x="0" y="166.4" textLength="12.2" clip-path="url(#breeze-help-line-6)">│</text><text class="breeze-help-r4" x="24.4" y="166.4" textLength="12.2" clip-path="url(#breeze-help-line-6)">-</text><text class="breeze-help-r4" x="36.6" y="166.4" textLength="97.6" clip-path="url(#breeze-help-line-6)">-backend</text><text class="breeze-help-r6" x="305" y="166.4" textLength="24.4" clip-path="url(#breeze-help-line-6)">-b</text><text class="breeze-help-r2" x="353.8 [...]
-</text><text class="breeze-help-r5" x="0" y="190.8" textLength="12.2" clip-path="url(#breeze-help-line-7)">│</text><text class="breeze-help-r4" x="24.4" y="190.8" textLength="12.2" clip-path="url(#breeze-help-line-7)">-</text><text class="breeze-help-r4" x="36.6" y="190.8" textLength="109.8" clip-path="url(#breeze-help-line-7)">-postgres</text><text class="breeze-help-r4" x="146.4" y="190.8" textLength="97.6" clip-path="url(#breeze-help-line-7)">-version</text><text class="breeze-help-r6 [...]
-</text><text class="breeze-help-r5" x="0" y="215.2" textLength="12.2" clip-path="url(#breeze-help-line-8)">│</text><text class="breeze-help-r4" x="24.4" y="215.2" textLength="12.2" clip-path="url(#breeze-help-line-8)">-</text><text class="breeze-help-r4" x="36.6" y="215.2" textLength="73.2" clip-path="url(#breeze-help-line-8)">-mysql</text><text class="breeze-help-r4" x="109.8" y="215.2" textLength="97.6" clip-path="url(#breeze-help-line-8)">-version</text><text class="breeze-help-r6" x= [...]
-</text><text class="breeze-help-r5" x="0" y="239.6" textLength="12.2" clip-path="url(#breeze-help-line-9)">│</text><text class="breeze-help-r4" x="24.4" y="239.6" textLength="12.2" clip-path="url(#breeze-help-line-9)">-</text><text class="breeze-help-r4" x="36.6" y="239.6" textLength="73.2" clip-path="url(#breeze-help-line-9)">-mssql</text><text class="breeze-help-r4" x="109.8" y="239.6" textLength="97.6" clip-path="url(#breeze-help-line-9)">-version</text><text class="breeze-help-r6" x= [...]
-</text><text class="breeze-help-r5" x="0" y="264" textLength="12.2" clip-path="url(#breeze-help-line-10)">│</text><text class="breeze-help-r4" x="24.4" y="264" textLength="12.2" clip-path="url(#breeze-help-line-10)">-</text><text class="breeze-help-r4" x="36.6" y="264" textLength="146.4" clip-path="url(#breeze-help-line-10)">-integration</text><text class="breeze-help-r2" x="353.8" y="264" textLength="744.2" clip-path="url(#breeze-help-line-10)">Integration(s) to enable wh [...]
-</text><text class="breeze-help-r5" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-help-line-11)">│</text><text class="breeze-help-r7" x="353.8" y="288.4" textLength="744.2" clip-path="url(#breeze-help-line-11)">(cassandra | kerberos | mongo | pinot | celery | trino | all)</text><text class="breeze-help-r5" x="1451.8" y="288.4" textLength="12.2" clip-path="url(#breeze-help-line-11)">│</text><text class="breeze-help-r2" [...]
-</text><text class="breeze-help-r5" x="0" y="312.8" textLength="12.2" clip-path="url(#breeze-help-line-12)">│</text><text class="breeze-help-r4" x="24.4" y="312.8" textLength="12.2" clip-path="url(#breeze-help-line-12)">-</text><text class="breeze-help-r4" x="36.6" y="312.8" textLength="97.6" clip-path="url(#breeze-help-line-12)">-forward</text><text class="breeze-help-r4" x="134.2" y="312.8" textLength="146.4" clip-path="url(#breeze-help-line-12)">-credentials</text><text class="breeze- [...]
-</text><text class="breeze-help-r5" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-help-line-13)">│</text><text class="breeze-help-r4" x="24.4" y="337.2" textLength="12.2" clip-path="url(#breeze-help-line-13)">-</text><text class="breeze-help-r4" x="36.6" y="337.2" textLength="36.6" clip-path="url(#breeze-help-line-13)">-db</text><text class="breeze-help-r4" x="73.2" y="337.2" textLength="73.2" clip-path="url(#breeze-help-line-13)">-reset</text><text class="breeze-help-r6" x="3 [...]
-</text><text class="breeze-help-r5" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-help-line-14)">│</text><text class="breeze-help-r4" x="24.4" y="361.6" textLength="12.2" clip-path="url(#breeze-help-line-14)">-</text><text class="breeze-help-r4" x="36.6" y="361.6" textLength="48.8" clip-path="url(#breeze-help-line-14)">-max</text><text class="breeze-help-r4" x="85.4" y="361.6" textLength="61" clip-path="url(#breeze-help-line-14)">-time</text><text class="breeze-help-r2" x="353 [...]
-</text><text class="breeze-help-r5" x="0" y="386" textLength="12.2" clip-path="url(#breeze-help-line-15)">│</text><text class="breeze-help-r7" x="353.8" y="386" textLength="1049.2" clip-path="url(#breeze-help-line-15)">(INTEGER RANGE)                                          & [...]
-</text><text class="breeze-help-r5" x="0" y="410.4" textLength="12.2" clip-path="url(#breeze-help-line-16)">│</text><text class="breeze-help-r4" x="24.4" y="410.4" textLength="12.2" clip-path="url(#breeze-help-line-16)">-</text><text class="breeze-help-r4" x="36.6" y="410.4" textLength="85.4" clip-path="url(#breeze-help-line-16)">-github</text><text class="breeze-help-r4" x="122" y="410.4" textLength="134.2" clip-path="url(#breeze-help-line-16)">-repository</text><text class="breeze-help [...]
-</text><text class="breeze-help-r5" x="0" y="434.8" textLength="1464" clip-path="url(#breeze-help-line-17)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-help-r2" x="1464" y="434.8" textLength="12.2" clip-path="url(#breeze-help-line-17)">
-</text><text class="breeze-help-r5" x="0" y="459.2" textLength="24.4" clip-path="url(#breeze-help-line-18)">╭─</text><text class="breeze-help-r5" x="24.4" y="459.2" textLength="195.2" clip-path="url(#breeze-help-line-18)"> Common options </text><text class="breeze-help-r5" x="219.6" y="459.2" textLength="1220" clip-path="url(#breeze-help-line-18)">────────────────────────────────────────────────────────────────────────────────────────────────────</text><text class="breeze- [...]
-</text><text class="breeze-help-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-help-line-19)">│</text><text class="breeze-help-r4" x="24.4" y="483.6" textLength="12.2" clip-path="url(#breeze-help-line-19)">-</text><text class="breeze-help-r4" x="36.6" y="483.6" textLength="97.6" clip-path="url(#breeze-help-line-19)">-verbose</text><text class="breeze-help-r6" x="158.6" y="483.6" textLength="24.4" clip-path="url(#breeze-help-line-19)">-v</text><text class="breeze-help-r2" x= [...]
-</text><text class="breeze-help-r5" x="0" y="508" textLength="12.2" clip-path="url(#breeze-help-line-20)">│</text><text class="breeze-help-r4" x="24.4" y="508" textLength="12.2" clip-path="url(#breeze-help-line-20)">-</text><text class="breeze-help-r4" x="36.6" y="508" textLength="48.8" clip-path="url(#breeze-help-line-20)">-dry</text><text class="breeze-help-r4" x="85.4" y="508" textLength="48.8" clip-path="url(#breeze-help-line-20)">-run</text><text class="breeze-help-r6" x="158.6" y=" [...]
-</text><text class="breeze-help-r5" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-help-line-21)">│</text><text class="breeze-help-r4" x="24.4" y="532.4" textLength="12.2" clip-path="url(#breeze-help-line-21)">-</text><text class="breeze-help-r4" x="36.6" y="532.4" textLength="85.4" clip-path="url(#breeze-help-line-21)">-answer</text><text class="breeze-help-r6" x="158.6" y="532.4" textLength="24.4" clip-path="url(#breeze-help-line-21)">-a</text><text class="breeze-help-r2" x=" [...]
-</text><text class="breeze-help-r5" x="0" y="556.8" textLength="12.2" clip-path="url(#breeze-help-line-22)">│</text><text class="breeze-help-r4" x="24.4" y="556.8" textLength="12.2" clip-path="url(#breeze-help-line-22)">-</text><text class="breeze-help-r4" x="36.6" y="556.8" textLength="61" clip-path="url(#breeze-help-line-22)">-help</text><text class="breeze-help-r6" x="158.6" y="556.8" textLength="24.4" clip-path="url(#breeze-help-line-22)">-h</text><text class="breeze-help-r2" x="207. [...]
-</text><text class="breeze-help-r5" x="0" y="581.2" textLength="1464" clip-path="url(#breeze-help-line-23)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-help-r2" x="1464" y="581.2" textLength="12.2" clip-path="url(#breeze-help-line-23)">
-</text><text class="breeze-help-r5" x="0" y="605.6" textLength="24.4" clip-path="url(#breeze-help-line-24)">╭─</text><text class="breeze-help-r5" x="24.4" y="605.6" textLength="317.2" clip-path="url(#breeze-help-line-24)"> Basic developer commands </text><text class="breeze-help-r5" x="341.6" y="605.6" textLength="1098" clip-path="url(#breeze-help-line-24)">──────────────────────────────────────────────────────────────────────────────────────────</text><text class="br [...]
-</text><text class="breeze-help-r5" x="0" y="630" textLength="12.2" clip-path="url(#breeze-help-line-25)">│</text><text class="breeze-help-r4" x="24.4" y="630" textLength="219.6" clip-path="url(#breeze-help-line-25)">start-airflow     </text><text class="breeze-help-r2" x="268.4" y="630" textLength="1171.2" clip-path="url(#breeze-help-line-25)">Enter breeze environment and starts all Airflow components in the tmux [...]
-</text><text class="breeze-help-r5" x="0" y="654.4" textLength="12.2" clip-path="url(#breeze-help-line-26)">│</text><text class="breeze-help-r2" x="268.4" y="654.4" textLength="1171.2" clip-path="url(#breeze-help-line-26)">if contents of www directory changed.                                   [...]
-</text><text class="breeze-help-r5" x="0" y="678.8" textLength="12.2" clip-path="url(#breeze-help-line-27)">│</text><text class="breeze-help-r4" x="24.4" y="678.8" textLength="219.6" clip-path="url(#breeze-help-line-27)">static-checks     </text><text class="breeze-help-r2" x="268.4" y="678.8" textLength="1171.2" clip-path="url(#breeze-help-line-27)">Run static checks.              &# [...]
-</text><text class="breeze-help-r5" x="0" y="703.2" textLength="12.2" clip-path="url(#breeze-help-line-28)">│</text><text class="breeze-help-r4" x="24.4" y="703.2" textLength="219.6" clip-path="url(#breeze-help-line-28)">build-docs        </text><text class="breeze-help-r2" x="268.4" y="703.2" textLength="1171.2" clip-path="url(#breeze-help-line-28)">Build documentation in the container.        [...]
-</text><text class="breeze-help-r5" x="0" y="727.6" textLength="12.2" clip-path="url(#breeze-help-line-29)">│</text><text class="breeze-help-r4" x="24.4" y="727.6" textLength="219.6" clip-path="url(#breeze-help-line-29)">stop              </text><text class="breeze-help-r2" x="268.4" y="727.6" textLength="1171.2" clip-path="url(#breeze-help-line-29)">Stop running breeze environment.    [...]
-</text><text class="breeze-help-r5" x="0" y="752" textLength="12.2" clip-path="url(#breeze-help-line-30)">│</text><text class="breeze-help-r4" x="24.4" y="752" textLength="219.6" clip-path="url(#breeze-help-line-30)">shell             </text><text class="breeze-help-r2" x="268.4" y="752" textLength="1171.2" clip-path="url(#breeze-help-line-30)">Enter breeze environment. this is the default  [...]
-</text><text class="breeze-help-r5" x="0" y="776.4" textLength="12.2" clip-path="url(#breeze-help-line-31)">│</text><text class="breeze-help-r4" x="24.4" y="776.4" textLength="219.6" clip-path="url(#breeze-help-line-31)">exec              </text><text class="breeze-help-r2" x="268.4" y="776.4" textLength="1171.2" clip-path="url(#breeze-help-line-31)">Joins the interactive shell of running  [...]
-</text><text class="breeze-help-r5" x="0" y="800.8" textLength="12.2" clip-path="url(#breeze-help-line-32)">│</text><text class="breeze-help-r4" x="24.4" y="800.8" textLength="219.6" clip-path="url(#breeze-help-line-32)">compile-www-assets</text><text class="breeze-help-r2" x="268.4" y="800.8" textLength="1171.2" clip-path="url(#breeze-help-line-32)">Compiles www assets.                  & [...]
-</text><text class="breeze-help-r5" x="0" y="825.2" textLength="12.2" clip-path="url(#breeze-help-line-33)">│</text><text class="breeze-help-r4" x="24.4" y="825.2" textLength="219.6" clip-path="url(#breeze-help-line-33)">cleanup           </text><text class="breeze-help-r2" x="268.4" y="825.2" textLength="805.2" clip-path="url(#breeze-help-line-33)">Cleans the cache of parameters, docker cache and&# [...]
-</text><text class="breeze-help-r5" x="0" y="849.6" textLength="1464" clip-path="url(#breeze-help-line-34)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-help-r2" x="1464" y="849.6" textLength="12.2" clip-path="url(#breeze-help-line-34)">
-</text><text class="breeze-help-r5" x="0" y="874" textLength="24.4" clip-path="url(#breeze-help-line-35)">╭─</text><text class="breeze-help-r5" x="24.4" y="874" textLength="305" clip-path="url(#breeze-help-line-35)"> Advanced command groups </text><text class="breeze-help-r5" x="329.4" y="874" textLength="1110.2" clip-path="url(#breeze-help-line-35)">───────────────────────────────────────────────────────────────────────────────────────────</text><text class="breeze-h [...]
-</text><text class="breeze-help-r5" x="0" y="898.4" textLength="12.2" clip-path="url(#breeze-help-line-36)">│</text><text class="breeze-help-r4" x="24.4" y="898.4" textLength="280.6" clip-path="url(#breeze-help-line-36)">testing                </text><text class="breeze-help-r2" x="329.4" y="898.4" textLength="1110.2" clip-path="url(#breeze-help-line-36)">Tools that developers can use  [...]
-</text><text class="breeze-help-r5" x="0" y="922.8" textLength="12.2" clip-path="url(#breeze-help-line-37)">│</text><text class="breeze-help-r4" x="24.4" y="922.8" textLength="280.6" clip-path="url(#breeze-help-line-37)">ci-image               </text><text class="breeze-help-r2" x="329.4" y="922.8" textLength="597.8" clip-path="url(#breeze-help-line-37)">Tools that developers can use to [...]
-</text><text class="breeze-help-r5" x="0" y="947.2" textLength="12.2" clip-path="url(#breeze-help-line-38)">│</text><text class="breeze-help-r4" x="24.4" y="947.2" textLength="280.6" clip-path="url(#breeze-help-line-38)">k8s                    </text><text class="breeze-help-r2" x="329.4" y="947.2" textLength="1110.2" clip-path="url(#breeze-help-line-38)">Tools that developers [...]
-</text><text class="breeze-help-r5" x="0" y="971.6" textLength="12.2" clip-path="url(#breeze-help-line-39)">│</text><text class="breeze-help-r4" x="24.4" y="971.6" textLength="280.6" clip-path="url(#breeze-help-line-39)">prod-image             </text><text class="breeze-help-r2" x="329.4" y="971.6" textLength="597.8" clip-path="url(#breeze-help-line-39)">Tools that developers can use to manuall [...]
-</text><text class="breeze-help-r5" x="0" y="996" textLength="12.2" clip-path="url(#breeze-help-line-40)">│</text><text class="breeze-help-r4" x="24.4" y="996" textLength="280.6" clip-path="url(#breeze-help-line-40)">setup                  </text><text class="breeze-help-r2" x="329.4" y="996" textLength="1110.2" clip-path="url(#breeze-help-line-40)">Tools that developers can use& [...]
-</text><text class="breeze-help-r5" x="0" y="1020.4" textLength="12.2" clip-path="url(#breeze-help-line-41)">│</text><text class="breeze-help-r4" x="24.4" y="1020.4" textLength="280.6" clip-path="url(#breeze-help-line-41)">release-management     </text><text class="breeze-help-r2" x="329.4" y="1020.4" textLength="1110.2" clip-path="url(#breeze-help-line-41)">Tools that release managers can use to prepare and manage [...]
-</text><text class="breeze-help-r5" x="0" y="1044.8" textLength="12.2" clip-path="url(#breeze-help-line-42)">│</text><text class="breeze-help-r4" x="24.4" y="1044.8" textLength="280.6" clip-path="url(#breeze-help-line-42)">ci                     </text><text class="breeze-help-r2" x="329.4" y="1044.8" textLength="134.2" clip-path="url(#breeze-help-line-42)">Tools that </text [...]
-</text><text class="breeze-help-r5" x="0" y="1069.2" textLength="1464" clip-path="url(#breeze-help-line-43)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-help-r2" x="1464" y="1069.2" textLength="12.2" clip-path="url(#breeze-help-line-43)">
+</text><text class="breeze-help-r4" x="0" y="93.2" textLength="24.4" clip-path="url(#breeze-help-line-3)">╭─</text><text class="breeze-help-r4" x="24.4" y="93.2" textLength="158.6" clip-path="url(#breeze-help-line-3)"> Basic flags </text><text class="breeze-help-r4" x="183" y="93.2" textLength="1256.6" clip-path="url(#breeze-help-line-3)">───────────────────────────────────────────────────────────────────────────────────────────────────────</text><text class="breeze-help-r [...]
+</text><text class="breeze-help-r4" x="0" y="117.6" textLength="12.2" clip-path="url(#breeze-help-line-4)">│</text><text class="breeze-help-r5" x="24.4" y="117.6" textLength="12.2" clip-path="url(#breeze-help-line-4)">-</text><text class="breeze-help-r5" x="36.6" y="117.6" textLength="85.4" clip-path="url(#breeze-help-line-4)">-python</text><text class="breeze-help-r6" x="305" y="117.6" textLength="24.4" clip-path="url(#breeze-help-line-4)">-p</text><text class="breeze-help-r2" x="353.8" [...]
+</text><text class="breeze-help-r4" x="0" y="142" textLength="12.2" clip-path="url(#breeze-help-line-5)">│</text><text class="breeze-help-r4" x="353.8" y="142" textLength="732" clip-path="url(#breeze-help-line-5)">[default: 3.7]                                           & [...]
+</text><text class="breeze-help-r4" x="0" y="166.4" textLength="12.2" clip-path="url(#breeze-help-line-6)">│</text><text class="breeze-help-r5" x="24.4" y="166.4" textLength="12.2" clip-path="url(#breeze-help-line-6)">-</text><text class="breeze-help-r5" x="36.6" y="166.4" textLength="97.6" clip-path="url(#breeze-help-line-6)">-backend</text><text class="breeze-help-r6" x="305" y="166.4" textLength="24.4" clip-path="url(#breeze-help-line-6)">-b</text><text class="breeze-help-r2" x="353.8 [...]
+</text><text class="breeze-help-r4" x="0" y="190.8" textLength="12.2" clip-path="url(#breeze-help-line-7)">│</text><text class="breeze-help-r5" x="24.4" y="190.8" textLength="12.2" clip-path="url(#breeze-help-line-7)">-</text><text class="breeze-help-r5" x="36.6" y="190.8" textLength="109.8" clip-path="url(#breeze-help-line-7)">-postgres</text><text class="breeze-help-r5" x="146.4" y="190.8" textLength="97.6" clip-path="url(#breeze-help-line-7)">-version</text><text class="breeze-help-r6 [...]
+</text><text class="breeze-help-r4" x="0" y="215.2" textLength="12.2" clip-path="url(#breeze-help-line-8)">│</text><text class="breeze-help-r5" x="24.4" y="215.2" textLength="12.2" clip-path="url(#breeze-help-line-8)">-</text><text class="breeze-help-r5" x="36.6" y="215.2" textLength="73.2" clip-path="url(#breeze-help-line-8)">-mysql</text><text class="breeze-help-r5" x="109.8" y="215.2" textLength="97.6" clip-path="url(#breeze-help-line-8)">-version</text><text class="breeze-help-r6" x= [...]
+</text><text class="breeze-help-r4" x="0" y="239.6" textLength="12.2" clip-path="url(#breeze-help-line-9)">│</text><text class="breeze-help-r5" x="24.4" y="239.6" textLength="12.2" clip-path="url(#breeze-help-line-9)">-</text><text class="breeze-help-r5" x="36.6" y="239.6" textLength="73.2" clip-path="url(#breeze-help-line-9)">-mssql</text><text class="breeze-help-r5" x="109.8" y="239.6" textLength="97.6" clip-path="url(#breeze-help-line-9)">-version</text><text class="breeze-help-r6" x= [...]
+</text><text class="breeze-help-r4" x="0" y="264" textLength="12.2" clip-path="url(#breeze-help-line-10)">│</text><text class="breeze-help-r5" x="24.4" y="264" textLength="12.2" clip-path="url(#breeze-help-line-10)">-</text><text class="breeze-help-r5" x="36.6" y="264" textLength="146.4" clip-path="url(#breeze-help-line-10)">-integration</text><text class="breeze-help-r2" x="353.8" y="264" textLength="744.2" clip-path="url(#breeze-help-line-10)">Integration(s) to enable wh [...]
+</text><text class="breeze-help-r4" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-help-line-11)">│</text><text class="breeze-help-r7" x="353.8" y="288.4" textLength="744.2" clip-path="url(#breeze-help-line-11)">(cassandra | kerberos | mongo | pinot | celery | trino | all)</text><text class="breeze-help-r4" x="1451.8" y="288.4" textLength="12.2" clip-path="url(#breeze-help-line-11)">│</text><text class="breeze-help-r2" [...]
+</text><text class="breeze-help-r4" x="0" y="312.8" textLength="12.2" clip-path="url(#breeze-help-line-12)">│</text><text class="breeze-help-r5" x="24.4" y="312.8" textLength="12.2" clip-path="url(#breeze-help-line-12)">-</text><text class="breeze-help-r5" x="36.6" y="312.8" textLength="97.6" clip-path="url(#breeze-help-line-12)">-forward</text><text class="breeze-help-r5" x="134.2" y="312.8" textLength="146.4" clip-path="url(#breeze-help-line-12)">-credentials</text><text class="breeze- [...]
+</text><text class="breeze-help-r4" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-help-line-13)">│</text><text class="breeze-help-r5" x="24.4" y="337.2" textLength="12.2" clip-path="url(#breeze-help-line-13)">-</text><text class="breeze-help-r5" x="36.6" y="337.2" textLength="36.6" clip-path="url(#breeze-help-line-13)">-db</text><text class="breeze-help-r5" x="73.2" y="337.2" textLength="73.2" clip-path="url(#breeze-help-line-13)">-reset</text><text class="breeze-help-r6" x="3 [...]
+</text><text class="breeze-help-r4" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-help-line-14)">│</text><text class="breeze-help-r5" x="24.4" y="361.6" textLength="12.2" clip-path="url(#breeze-help-line-14)">-</text><text class="breeze-help-r5" x="36.6" y="361.6" textLength="48.8" clip-path="url(#breeze-help-line-14)">-max</text><text class="breeze-help-r5" x="85.4" y="361.6" textLength="61" clip-path="url(#breeze-help-line-14)">-time</text><text class="breeze-help-r2" x="353 [...]
+</text><text class="breeze-help-r4" x="0" y="386" textLength="12.2" clip-path="url(#breeze-help-line-15)">│</text><text class="breeze-help-r7" x="353.8" y="386" textLength="1049.2" clip-path="url(#breeze-help-line-15)">(INTEGER RANGE)                                          & [...]
+</text><text class="breeze-help-r4" x="0" y="410.4" textLength="12.2" clip-path="url(#breeze-help-line-16)">│</text><text class="breeze-help-r5" x="24.4" y="410.4" textLength="12.2" clip-path="url(#breeze-help-line-16)">-</text><text class="breeze-help-r5" x="36.6" y="410.4" textLength="85.4" clip-path="url(#breeze-help-line-16)">-github</text><text class="breeze-help-r5" x="122" y="410.4" textLength="134.2" clip-path="url(#breeze-help-line-16)">-repository</text><text class="breeze-help [...]
+</text><text class="breeze-help-r4" x="0" y="434.8" textLength="1464" clip-path="url(#breeze-help-line-17)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-help-r2" x="1464" y="434.8" textLength="12.2" clip-path="url(#breeze-help-line-17)">
+</text><text class="breeze-help-r4" x="0" y="459.2" textLength="24.4" clip-path="url(#breeze-help-line-18)">╭─</text><text class="breeze-help-r4" x="24.4" y="459.2" textLength="195.2" clip-path="url(#breeze-help-line-18)"> Common options </text><text class="breeze-help-r4" x="219.6" y="459.2" textLength="1220" clip-path="url(#breeze-help-line-18)">────────────────────────────────────────────────────────────────────────────────────────────────────</text><text class="breeze- [...]
+</text><text class="breeze-help-r4" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-help-line-19)">│</text><text class="breeze-help-r5" x="24.4" y="483.6" textLength="12.2" clip-path="url(#breeze-help-line-19)">-</text><text class="breeze-help-r5" x="36.6" y="483.6" textLength="97.6" clip-path="url(#breeze-help-line-19)">-verbose</text><text class="breeze-help-r6" x="158.6" y="483.6" textLength="24.4" clip-path="url(#breeze-help-line-19)">-v</text><text class="breeze-help-r2" x= [...]
+</text><text class="breeze-help-r4" x="0" y="508" textLength="12.2" clip-path="url(#breeze-help-line-20)">│</text><text class="breeze-help-r5" x="24.4" y="508" textLength="12.2" clip-path="url(#breeze-help-line-20)">-</text><text class="breeze-help-r5" x="36.6" y="508" textLength="48.8" clip-path="url(#breeze-help-line-20)">-dry</text><text class="breeze-help-r5" x="85.4" y="508" textLength="48.8" clip-path="url(#breeze-help-line-20)">-run</text><text class="breeze-help-r6" x="158.6" y=" [...]
+</text><text class="breeze-help-r4" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-help-line-21)">│</text><text class="breeze-help-r5" x="24.4" y="532.4" textLength="12.2" clip-path="url(#breeze-help-line-21)">-</text><text class="breeze-help-r5" x="36.6" y="532.4" textLength="85.4" clip-path="url(#breeze-help-line-21)">-answer</text><text class="breeze-help-r6" x="158.6" y="532.4" textLength="24.4" clip-path="url(#breeze-help-line-21)">-a</text><text class="breeze-help-r2" x=" [...]
+</text><text class="breeze-help-r4" x="0" y="556.8" textLength="12.2" clip-path="url(#breeze-help-line-22)">│</text><text class="breeze-help-r5" x="24.4" y="556.8" textLength="12.2" clip-path="url(#breeze-help-line-22)">-</text><text class="breeze-help-r5" x="36.6" y="556.8" textLength="61" clip-path="url(#breeze-help-line-22)">-help</text><text class="breeze-help-r6" x="158.6" y="556.8" textLength="24.4" clip-path="url(#breeze-help-line-22)">-h</text><text class="breeze-help-r2" x="207. [...]
+</text><text class="breeze-help-r4" x="0" y="581.2" textLength="1464" clip-path="url(#breeze-help-line-23)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-help-r2" x="1464" y="581.2" textLength="12.2" clip-path="url(#breeze-help-line-23)">
+</text><text class="breeze-help-r4" x="0" y="605.6" textLength="24.4" clip-path="url(#breeze-help-line-24)">╭─</text><text class="breeze-help-r4" x="24.4" y="605.6" textLength="317.2" clip-path="url(#breeze-help-line-24)"> Basic developer commands </text><text class="breeze-help-r4" x="341.6" y="605.6" textLength="1098" clip-path="url(#breeze-help-line-24)">──────────────────────────────────────────────────────────────────────────────────────────</text><text class="br [...]
+</text><text class="breeze-help-r4" x="0" y="630" textLength="12.2" clip-path="url(#breeze-help-line-25)">│</text><text class="breeze-help-r5" x="24.4" y="630" textLength="219.6" clip-path="url(#breeze-help-line-25)">start-airflow     </text><text class="breeze-help-r2" x="268.4" y="630" textLength="1171.2" clip-path="url(#breeze-help-line-25)">Enter breeze environment and starts all Airflow components in the tmux [...]
+</text><text class="breeze-help-r4" x="0" y="654.4" textLength="12.2" clip-path="url(#breeze-help-line-26)">│</text><text class="breeze-help-r2" x="268.4" y="654.4" textLength="1171.2" clip-path="url(#breeze-help-line-26)">if contents of www directory changed.                                   [...]
+</text><text class="breeze-help-r4" x="0" y="678.8" textLength="12.2" clip-path="url(#breeze-help-line-27)">│</text><text class="breeze-help-r5" x="24.4" y="678.8" textLength="219.6" clip-path="url(#breeze-help-line-27)">static-checks     </text><text class="breeze-help-r2" x="268.4" y="678.8" textLength="1171.2" clip-path="url(#breeze-help-line-27)">Run static checks.              &# [...]
+</text><text class="breeze-help-r4" x="0" y="703.2" textLength="12.2" clip-path="url(#breeze-help-line-28)">│</text><text class="breeze-help-r5" x="24.4" y="703.2" textLength="219.6" clip-path="url(#breeze-help-line-28)">build-docs        </text><text class="breeze-help-r2" x="268.4" y="703.2" textLength="1171.2" clip-path="url(#breeze-help-line-28)">Build documentation in the container.        [...]
+</text><text class="breeze-help-r4" x="0" y="727.6" textLength="12.2" clip-path="url(#breeze-help-line-29)">│</text><text class="breeze-help-r5" x="24.4" y="727.6" textLength="219.6" clip-path="url(#breeze-help-line-29)">stop              </text><text class="breeze-help-r2" x="268.4" y="727.6" textLength="1171.2" clip-path="url(#breeze-help-line-29)">Stop running breeze environment.    [...]
+</text><text class="breeze-help-r4" x="0" y="752" textLength="12.2" clip-path="url(#breeze-help-line-30)">│</text><text class="breeze-help-r5" x="24.4" y="752" textLength="219.6" clip-path="url(#breeze-help-line-30)">shell             </text><text class="breeze-help-r2" x="268.4" y="752" textLength="1171.2" clip-path="url(#breeze-help-line-30)">Enter breeze environment. this is the default  [...]
+</text><text class="breeze-help-r4" x="0" y="776.4" textLength="12.2" clip-path="url(#breeze-help-line-31)">│</text><text class="breeze-help-r5" x="24.4" y="776.4" textLength="219.6" clip-path="url(#breeze-help-line-31)">exec              </text><text class="breeze-help-r2" x="268.4" y="776.4" textLength="1171.2" clip-path="url(#breeze-help-line-31)">Joins the interactive shell of running  [...]
+</text><text class="breeze-help-r4" x="0" y="800.8" textLength="12.2" clip-path="url(#breeze-help-line-32)">│</text><text class="breeze-help-r5" x="24.4" y="800.8" textLength="219.6" clip-path="url(#breeze-help-line-32)">compile-www-assets</text><text class="breeze-help-r2" x="268.4" y="800.8" textLength="1171.2" clip-path="url(#breeze-help-line-32)">Compiles www assets.                  & [...]
+</text><text class="breeze-help-r4" x="0" y="825.2" textLength="12.2" clip-path="url(#breeze-help-line-33)">│</text><text class="breeze-help-r5" x="24.4" y="825.2" textLength="219.6" clip-path="url(#breeze-help-line-33)">cleanup           </text><text class="breeze-help-r2" x="268.4" y="825.2" textLength="1171.2" clip-path="url(#breeze-help-line-33)">Cleans the cache of parameters, docker cache and& [...]
+</text><text class="breeze-help-r4" x="0" y="849.6" textLength="1464" clip-path="url(#breeze-help-line-34)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-help-r2" x="1464" y="849.6" textLength="12.2" clip-path="url(#breeze-help-line-34)">
+</text><text class="breeze-help-r4" x="0" y="874" textLength="24.4" clip-path="url(#breeze-help-line-35)">╭─</text><text class="breeze-help-r4" x="24.4" y="874" textLength="305" clip-path="url(#breeze-help-line-35)"> Advanced command groups </text><text class="breeze-help-r4" x="329.4" y="874" textLength="1110.2" clip-path="url(#breeze-help-line-35)">───────────────────────────────────────────────────────────────────────────────────────────</text><text class="breeze-h [...]
+</text><text class="breeze-help-r4" x="0" y="898.4" textLength="12.2" clip-path="url(#breeze-help-line-36)">│</text><text class="breeze-help-r5" x="24.4" y="898.4" textLength="280.6" clip-path="url(#breeze-help-line-36)">testing                </text><text class="breeze-help-r2" x="329.4" y="898.4" textLength="1110.2" clip-path="url(#breeze-help-line-36)">Tools that developers can use  [...]
+</text><text class="breeze-help-r4" x="0" y="922.8" textLength="12.2" clip-path="url(#breeze-help-line-37)">│</text><text class="breeze-help-r5" x="24.4" y="922.8" textLength="280.6" clip-path="url(#breeze-help-line-37)">ci-image               </text><text class="breeze-help-r2" x="329.4" y="922.8" textLength="1110.2" clip-path="url(#breeze-help-line-37)">Tools that developers can use to&# [...]
+</text><text class="breeze-help-r4" x="0" y="947.2" textLength="12.2" clip-path="url(#breeze-help-line-38)">│</text><text class="breeze-help-r5" x="24.4" y="947.2" textLength="280.6" clip-path="url(#breeze-help-line-38)">k8s                    </text><text class="breeze-help-r2" x="329.4" y="947.2" textLength="1110.2" clip-path="url(#breeze-help-line-38)">Tools that developers [...]
+</text><text class="breeze-help-r4" x="0" y="971.6" textLength="12.2" clip-path="url(#breeze-help-line-39)">│</text><text class="breeze-help-r5" x="24.4" y="971.6" textLength="280.6" clip-path="url(#breeze-help-line-39)">prod-image             </text><text class="breeze-help-r2" x="329.4" y="971.6" textLength="1110.2" clip-path="url(#breeze-help-line-39)">Tools that developers can use to manual [...]
+</text><text class="breeze-help-r4" x="0" y="996" textLength="12.2" clip-path="url(#breeze-help-line-40)">│</text><text class="breeze-help-r5" x="24.4" y="996" textLength="280.6" clip-path="url(#breeze-help-line-40)">setup                  </text><text class="breeze-help-r2" x="329.4" y="996" textLength="1110.2" clip-path="url(#breeze-help-line-40)">Tools that developers can use& [...]
+</text><text class="breeze-help-r4" x="0" y="1020.4" textLength="12.2" clip-path="url(#breeze-help-line-41)">│</text><text class="breeze-help-r5" x="24.4" y="1020.4" textLength="280.6" clip-path="url(#breeze-help-line-41)">release-management     </text><text class="breeze-help-r2" x="329.4" y="1020.4" textLength="1110.2" clip-path="url(#breeze-help-line-41)">Tools that release managers can use to prepare and manage [...]
+</text><text class="breeze-help-r4" x="0" y="1044.8" textLength="12.2" clip-path="url(#breeze-help-line-42)">│</text><text class="breeze-help-r5" x="24.4" y="1044.8" textLength="280.6" clip-path="url(#breeze-help-line-42)">ci                     </text><text class="breeze-help-r2" x="329.4" y="1044.8" textLength="1110.2" clip-path="url(#breeze-help-line-42)">Tools that CI [...]
+</text><text class="breeze-help-r4" x="0" y="1069.2" textLength="1464" clip-path="url(#breeze-help-line-43)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-help-r2" x="1464" y="1069.2" textLength="12.2" clip-path="url(#breeze-help-line-43)">
</text>
</g>
</g>
diff --git a/images/breeze/output_release-management.svg b/images/breeze/output_release-management.svg
index c81b2ac637..1a70516162 100644
--- a/images/breeze/output_release-management.svg
+++ b/images/breeze/output_release-management.svg
@@ -1,4 +1,4 @@
-<svg class="rich-terminal" viewBox="0 0 1482 440.4" xmlns="http://www.w3.org/2000/svg">
+<svg class="rich-terminal" viewBox="0 0 1482 464.79999999999995" xmlns="http://www.w3.org/2000/svg">
<!-- Generated with Rich https://www.textualize.io -->
<style>
@@ -32,17 +32,17 @@
font-family: arial;
}
- .terminal-1770299259-r1 { fill: #c5c8c6;font-weight: bold }
-.terminal-1770299259-r2 { fill: #c5c8c6 }
-.terminal-1770299259-r3 { fill: #d0b344;font-weight: bold }
-.terminal-1770299259-r4 { fill: #68a0b3;font-weight: bold }
-.terminal-1770299259-r5 { fill: #868887 }
-.terminal-1770299259-r6 { fill: #98a84b;font-weight: bold }
+ .breeze-release-management-r1 { fill: #c5c8c6;font-weight: bold }
+.breeze-release-management-r2 { fill: #c5c8c6 }
+.breeze-release-management-r3 { fill: #d0b344;font-weight: bold }
+.breeze-release-management-r4 { fill: #868887 }
+.breeze-release-management-r5 { fill: #68a0b3;font-weight: bold }
+.breeze-release-management-r6 { fill: #98a84b;font-weight: bold }
</style>
<defs>
- <clipPath id="terminal-1770299259-clip-terminal">
- <rect x="0" y="0" width="1463.0" height="389.4" />
+ <clipPath id="breeze-release-management-clip-terminal">
+ <rect x="0" y="0" width="1463.0" height="413.79999999999995" />
</clipPath>
<clipPath id="terminal-1770299259-line-0">
<rect x="0" y="1.5" width="1464" height="24.65"/>
@@ -89,9 +89,12 @@
<clipPath id="terminal-1770299259-line-14">
<rect x="0" y="343.1" width="1464" height="24.65"/>
</clipPath>
+<clipPath id="breeze-release-management-line-15">
+ <rect x="0" y="367.5" width="1464" height="24.65"/>
+ </clipPath>
</defs>
- <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="438.4" rx="8"/><text class="terminal-1770299259-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: release-management</text>
+ <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="462.8" rx="8"/><text class="breeze-release-management-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: release-management</text>
<g transform="translate(26,22)">
<circle cx="0" cy="0" r="7" fill="#ff5f57"/>
<circle cx="22" cy="0" r="7" fill="#febc2e"/>
@@ -100,23 +103,24 @@
<g transform="translate(9, 41)" clip-path="url(#terminal-1770299259-clip-terminal)">
- <g class="terminal-1770299259-matrix">
- <text class="terminal-1770299259-r2" x="1464" y="20" textLength="12.2" clip-path="url(#terminal-1770299259-line-0)">
-</text><text class="terminal-1770299259-r3" x="12.2" y="44.4" textLength="85.4" clip-path="url(#terminal-1770299259-line-1)">Usage: </text><text class="terminal-1770299259-r1" x="97.6" y="44.4" textLength="329.4" clip-path="url(#terminal-1770299259-line-1)">breeze release-management [</text><text class="terminal-1770299259-r4" x="427" y="44.4" textLength="85.4" clip-path="url(#terminal-1770299259-line-1)">OPTIONS</text><text class="terminal-1770299259-r1" x="512.4" y="44.4 [...]
-</text><text class="terminal-1770299259-r2" x="1464" y="68.8" textLength="12.2" clip-path="url(#terminal-1770299259-line-2)">
-</text><text class="terminal-1770299259-r2" x="12.2" y="93.2" textLength="902.8" clip-path="url(#terminal-1770299259-line-3)">Tools that release managers can use to prepare and manage Airflow releases</text><text class="terminal-1770299259-r2" x="1464" y="93.2" textLength="12.2" clip-path="url(#terminal-1770299259-line-3)">
-</text><text class="terminal-1770299259-r2" x="1464" y="117.6" textLength="12.2" clip-path="url(#terminal-1770299259-line-4)">
-</text><text class="terminal-1770299259-r5" x="0" y="142" textLength="24.4" clip-path="url(#terminal-1770299259-line-5)">╭─</text><text class="terminal-1770299259-r5" x="24.4" y="142" textLength="195.2" clip-path="url(#terminal-1770299259-line-5)"> Common options </text><text class="terminal-1770299259-r5" x="219.6" y="142" textLength="1220" clip-path="url(#terminal-1770299259-line-5)">──────────────────────────────────────────────────────────────────────────────────────── [...]
-</text><text class="terminal-1770299259-r5" x="0" y="166.4" textLength="12.2" clip-path="url(#terminal-1770299259-line-6)">│</text><text class="terminal-1770299259-r4" x="24.4" y="166.4" textLength="12.2" clip-path="url(#terminal-1770299259-line-6)">-</text><text class="terminal-1770299259-r4" x="36.6" y="166.4" textLength="61" clip-path="url(#terminal-1770299259-line-6)">-help</text><text class="terminal-1770299259-r6" x="122" y="166.4" textLength="24.4" clip-path="url(#terminal-1770299 [...]
-</text><text class="terminal-1770299259-r5" x="0" y="190.8" textLength="1464" clip-path="url(#terminal-1770299259-line-7)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="terminal-1770299259-r2" x="1464" y="190.8" textLength="12.2" clip-path="url(#terminal-1770299259-line-7)">
-</text><text class="terminal-1770299259-r5" x="0" y="215.2" textLength="24.4" clip-path="url(#terminal-1770299259-line-8)">╭─</text><text class="terminal-1770299259-r5" x="24.4" y="215.2" textLength="122" clip-path="url(#terminal-1770299259-line-8)"> Commands </text><text class="terminal-1770299259-r5" x="146.4" y="215.2" textLength="1293.2" clip-path="url(#terminal-1770299259-line-8)">───────────────────────────────────────────────────────────────────────────────────────────── [...]
-</text><text class="terminal-1770299259-r5" x="0" y="239.6" textLength="12.2" clip-path="url(#terminal-1770299259-line-9)">│</text><text class="terminal-1770299259-r4" x="24.4" y="239.6" textLength="402.6" clip-path="url(#terminal-1770299259-line-9)">generate-constraints             </text><text class="terminal-1770299259-r2" x="451.4" y="239.6" textLength="988.2" clip-path="url(#terminal-1770299259-line-9)">Generates  [...]
-</text><text class="terminal-1770299259-r5" x="0" y="264" textLength="12.2" clip-path="url(#terminal-1770299259-line-10)">│</text><text class="terminal-1770299259-r4" x="24.4" y="264" textLength="402.6" clip-path="url(#terminal-1770299259-line-10)">prepare-airflow-package          </text><text class="terminal-1770299259-r2" x="451.4" y="264" textLength="988.2" clip-path="url(#terminal-1770299259-line-10)">Prepare sdist/whl packa [...]
-</text><text class="terminal-1770299259-r5" x="0" y="288.4" textLength="12.2" clip-path="url(#terminal-1770299259-line-11)">│</text><text class="terminal-1770299259-r4" x="24.4" y="288.4" textLength="402.6" clip-path="url(#terminal-1770299259-line-11)">prepare-provider-documentation   </text><text class="terminal-1770299259-r2" x="451.4" y="288.4" textLength="97.6" clip-path="url(#terminal-1770299259-line-11)">Prepare </text><text class="terminal-1770299259-r4" x="549 [...]
-</text><text class="terminal-1770299259-r5" x="0" y="312.8" textLength="12.2" clip-path="url(#terminal-1770299259-line-12)">│</text><text class="terminal-1770299259-r4" x="24.4" y="312.8" textLength="402.6" clip-path="url(#terminal-1770299259-line-12)">prepare-provider-packages        </text><text class="terminal-1770299259-r2" x="451.4" y="312.8" textLength="988.2" clip-path="url(#terminal-1770299259-line-12)">Prepare sdist/whl packages& [...]
-</text><text class="terminal-1770299259-r5" x="0" y="337.2" textLength="12.2" clip-path="url(#terminal-1770299259-line-13)">│</text><text class="terminal-1770299259-r4" x="24.4" y="337.2" textLength="402.6" clip-path="url(#terminal-1770299259-line-13)">release-prod-images              </text><text class="terminal-1770299259-r2" x="451.4" y="337.2" textLength="988.2" clip-path="url(#terminal-1770299259-line-13)">Release [...]
-</text><text class="terminal-1770299259-r5" x="0" y="361.6" textLength="12.2" clip-path="url(#terminal-1770299259-line-14)">│</text><text class="terminal-1770299259-r4" x="24.4" y="361.6" textLength="402.6" clip-path="url(#terminal-1770299259-line-14)">verify-provider-packages         </text><text class="terminal-1770299259-r2" x="451.4" y="361.6" textLength="988.2" clip-path="url(#terminal-1770299259-line-14)">Verifies if all p [...]
-</text><text class="terminal-1770299259-r5" x="0" y="386" textLength="1464" clip-path="url(#terminal-1770299259-line-15)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="terminal-1770299259-r2" x="1464" y="386" textLength="12.2" clip-path="url(#terminal-1770299259-line-15)">
+ <g class="breeze-release-management-matrix">
+ <text class="breeze-release-management-r2" x="1464" y="20" textLength="12.2" clip-path="url(#breeze-release-management-line-0)">
+</text><text class="breeze-release-management-r3" x="12.2" y="44.4" textLength="85.4" clip-path="url(#breeze-release-management-line-1)">Usage: </text><text class="breeze-release-management-r1" x="97.6" y="44.4" textLength="646.6" clip-path="url(#breeze-release-management-line-1)">breeze release-management [OPTIONS] COMMAND [ARGS]...</text><text class="breeze-release-management-r2" x="1464" y="44.4" textLength="12.2" clip-path="url(#breeze-release-management-line-1)">
+</text><text class="breeze-release-management-r2" x="1464" y="68.8" textLength="12.2" clip-path="url(#breeze-release-management-line-2)">
+</text><text class="breeze-release-management-r2" x="12.2" y="93.2" textLength="902.8" clip-path="url(#breeze-release-management-line-3)">Tools that release managers can use to prepare and manage Airflow releases</text><text class="breeze-release-management-r2" x="1464" y="93.2" textLength="12.2" clip-path="url(#breeze-release-management-line-3)">
+</text><text class="breeze-release-management-r2" x="1464" y="117.6" textLength="12.2" clip-path="url(#breeze-release-management-line-4)">
+</text><text class="breeze-release-management-r4" x="0" y="142" textLength="24.4" clip-path="url(#breeze-release-management-line-5)">╭─</text><text class="breeze-release-management-r4" x="24.4" y="142" textLength="195.2" clip-path="url(#breeze-release-management-line-5)"> Common options </text><text class="breeze-release-management-r4" x="219.6" y="142" textLength="1220" clip-path="url(#breeze-release-management-line-5)">──────────────────────────────────────────────────── [...]
+</text><text class="breeze-release-management-r4" x="0" y="166.4" textLength="12.2" clip-path="url(#breeze-release-management-line-6)">│</text><text class="breeze-release-management-r5" x="24.4" y="166.4" textLength="12.2" clip-path="url(#breeze-release-management-line-6)">-</text><text class="breeze-release-management-r5" x="36.6" y="166.4" textLength="61" clip-path="url(#breeze-release-management-line-6)">-help</text><text class="breeze-release-management-r6" x="122" y="166.4" textLeng [...]
+</text><text class="breeze-release-management-r4" x="0" y="190.8" textLength="1464" clip-path="url(#breeze-release-management-line-7)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-release-management-r2" x="1464" y="190.8" textLength="12.2" clip-path="url(#breeze-release-management-line-7)">
+</text><text class="breeze-release-management-r4" x="0" y="215.2" textLength="24.4" clip-path="url(#breeze-release-management-line-8)">╭─</text><text class="breeze-release-management-r4" x="24.4" y="215.2" textLength="122" clip-path="url(#breeze-release-management-line-8)"> Commands </text><text class="breeze-release-management-r4" x="146.4" y="215.2" textLength="1293.2" clip-path="url(#breeze-release-management-line-8)">───────────────────────────────────────────────────────── [...]
+</text><text class="breeze-release-management-r4" x="0" y="239.6" textLength="12.2" clip-path="url(#breeze-release-management-line-9)">│</text><text class="breeze-release-management-r5" x="24.4" y="239.6" textLength="402.6" clip-path="url(#breeze-release-management-line-9)">generate-constraints             </text><text class="breeze-release-management-r2" x="451.4" y="239.6" textLength="988.2" clip-path="url(#breeze-release [...]
+</text><text class="breeze-release-management-r4" x="0" y="264" textLength="12.2" clip-path="url(#breeze-release-management-line-10)">│</text><text class="breeze-release-management-r5" x="24.4" y="264" textLength="402.6" clip-path="url(#breeze-release-management-line-10)">generate-issue-content           </text><text class="breeze-release-management-r2" x="451.4" y="264" textLength="988.2" clip-path="url(#breeze-release-management-li [...]
+</text><text class="breeze-release-management-r4" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-release-management-line-11)">│</text><text class="breeze-release-management-r5" x="24.4" y="288.4" textLength="402.6" clip-path="url(#breeze-release-management-line-11)">prepare-airflow-package          </text><text class="breeze-release-management-r2" x="451.4" y="288.4" textLength="988.2" clip-path="url(#breeze-release-management-l [...]
+</text><text class="breeze-release-management-r4" x="0" y="312.8" textLength="12.2" clip-path="url(#breeze-release-management-line-12)">│</text><text class="breeze-release-management-r5" x="24.4" y="312.8" textLength="402.6" clip-path="url(#breeze-release-management-line-12)">prepare-provider-documentation   </text><text class="breeze-release-management-r2" x="451.4" y="312.8" textLength="988.2" clip-path="url(#breeze-release-management-line-12)">Prepare CHANGELOG, [...]
+</text><text class="breeze-release-management-r4" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-release-management-line-13)">│</text><text class="breeze-release-management-r5" x="24.4" y="337.2" textLength="402.6" clip-path="url(#breeze-release-management-line-13)">prepare-provider-packages        </text><text class="breeze-release-management-r2" x="451.4" y="337.2" textLength="988.2" clip-path="url(#breeze-release-management-line-13)">P [...]
+</text><text class="breeze-release-management-r4" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-release-management-line-14)">│</text><text class="breeze-release-management-r5" x="24.4" y="361.6" textLength="402.6" clip-path="url(#breeze-release-management-line-14)">release-prod-images              </text><text class="breeze-release-management-r2" x="451.4" y="361.6" textLength="988.2" clip-path="url(#breeze- [...]
+</text><text class="breeze-release-management-r4" x="0" y="386" textLength="12.2" clip-path="url(#breeze-release-management-line-15)">│</text><text class="breeze-release-management-r5" x="24.4" y="386" textLength="402.6" clip-path="url(#breeze-release-management-line-15)">verify-provider-packages         </text><text class="breeze-release-management-r2" x="451.4" y="386" textLength="988.2" clip-path="url(#breeze-release-management-line-15)">Ve [...]
+</text><text class="breeze-release-management-r4" x="0" y="410.4" textLength="1464" clip-path="url(#breeze-release-management-line-16)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-release-management-r2" x="1464" y="410.4" textLength="12.2" clip-path="url(#breeze-release-management-line-16)">
</text>
</g>
</g>
diff --git a/images/breeze/output_release-management_generate-issue-content.svg b/images/breeze/output_release-management_generate-issue-content.svg
new file mode 100644
index 0000000000..4dfa2f3649
--- /dev/null
+++ b/images/breeze/output_release-management_generate-issue-content.svg
@@ -0,0 +1,176 @@
+<svg class="rich-terminal" viewBox="0 0 1482 757.5999999999999" xmlns="http://www.w3.org/2000/svg">
+ <!-- Generated with Rich https://www.textualize.io -->
+ <style>
+
+ @font-face {
+ font-family: "Fira Code";
+ src: local("FiraCode-Regular"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff2/FiraCode-Regular.woff2") format("woff2"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff/FiraCode-Regular.woff") format("woff");
+ font-style: normal;
+ font-weight: 400;
+ }
+ @font-face {
+ font-family: "Fira Code";
+ src: local("FiraCode-Bold"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff2/FiraCode-Bold.woff2") format("woff2"),
+ url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff/FiraCode-Bold.woff") format("woff");
+ font-style: bold;
+ font-weight: 700;
+ }
+
+ .breeze-release-management-generate-issue-content-matrix {
+ font-family: Fira Code, monospace;
+ font-size: 20px;
+ line-height: 24.4px;
+ font-variant-east-asian: full-width;
+ }
+
+ .breeze-release-management-generate-issue-content-title {
+ font-size: 18px;
+ font-weight: bold;
+ font-family: arial;
+ }
+
+ .breeze-release-management-generate-issue-content-r1 { fill: #c5c8c6;font-weight: bold }
+.breeze-release-management-generate-issue-content-r2 { fill: #c5c8c6 }
+.breeze-release-management-generate-issue-content-r3 { fill: #d0b344;font-weight: bold }
+.breeze-release-management-generate-issue-content-r4 { fill: #868887 }
+.breeze-release-management-generate-issue-content-r5 { fill: #68a0b3;font-weight: bold }
+.breeze-release-management-generate-issue-content-r6 { fill: #8d7b39 }
+.breeze-release-management-generate-issue-content-r7 { fill: #98a84b;font-weight: bold }
+ </style>
+
+ <defs>
+ <clipPath id="breeze-release-management-generate-issue-content-clip-terminal">
+ <rect x="0" y="0" width="1463.0" height="706.5999999999999" />
+ </clipPath>
+ <clipPath id="breeze-release-management-generate-issue-content-line-0">
+ <rect x="0" y="1.5" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-generate-issue-content-line-1">
+ <rect x="0" y="25.9" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-generate-issue-content-line-2">
+ <rect x="0" y="50.3" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-generate-issue-content-line-3">
+ <rect x="0" y="74.7" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-generate-issue-content-line-4">
+ <rect x="0" y="99.1" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-generate-issue-content-line-5">
+ <rect x="0" y="123.5" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-generate-issue-content-line-6">
+ <rect x="0" y="147.9" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-generate-issue-content-line-7">
+ <rect x="0" y="172.3" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-generate-issue-content-line-8">
+ <rect x="0" y="196.7" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-generate-issue-content-line-9">
+ <rect x="0" y="221.1" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-generate-issue-content-line-10">
+ <rect x="0" y="245.5" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-generate-issue-content-line-11">
+ <rect x="0" y="269.9" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-generate-issue-content-line-12">
+ <rect x="0" y="294.3" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-generate-issue-content-line-13">
+ <rect x="0" y="318.7" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-generate-issue-content-line-14">
+ <rect x="0" y="343.1" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-generate-issue-content-line-15">
+ <rect x="0" y="367.5" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-generate-issue-content-line-16">
+ <rect x="0" y="391.9" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-generate-issue-content-line-17">
+ <rect x="0" y="416.3" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-generate-issue-content-line-18">
+ <rect x="0" y="440.7" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-generate-issue-content-line-19">
+ <rect x="0" y="465.1" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-generate-issue-content-line-20">
+ <rect x="0" y="489.5" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-generate-issue-content-line-21">
+ <rect x="0" y="513.9" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-generate-issue-content-line-22">
+ <rect x="0" y="538.3" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-generate-issue-content-line-23">
+ <rect x="0" y="562.7" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-generate-issue-content-line-24">
+ <rect x="0" y="587.1" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-generate-issue-content-line-25">
+ <rect x="0" y="611.5" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-generate-issue-content-line-26">
+ <rect x="0" y="635.9" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-release-management-generate-issue-content-line-27">
+ <rect x="0" y="660.3" width="1464" height="24.65"/>
+ </clipPath>
+ </defs>
+
+ <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="755.6" rx="8"/><text class="breeze-release-management-generate-issue-content-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: release-management generate-issue-content</text>
+ <g transform="translate(26,22)">
+ <circle cx="0" cy="0" r="7" fill="#ff5f57"/>
+ <circle cx="22" cy="0" r="7" fill="#febc2e"/>
+ <circle cx="44" cy="0" r="7" fill="#28c840"/>
+ </g>
+
+ <g transform="translate(9, 41)" clip-path="url(#breeze-release-management-generate-issue-content-clip-terminal)">
+
+ <g class="breeze-release-management-generate-issue-content-matrix">
+ <text class="breeze-release-management-generate-issue-content-r2" x="1464" y="20" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-0)">
+</text><text class="breeze-release-management-generate-issue-content-r3" x="12.2" y="44.4" textLength="85.4" clip-path="url(#breeze-release-management-generate-issue-content-line-1)">Usage: </text><text class="breeze-release-management-generate-issue-content-r1" x="97.6" y="44.4" textLength="1244.4" clip-path="url(#breeze-release-management-generate-issue-content-line-1)">breeze release-management generate-issue-content [OPTIONS] [airbyte | alibaba  [...]
+</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="68.8" textLength="1439.6" clip-path="url(#breeze-release-management-generate-issue-content-line-2)">                                                   & [...]
+</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="93.2" textLength="1354.2" clip-path="url(#breeze-release-management-generate-issue-content-line-3)">                                                   & [...]
+</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="117.6" textLength="1427.4" clip-path="url(#breeze-release-management-generate-issue-content-line-4)">                                                    [...]
+</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="142" textLength="1366.4" clip-path="url(#breeze-release-management-generate-issue-content-line-5)">                                                   &# [...]
+</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="166.4" textLength="1390.8" clip-path="url(#breeze-release-management-generate-issue-content-line-6)">                                                    [...]
+</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="190.8" textLength="1415.2" clip-path="url(#breeze-release-management-generate-issue-content-line-7)">                                                    [...]
+</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="215.2" textLength="1427.4" clip-path="url(#breeze-release-management-generate-issue-content-line-8)">                                                    [...]
+</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="239.6" textLength="1317.6" clip-path="url(#breeze-release-management-generate-issue-content-line-9)">                                                    [...]
+</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="264" textLength="1390.8" clip-path="url(#breeze-release-management-generate-issue-content-line-10)">                                                   & [...]
+</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="288.4" textLength="1427.4" clip-path="url(#breeze-release-management-generate-issue-content-line-11)">                                                    [...]
+</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="312.8" textLength="1390.8" clip-path="url(#breeze-release-management-generate-issue-content-line-12)">                                                    [...]
+</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="337.2" textLength="1378.6" clip-path="url(#breeze-release-management-generate-issue-content-line-13)">                                                    [...]
+</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="361.6" textLength="1378.6" clip-path="url(#breeze-release-management-generate-issue-content-line-14)">                                                    [...]
+</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="386" textLength="1146.8" clip-path="url(#breeze-release-management-generate-issue-content-line-15)">                                                   & [...]
+</text><text class="breeze-release-management-generate-issue-content-r2" x="1464" y="410.4" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-16)">
+</text><text class="breeze-release-management-generate-issue-content-r2" x="12.2" y="434.8" textLength="585.6" clip-path="url(#breeze-release-management-generate-issue-content-line-17)">Generates content for issue to test the release.</text><text class="breeze-release-management-generate-issue-content-r2" x="1464" y="434.8" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-17)">
+</text><text class="breeze-release-management-generate-issue-content-r2" x="1464" y="459.2" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-18)">
+</text><text class="breeze-release-management-generate-issue-content-r4" x="0" y="483.6" textLength="24.4" clip-path="url(#breeze-release-management-generate-issue-content-line-19)">╭─</text><text class="breeze-release-management-generate-issue-content-r4" x="24.4" y="483.6" textLength="195.2" clip-path="url(#breeze-release-management-generate-issue-content-line-19)"> Common options </text><text class="breeze-release-management-generate-issue-content-r4" x="219.6" y="483.6 [...]
+</text><text class="breeze-release-management-generate-issue-content-r4" x="0" y="508" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-20)">│</text><text class="breeze-release-management-generate-issue-content-r5" x="24.4" y="508" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-20)">-</text><text class="breeze-release-management-generate-issue-content-r5" x="36.6" y="508" textLength="85.4" clip-path="url(#bre [...]
+</text><text class="breeze-release-management-generate-issue-content-r4" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-21)">│</text><text class="breeze-release-management-generate-issue-content-r2" x="390.4" y="532.4" textLength="1049.2" clip-path="url(#breeze-release-management-generate-issue-content-line-21)">variable set. Can be generated with:           [...]
+</text><text class="breeze-release-management-generate-issue-content-r4" x="0" y="556.8" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-22)">│</text><text class="breeze-release-management-generate-issue-content-r2" x="390.4" y="556.8" textLength="1049.2" clip-path="url(#breeze-release-management-generate-issue-content-line-22)">https://github.com/settings/tokens/new?description=Read%20sssues&scopes=repo:status   </text><text cla [...]
+</text><text class="breeze-release-management-generate-issue-content-r4" x="0" y="581.2" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-23)">│</text><text class="breeze-release-management-generate-issue-content-r6" x="390.4" y="581.2" textLength="1049.2" clip-path="url(#breeze-release-management-generate-issue-content-line-23)">(TEXT)                   & [...]
+</text><text class="breeze-release-management-generate-issue-content-r4" x="0" y="605.6" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-24)">│</text><text class="breeze-release-management-generate-issue-content-r5" x="24.4" y="605.6" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-24)">-</text><text class="breeze-release-management-generate-issue-content-r5" x="36.6" y="605.6" textLength="85.4" clip-path="ur [...]
+</text><text class="breeze-release-management-generate-issue-content-r4" x="0" y="630" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-25)">│</text><text class="breeze-release-management-generate-issue-content-r5" x="24.4" y="630" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-25)">-</text><text class="breeze-release-management-generate-issue-content-r5" x="36.6" y="630" textLength="61" clip-path="url(#breez [...]
+</text><text class="breeze-release-management-generate-issue-content-r4" x="0" y="654.4" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-26)">│</text><text class="breeze-release-management-generate-issue-content-r5" x="24.4" y="654.4" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-26)">-</text><text class="breeze-release-management-generate-issue-content-r5" x="36.6" y="654.4" textLength="109.8" clip-path="u [...]
+</text><text class="breeze-release-management-generate-issue-content-r4" x="0" y="678.8" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-27)">│</text><text class="breeze-release-management-generate-issue-content-r5" x="24.4" y="678.8" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-27)">-</text><text class="breeze-release-management-generate-issue-content-r5" x="36.6" y="678.8" textLength="61" clip-path="url( [...]
+</text><text class="breeze-release-management-generate-issue-content-r4" x="0" y="703.2" textLength="1464" clip-path="url(#breeze-release-management-generate-issue-content-line-28)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-release-management-generate-issue-content-r2" x="1464" y="703.2" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-28)">
+</text>
+ </g>
+ </g>
+</svg>
diff --git a/images/breeze/output_setup.svg b/images/breeze/output_setup.svg
index 206ba39938..91df1eb737 100644
--- a/images/breeze/output_setup.svg
+++ b/images/breeze/output_setup.svg
@@ -35,8 +35,8 @@
.breeze-setup-r1 { fill: #c5c8c6;font-weight: bold }
.breeze-setup-r2 { fill: #c5c8c6 }
.breeze-setup-r3 { fill: #d0b344;font-weight: bold }
-.breeze-setup-r4 { fill: #68a0b3;font-weight: bold }
-.breeze-setup-r5 { fill: #868887 }
+.breeze-setup-r4 { fill: #868887 }
+.breeze-setup-r5 { fill: #68a0b3;font-weight: bold }
.breeze-setup-r6 { fill: #98a84b;font-weight: bold }
</style>
@@ -99,20 +99,20 @@
<g class="breeze-setup-matrix">
<text class="breeze-setup-r2" x="1464" y="20" textLength="12.2" clip-path="url(#breeze-setup-line-0)">
-</text><text class="breeze-setup-r3" x="12.2" y="44.4" textLength="85.4" clip-path="url(#breeze-setup-line-1)">Usage: </text><text class="breeze-setup-r1" x="97.6" y="44.4" textLength="170.8" clip-path="url(#breeze-setup-line-1)">breeze setup [</text><text class="breeze-setup-r4" x="268.4" y="44.4" textLength="85.4" clip-path="url(#breeze-setup-line-1)">OPTIONS</text><text class="breeze-setup-r1" x="353.8" y="44.4" textLength="24.4" clip-path="url(#breeze-setup-line-1)">]& [...]
+</text><text class="breeze-setup-r3" x="12.2" y="44.4" textLength="85.4" clip-path="url(#breeze-setup-line-1)">Usage: </text><text class="breeze-setup-r1" x="97.6" y="44.4" textLength="488" clip-path="url(#breeze-setup-line-1)">breeze setup [OPTIONS] COMMAND [ARGS]...</text><text class="breeze-setup-r2" x="1464" y="44.4" textLength="12.2" clip-path="url(#breeze-setup-line-1)">
</text><text class="breeze-setup-r2" x="1464" y="68.8" textLength="12.2" clip-path="url(#breeze-setup-line-2)">
</text><text class="breeze-setup-r2" x="12.2" y="93.2" textLength="597.8" clip-path="url(#breeze-setup-line-3)">Tools that developers can use to configure Breeze</text><text class="breeze-setup-r2" x="1464" y="93.2" textLength="12.2" clip-path="url(#breeze-setup-line-3)">
</text><text class="breeze-setup-r2" x="1464" y="117.6" textLength="12.2" clip-path="url(#breeze-setup-line-4)">
-</text><text class="breeze-setup-r5" x="0" y="142" textLength="24.4" clip-path="url(#breeze-setup-line-5)">╭─</text><text class="breeze-setup-r5" x="24.4" y="142" textLength="195.2" clip-path="url(#breeze-setup-line-5)"> Common options </text><text class="breeze-setup-r5" x="219.6" y="142" textLength="1220" clip-path="url(#breeze-setup-line-5)">────────────────────────────────────────────────────────────────────────────────────────────────────</text><text class="breeze-set [...]
-</text><text class="breeze-setup-r5" x="0" y="166.4" textLength="12.2" clip-path="url(#breeze-setup-line-6)">│</text><text class="breeze-setup-r4" x="24.4" y="166.4" textLength="12.2" clip-path="url(#breeze-setup-line-6)">-</text><text class="breeze-setup-r4" x="36.6" y="166.4" textLength="61" clip-path="url(#breeze-setup-line-6)">-help</text><text class="breeze-setup-r6" x="122" y="166.4" textLength="24.4" clip-path="url(#breeze-setup-line-6)">-h</text><text class="breeze-setup-r2" x="1 [...]
-</text><text class="breeze-setup-r5" x="0" y="190.8" textLength="1464" clip-path="url(#breeze-setup-line-7)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-setup-r2" x="1464" y="190.8" textLength="12.2" clip-path="url(#breeze-setup-line-7)">
-</text><text class="breeze-setup-r5" x="0" y="215.2" textLength="24.4" clip-path="url(#breeze-setup-line-8)">╭─</text><text class="breeze-setup-r5" x="24.4" y="215.2" textLength="122" clip-path="url(#breeze-setup-line-8)"> Commands </text><text class="breeze-setup-r5" x="146.4" y="215.2" textLength="1293.2" clip-path="url(#breeze-setup-line-8)">──────────────────────────────────────────────────────────────────────────────────────────────────────────</text><text class="breeze-se [...]
-</text><text class="breeze-setup-r5" x="0" y="239.6" textLength="12.2" clip-path="url(#breeze-setup-line-9)">│</text><text class="breeze-setup-r4" x="24.4" y="239.6" textLength="390.4" clip-path="url(#breeze-setup-line-9)">autocomplete                    </text><text class="breeze-setup-r2" x="439.2" y="239.6" textLength="1000.4" clip-path="url(#breeze-setup-line-9)">Enables autocompl [...]
-</text><text class="breeze-setup-r5" x="0" y="264" textLength="12.2" clip-path="url(#breeze-setup-line-10)">│</text><text class="breeze-setup-r4" x="24.4" y="264" textLength="390.4" clip-path="url(#breeze-setup-line-10)">config                          </text><text class="breeze-setup-r2" x="439.2" y="264" textLength="683.2" clip-path="url(#breeze-setup-line-1 [...]
-</text><text class="breeze-setup-r5" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-setup-line-11)">│</text><text class="breeze-setup-r4" x="24.4" y="288.4" textLength="390.4" clip-path="url(#breeze-setup-line-11)">regenerate-command-images       </text><text class="breeze-setup-r2" x="439.2" y="288.4" textLength="1000.4" clip-path="url(#breeze-setup-line-11)">Regenerate breeze command images.       [...]
-</text><text class="breeze-setup-r5" x="0" y="312.8" textLength="12.2" clip-path="url(#breeze-setup-line-12)">│</text><text class="breeze-setup-r4" x="24.4" y="312.8" textLength="390.4" clip-path="url(#breeze-setup-line-12)">self-upgrade                    </text><text class="breeze-setup-r2" x="439.2" y="312.8" textLength="1000.4" clip-path="url(#breeze-setup-line-12)">Self upgrade&# [...]
-</text><text class="breeze-setup-r5" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-setup-line-13)">│</text><text class="breeze-setup-r4" x="24.4" y="337.2" textLength="390.4" clip-path="url(#breeze-setup-line-13)">version                         </text><text class="breeze-setup-r2" x="439.2" y="337.2" textLength="1000.4" clip-path="url(#breeze-setup-line [...]
-</text><text class="breeze-setup-r5" x="0" y="361.6" textLength="1464" clip-path="url(#breeze-setup-line-14)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-setup-r2" x="1464" y="361.6" textLength="12.2" clip-path="url(#breeze-setup-line-14)">
+</text><text class="breeze-setup-r4" x="0" y="142" textLength="24.4" clip-path="url(#breeze-setup-line-5)">╭─</text><text class="breeze-setup-r4" x="24.4" y="142" textLength="195.2" clip-path="url(#breeze-setup-line-5)"> Common options </text><text class="breeze-setup-r4" x="219.6" y="142" textLength="1220" clip-path="url(#breeze-setup-line-5)">────────────────────────────────────────────────────────────────────────────────────────────────────</text><text class="breeze-set [...]
+</text><text class="breeze-setup-r4" x="0" y="166.4" textLength="12.2" clip-path="url(#breeze-setup-line-6)">│</text><text class="breeze-setup-r5" x="24.4" y="166.4" textLength="12.2" clip-path="url(#breeze-setup-line-6)">-</text><text class="breeze-setup-r5" x="36.6" y="166.4" textLength="61" clip-path="url(#breeze-setup-line-6)">-help</text><text class="breeze-setup-r6" x="122" y="166.4" textLength="24.4" clip-path="url(#breeze-setup-line-6)">-h</text><text class="breeze-setup-r2" x="1 [...]
+</text><text class="breeze-setup-r4" x="0" y="190.8" textLength="1464" clip-path="url(#breeze-setup-line-7)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-setup-r2" x="1464" y="190.8" textLength="12.2" clip-path="url(#breeze-setup-line-7)">
+</text><text class="breeze-setup-r4" x="0" y="215.2" textLength="24.4" clip-path="url(#breeze-setup-line-8)">╭─</text><text class="breeze-setup-r4" x="24.4" y="215.2" textLength="122" clip-path="url(#breeze-setup-line-8)"> Commands </text><text class="breeze-setup-r4" x="146.4" y="215.2" textLength="1293.2" clip-path="url(#breeze-setup-line-8)">──────────────────────────────────────────────────────────────────────────────────────────────────────────</text><text class="breeze-se [...]
+</text><text class="breeze-setup-r4" x="0" y="239.6" textLength="12.2" clip-path="url(#breeze-setup-line-9)">│</text><text class="breeze-setup-r5" x="24.4" y="239.6" textLength="390.4" clip-path="url(#breeze-setup-line-9)">autocomplete                    </text><text class="breeze-setup-r2" x="439.2" y="239.6" textLength="1000.4" clip-path="url(#breeze-setup-line-9)">Enables autocompl [...]
+</text><text class="breeze-setup-r4" x="0" y="264" textLength="12.2" clip-path="url(#breeze-setup-line-10)">│</text><text class="breeze-setup-r5" x="24.4" y="264" textLength="390.4" clip-path="url(#breeze-setup-line-10)">config                          </text><text class="breeze-setup-r2" x="439.2" y="264" textLength="1000.4" clip-path="url(#breeze-setup-line- [...]
+</text><text class="breeze-setup-r4" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-setup-line-11)">│</text><text class="breeze-setup-r5" x="24.4" y="288.4" textLength="390.4" clip-path="url(#breeze-setup-line-11)">regenerate-command-images       </text><text class="breeze-setup-r2" x="439.2" y="288.4" textLength="1000.4" clip-path="url(#breeze-setup-line-11)">Regenerate breeze command images.       [...]
+</text><text class="breeze-setup-r4" x="0" y="312.8" textLength="12.2" clip-path="url(#breeze-setup-line-12)">│</text><text class="breeze-setup-r5" x="24.4" y="312.8" textLength="390.4" clip-path="url(#breeze-setup-line-12)">self-upgrade                    </text><text class="breeze-setup-r2" x="439.2" y="312.8" textLength="1000.4" clip-path="url(#breeze-setup-line-12)">Self upgrade&# [...]
+</text><text class="breeze-setup-r4" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-setup-line-13)">│</text><text class="breeze-setup-r5" x="24.4" y="337.2" textLength="390.4" clip-path="url(#breeze-setup-line-13)">version                         </text><text class="breeze-setup-r2" x="439.2" y="337.2" textLength="1000.4" clip-path="url(#breeze-setup-line [...]
+</text><text class="breeze-setup-r4" x="0" y="361.6" textLength="1464" clip-path="url(#breeze-setup-line-14)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-setup-r2" x="1464" y="361.6" textLength="12.2" clip-path="url(#breeze-setup-line-14)">
</text>
</g>
</g>
diff --git a/images/breeze/output_setup_regenerate-command-images.svg b/images/breeze/output_setup_regenerate-command-images.svg
index dc74dc7852..3ecbb459e5 100644
--- a/images/breeze/output_setup_regenerate-command-images.svg
+++ b/images/breeze/output_setup_regenerate-command-images.svg
@@ -35,8 +35,8 @@
.breeze-setup-regenerate-command-images-r1 { fill: #c5c8c6;font-weight: bold }
.breeze-setup-regenerate-command-images-r2 { fill: #c5c8c6 }
.breeze-setup-regenerate-command-images-r3 { fill: #d0b344;font-weight: bold }
-.breeze-setup-regenerate-command-images-r4 { fill: #68a0b3;font-weight: bold }
-.breeze-setup-regenerate-command-images-r5 { fill: #868887 }
+.breeze-setup-regenerate-command-images-r4 { fill: #868887 }
+.breeze-setup-regenerate-command-images-r5 { fill: #68a0b3;font-weight: bold }
.breeze-setup-regenerate-command-images-r6 { fill: #8d7b39 }
.breeze-setup-regenerate-command-images-r7 { fill: #98a84b;font-weight: bold }
</style>
@@ -142,34 +142,34 @@
<g class="breeze-setup-regenerate-command-images-matrix">
<text class="breeze-setup-regenerate-command-images-r2" x="1464" y="20" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-0)">
-</text><text class="breeze-setup-regenerate-command-images-r3" x="12.2" y="44.4" textLength="85.4" clip-path="url(#breeze-setup-regenerate-command-images-line-1)">Usage: </text><text class="breeze-setup-regenerate-command-images-r1" x="97.6" y="44.4" textLength="488" clip-path="url(#breeze-setup-regenerate-command-images-line-1)">breeze setup regenerate-command-images [</text><text class="breeze-setup-regenerate-command-images-r4" x="585.6" y="44.4" textLength="85.4" [...]
+</text><text class="breeze-setup-regenerate-command-images-r3" x="12.2" y="44.4" textLength="85.4" clip-path="url(#breeze-setup-regenerate-command-images-line-1)">Usage: </text><text class="breeze-setup-regenerate-command-images-r1" x="97.6" y="44.4" textLength="585.6" clip-path="url(#breeze-setup-regenerate-command-images-line-1)">breeze setup regenerate-command-images [OPTIONS]</text><text class="breeze-setup-regenerate-command-images-r2" x="1464" y="44.4" textLengt [...]
</text><text class="breeze-setup-regenerate-command-images-r2" x="1464" y="68.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-2)">
</text><text class="breeze-setup-regenerate-command-images-r2" x="12.2" y="93.2" textLength="402.6" clip-path="url(#breeze-setup-regenerate-command-images-line-3)">Regenerate breeze command images.</text><text class="breeze-setup-regenerate-command-images-r2" x="1464" y="93.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-3)">
</text><text class="breeze-setup-regenerate-command-images-r2" x="1464" y="117.6" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-4)">
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="142" textLength="24.4" clip-path="url(#breeze-setup-regenerate-command-images-line-5)">╭─</text><text class="breeze-setup-regenerate-command-images-r5" x="24.4" y="142" textLength="329.4" clip-path="url(#breeze-setup-regenerate-command-images-line-5)"> Image regeneration option </text><text class="breeze-setup-regenerate-command-images-r5" x="353.8" y="142" textLength="1085.8" clip-path="url(#bree [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="166.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-6)">│</text><text class="breeze-setup-regenerate-command-images-r4" x="24.4" y="166.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-6)">-</text><text class="breeze-setup-regenerate-command-images-r4" x="36.6" y="166.4" textLength="73.2" clip-path="url(#breeze-setup-regenerate-command-images-line-6)">- [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="190.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-7)">│</text><text class="breeze-setup-regenerate-command-images-r4" x="24.4" y="190.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-7)">-</text><text class="breeze-setup-regenerate-command-images-r4" x="36.6" y="190.8" textLength="97.6" clip-path="url(#breeze-setup-regenerate-command-images-line-7)">- [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="215.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-8)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="215.2" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-8)">(main | build-docs | ci:find-newer-dependencies | ci:fix-ownership | ci:free-space |       [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="239.6" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-9)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="239.6" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-9)">ci:get-workflow-info | ci:resource-check | ci:selective-check | ci | ci-image:build | ci-image:pull </text><te [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="264" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-10)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="264" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-10)">| ci-image:verify | ci-image | cleanup | compile-www-assets | exec | k8s:build-k8s-image |    [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-11)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="288.4" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-11)">k8s:configure-cluster | k8s:create-cluster | k8s:delete-cluster | k8s:deploy-airflow | k8s:k9s |    [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="312.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-12)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="312.8" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-12)">k8s:logs | k8s:run-complete-tests | k8s:setup-env | k8s:shell | k8s:status | k8s:tests |   &# [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-13)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="337.2" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-13)">k8s:upload-k8s-image | k8s | prod-image:build | prod-image:pull | prod-image:verify | prod-image |  [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-14)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="361.6" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-14)">release-management:generate-constraints | release-management:prepare-airflow-package |          &# [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="386" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-15)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="386" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-15)">release-management:prepare-provider-documentation | release-management:prepare-provider-packages |  </text><text class="breeze-setup-regenerat [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="410.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-16)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="410.4" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-16)">release-management:release-prod-images | release-management:verify-provider-packages |          &# [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="434.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-17)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="434.8" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-17)">release-management | setup:autocomplete | setup:config | setup:regenerate-command-images |      &# [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="459.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-18)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="459.2" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-18)">setup:self-upgrade | setup:version | setup | shell | start-airflow | static-checks | stop | & [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-19)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="483.6" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-19)">testing:docker-compose-tests | testing:helm-tests | testing:integration-tests | testing:tests |     </t [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="508" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-20)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="508" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-20)">testing)                           [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-21)">│</text><text class="breeze-setup-regenerate-command-images-r4" x="24.4" y="532.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-21)">-</text><text class="breeze-setup-regenerate-command-images-r4" x="36.6" y="532.4" textLength="73.2" clip-path="url(#breeze-setup-regenerate-command-images-line-21) [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="556.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-22)">│</text><text class="breeze-setup-regenerate-command-images-r2" x="219.6" y="556.8" textLength="170.8" clip-path="url(#breeze-setup-regenerate-command-images-line-22)">together with </text><text class="breeze-setup-regenerate-command-images-r4" x="390.4" y="556.8" textLength="12.2" clip-path="url(#breeze-setup-regenera [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="581.2" textLength="1464" clip-path="url(#breeze-setup-regenerate-command-images-line-23)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-setup-regenerate-command-images-r2" x="1464" y="581.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-23)">
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="605.6" textLength="24.4" clip-path="url(#breeze-setup-regenerate-command-images-line-24)">╭─</text><text class="breeze-setup-regenerate-command-images-r5" x="24.4" y="605.6" textLength="195.2" clip-path="url(#breeze-setup-regenerate-command-images-line-24)"> Common options </text><text class="breeze-setup-regenerate-command-images-r5" x="219.6" y="605.6" textLength="1220" clip-path="url(#breeze-setup-r [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="630" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-25)">│</text><text class="breeze-setup-regenerate-command-images-r4" x="24.4" y="630" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-25)">-</text><text class="breeze-setup-regenerate-command-images-r4" x="36.6" y="630" textLength="97.6" clip-path="url(#breeze-setup-regenerate-command-images-line-25)">-ver [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="654.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-26)">│</text><text class="breeze-setup-regenerate-command-images-r4" x="24.4" y="654.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-26)">-</text><text class="breeze-setup-regenerate-command-images-r4" x="36.6" y="654.4" textLength="48.8" clip-path="url(#breeze-setup-regenerate-command-images-line-26) [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="678.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-27)">│</text><text class="breeze-setup-regenerate-command-images-r4" x="24.4" y="678.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-27)">-</text><text class="breeze-setup-regenerate-command-images-r4" x="36.6" y="678.8" textLength="61" clip-path="url(#breeze-setup-regenerate-command-images-line-27)"> [...]
-</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="703.2" textLength="1464" clip-path="url(#breeze-setup-regenerate-command-images-line-28)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-setup-regenerate-command-images-r2" x="1464" y="703.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-28)">
+</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="142" textLength="24.4" clip-path="url(#breeze-setup-regenerate-command-images-line-5)">╭─</text><text class="breeze-setup-regenerate-command-images-r4" x="24.4" y="142" textLength="329.4" clip-path="url(#breeze-setup-regenerate-command-images-line-5)"> Image regeneration option </text><text class="breeze-setup-regenerate-command-images-r4" x="353.8" y="142" textLength="1085.8" clip-path="url(#bree [...]
+</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="166.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-6)">│</text><text class="breeze-setup-regenerate-command-images-r5" x="24.4" y="166.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-6)">-</text><text class="breeze-setup-regenerate-command-images-r5" x="36.6" y="166.4" textLength="73.2" clip-path="url(#breeze-setup-regenerate-command-images-line-6)">- [...]
+</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="190.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-7)">│</text><text class="breeze-setup-regenerate-command-images-r5" x="24.4" y="190.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-7)">-</text><text class="breeze-setup-regenerate-command-images-r5" x="36.6" y="190.8" textLength="97.6" clip-path="url(#breeze-setup-regenerate-command-images-line-7)">- [...]
+</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="215.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-8)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="215.2" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-8)">(main | build-docs | ci:find-newer-dependencies | ci:fix-ownership | ci:free-space |       [...]
+</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="239.6" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-9)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="239.6" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-9)">ci:get-workflow-info | ci:resource-check | ci:selective-check | ci | ci-image:build | ci-image:pull </text><te [...]
+</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="264" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-10)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="264" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-10)">| ci-image:verify | ci-image | cleanup | compile-www-assets | exec | k8s:build-k8s-image |    [...]
+</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-11)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="288.4" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-11)">k8s:configure-cluster | k8s:create-cluster | k8s:delete-cluster | k8s:deploy-airflow | k8s:k9s |    [...]
+</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="312.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-12)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="312.8" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-12)">k8s:logs | k8s:run-complete-tests | k8s:setup-env | k8s:shell | k8s:status | k8s:tests |   &# [...]
+</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-13)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="337.2" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-13)">k8s:upload-k8s-image | k8s | prod-image:build | prod-image:pull | prod-image:verify | prod-image |  [...]
+</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-14)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="361.6" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-14)">release-management:generate-constraints | release-management:generate-issue-content |           [...]
+</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="386" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-15)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="386" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-15)">release-management:prepare-airflow-package | release-management:prepare-provider-documentation |    </text><text class="breeze-setup [...]
+</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="410.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-16)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="410.4" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-16)">release-management:prepare-provider-packages | release-management:release-prod-images |          & [...]
+</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="434.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-17)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="434.8" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-17)">release-management:verify-provider-packages | release-management | setup:autocomplete | setup:config</text><text class="breeze-setup- [...]
+</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="459.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-18)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="459.2" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-18)">| setup:regenerate-command-images | setup:self-upgrade | setup:version | setup | shell |    & [...]
+</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-19)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="483.6" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-19)">start-airflow | static-checks | stop | testing:docker-compose-tests | testing:helm-tests |     [...]
+</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="508" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-20)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="219.6" y="508" textLength="1220" clip-path="url(#breeze-setup-regenerate-command-images-line-20)">testing:integration-tests | testing:tests | testing)                 [...]
+</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-21)">│</text><text class="breeze-setup-regenerate-command-images-r5" x="24.4" y="532.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-21)">-</text><text class="breeze-setup-regenerate-command-images-r5" x="36.6" y="532.4" textLength="73.2" clip-path="url(#breeze-setup-regenerate-command-images-line-21) [...]
+</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="556.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-22)">│</text><text class="breeze-setup-regenerate-command-images-r2" x="219.6" y="556.8" textLength="170.8" clip-path="url(#breeze-setup-regenerate-command-images-line-22)">together with </text><text class="breeze-setup-regenerate-command-images-r5" x="390.4" y="556.8" textLength="12.2" clip-path="url(#breeze-setup-regenera [...]
+</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="581.2" textLength="1464" clip-path="url(#breeze-setup-regenerate-command-images-line-23)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-setup-regenerate-command-images-r2" x="1464" y="581.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-23)">
+</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="605.6" textLength="24.4" clip-path="url(#breeze-setup-regenerate-command-images-line-24)">╭─</text><text class="breeze-setup-regenerate-command-images-r4" x="24.4" y="605.6" textLength="195.2" clip-path="url(#breeze-setup-regenerate-command-images-line-24)"> Common options </text><text class="breeze-setup-regenerate-command-images-r4" x="219.6" y="605.6" textLength="1220" clip-path="url(#breeze-setup-r [...]
+</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="630" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-25)">│</text><text class="breeze-setup-regenerate-command-images-r5" x="24.4" y="630" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-25)">-</text><text class="breeze-setup-regenerate-command-images-r5" x="36.6" y="630" textLength="97.6" clip-path="url(#breeze-setup-regenerate-command-images-line-25)">-ver [...]
+</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="654.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-26)">│</text><text class="breeze-setup-regenerate-command-images-r5" x="24.4" y="654.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-26)">-</text><text class="breeze-setup-regenerate-command-images-r5" x="36.6" y="654.4" textLength="48.8" clip-path="url(#breeze-setup-regenerate-command-images-line-26) [...]
+</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="678.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-27)">│</text><text class="breeze-setup-regenerate-command-images-r5" x="24.4" y="678.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-27)">-</text><text class="breeze-setup-regenerate-command-images-r5" x="36.6" y="678.8" textLength="61" clip-path="url(#breeze-setup-regenerate-command-images-line-27)"> [...]
+</text><text class="breeze-setup-regenerate-command-images-r4" x="0" y="703.2" textLength="1464" clip-path="url(#breeze-setup-regenerate-command-images-line-28)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-setup-regenerate-command-images-r2" x="1464" y="703.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-28)">
</text>
</g>
</g>
diff --git a/scripts/in_container/run_prepare_provider_documentation.sh b/scripts/in_container/run_prepare_provider_documentation.sh
index 9b1e11f111..37044bdbbe 100755
--- a/scripts/in_container/run_prepare_provider_documentation.sh
+++ b/scripts/in_container/run_prepare_provider_documentation.sh
@@ -107,11 +107,6 @@ function run_prepare_documentation() {
echo "${COLOR_RED}There were errors when preparing documentation. Exiting! ${COLOR_RESET}"
exit 1
else
- if [[ ${GENERATE_PROVIDERS_ISSUE=} == "true" || ${GENERATE_PROVIDERS_ISSUE} == "True" ]]; then
- echo
- python3 dev/provider_packages/prepare_provider_packages.py generate-issue-content "${prepared_documentation[@]}"
- echo
- fi
echo
echo "${COLOR_YELLOW}Please review the updated files, classify the changelog entries and commit the changes!${COLOR_RESET}"
echo
[airflow] 04/37: listener plugin example added (#27905)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 1456f11a99a1a3fe5f8513b4082a89e64e417866
Author: Bowrna <ma...@gmail.com>
AuthorDate: Sat Jan 21 02:58:27 2023 +0530
listener plugin example added (#27905)
(cherry picked from commit 100bb8d79a1e0c5fe6fca4b69c529b447cc992d1)
---
airflow/example_dags/plugins/event_listener.py | 156 ++++++++++++++++++++++++
airflow/example_dags/plugins/listener_plugin.py | 26 ++++
docs/apache-airflow/howto/index.rst | 1 +
docs/apache-airflow/howto/listener-plugin.rst | 95 +++++++++++++++
docs/spelling_wordlist.txt | 1 +
5 files changed, 279 insertions(+)
diff --git a/airflow/example_dags/plugins/event_listener.py b/airflow/example_dags/plugins/event_listener.py
new file mode 100644
index 0000000000..2e2d01800b
--- /dev/null
+++ b/airflow/example_dags/plugins/event_listener.py
@@ -0,0 +1,156 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+from airflow.listeners import hookimpl
+
+if TYPE_CHECKING:
+ from airflow.models.dagrun import DagRun
+ from airflow.models.taskinstance import TaskInstance
+ from airflow.utils.state import TaskInstanceState
+
+
+# [START howto_listen_ti_running_task]
+@hookimpl
+def on_task_instance_running(previous_state: TaskInstanceState, task_instance: TaskInstance, session):
+ """
+ This method is called when task state changes to RUNNING.
+ Through callback, parameters like previous_task_state, task_instance object can be accessed.
+ This will give more information about current task_instance that is running its dag_run,
+ task and dag information.
+ """
+ print("Task instance is in running state")
+ print(" Previous state of the Task instance:", previous_state)
+
+ state: TaskInstanceState = task_instance.state
+ name: str = task_instance.task_id
+ start_date = task_instance.start_date
+
+ dagrun = task_instance.dag_run
+ dagrun_status = dagrun.state
+
+ task = task_instance.task
+
+ dag = task.dag
+ dag_name = None
+ if dag:
+ dag_name = dag.dag_id
+ print(f"Current task name:{name} state:{state} start_date:{start_date}")
+ print(f"Dag name:{dag_name} and current dag run status:{dagrun_status}")
+
+
+# [END howto_listen_ti_running_task]
+
+# [START howto_listen_ti_success_task]
+@hookimpl
+def on_task_instance_success(previous_state: TaskInstanceState, task_instance: TaskInstance, session):
+ """
+ This method is called when task state changes to SUCCESS.
+ Through callback, parameters like previous_task_state, task_instance object can be accessed.
+ This will give more information about current task_instance that has succeeded its
+ dag_run, task and dag information.
+ """
+ print("Task instance in success state")
+ print(" Previous state of the Task instance:", previous_state)
+
+ dag_id = task_instance.dag_id
+ hostname = task_instance.hostname
+ operator = task_instance.operator
+
+ dagrun = task_instance.dag_run
+ queued_at = dagrun.queued_at
+ print(f"Dag name:{dag_id} queued_at:{queued_at}")
+ print(f"Task hostname:{hostname} operator:{operator}")
+
+
+# [END howto_listen_ti_success_task]
+
+# [START howto_listen_ti_failure_task]
+@hookimpl
+def on_task_instance_failed(previous_state: TaskInstanceState, task_instance: TaskInstance, session):
+ """
+ This method is called when task state changes to FAILED.
+ Through callback, parameters like previous_task_state, task_instance object can be accessed.
+ This will give more information about current task_instance that has failed its dag_run,
+ task and dag information.
+ """
+ print("Task instance in failure state")
+
+ start_date = task_instance.start_date
+ end_date = task_instance.end_date
+ duration = task_instance.duration
+
+ dagrun = task_instance.dag_run
+
+ task = task_instance.task
+
+ dag = task_instance.task.dag
+
+ print(f"Task start:{start_date} end:{end_date} duration:{duration}")
+ print(f"Task:{task} dag:{dag} dagrun:{dagrun}")
+
+
+# [END howto_listen_ti_failure_task]
+
+# [START howto_listen_dagrun_success_task]
+@hookimpl
+def on_dag_run_success(dag_run: DagRun, message: str):
+ """
+ This method is called when dag run state changes to SUCCESS.
+ """
+ print("Dag run in success state")
+ start_date = dag_run.start_date
+ end_date = dag_run.end_date
+
+ print(f"Dag run start:{start_date} end:{end_date}")
+
+
+# [END howto_listen_dagrun_success_task]
+
+# [START howto_listen_dagrun_failure_task]
+@hookimpl
+def on_dag_run_failed(dag_run: DagRun, message: str):
+ """
+ This method is called when dag run state changes to FAILED.
+ """
+ print("Dag run in failure state")
+ dag_id = dag_run.dag_id
+ run_id = dag_run.run_id
+ external_trigger = dag_run.external_trigger
+
+ print(f"Dag information:{dag_id} Run id: {run_id} external trigger: {external_trigger}")
+
+
+# [END howto_listen_dagrun_failure_task]
+
+# [START howto_listen_dagrun_running_task]
+@hookimpl
+def on_dag_run_running(dag_run: DagRun, message: str):
+ """
+ This method is called when dag run state changes to RUNNING.
+ """
+ print("Dag run in running state")
+ queued_at = dag_run.queued_at
+ dag_hash_info = dag_run.dag_hash
+
+ print(f"Dag information Queued at: {queued_at} hash info: {dag_hash_info}")
+
+
+# [END howto_listen_dagrun_running_task]
diff --git a/airflow/example_dags/plugins/listener_plugin.py b/airflow/example_dags/plugins/listener_plugin.py
new file mode 100644
index 0000000000..a365d57e3f
--- /dev/null
+++ b/airflow/example_dags/plugins/listener_plugin.py
@@ -0,0 +1,26 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from airflow.example_dags.plugins import event_listener
+from airflow.plugins_manager import AirflowPlugin
+
+
+class MetadataCollectionPlugin(AirflowPlugin):
+ name = "MetadataCollectionPlugin"
+ listeners = [event_listener]
diff --git a/docs/apache-airflow/howto/index.rst b/docs/apache-airflow/howto/index.rst
index 5d6b54fd77..7a170828d5 100644
--- a/docs/apache-airflow/howto/index.rst
+++ b/docs/apache-airflow/howto/index.rst
@@ -37,6 +37,7 @@ configuring an Airflow environment.
operator/index
timetable
custom-view-plugin
+ listener-plugin
customize-ui
custom-operator
create-custom-decorator
diff --git a/docs/apache-airflow/howto/listener-plugin.rst b/docs/apache-airflow/howto/listener-plugin.rst
new file mode 100644
index 0000000000..7b46a9de8a
--- /dev/null
+++ b/docs/apache-airflow/howto/listener-plugin.rst
@@ -0,0 +1,95 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+Listener Plugin of Airflow
+==========================
+
+Airflow has feature that allows to add listener for monitoring and tracking
+the task state using Plugins.
+
+This is a simple example listener plugin of Airflow that helps to track the task
+state and collect useful metadata information about the task, dag run and dag.
+
+This is an example plugin for Airflow that allows to create listener plugin of Airflow.
+This plugin works by using SQLAlchemy's event mechanism. It watches
+the task instance state change in the table level and triggers event.
+This will be notified for all the tasks across all the DAGs.
+
+In this plugin, an object reference is derived from the base class
+``airflow.plugins_manager.AirflowPlugin``.
+
+Listener plugin uses pluggy app under the hood. Pluggy is an app built for plugin
+management and hook calling for Pytest. Pluggy enables function hooking so it allows
+building "pluggable" systems with your own customization over that hooking.
+
+Using this plugin, following events can be listened:
+ * task instance is in running state.
+ * task instance is in success state.
+ * task instance is in failure state.
+ * dag run is in running state.
+ * dag run is in success state.
+ * dag run is in failure state.
+ * on start before event like airflow job, scheduler or backfilljob
+ * before stop for event like airflow job, scheduler or backfilljob
+
+Listener Registration
+---------------------
+
+A listener plugin with object reference to listener object is registered
+as part of airflow plugin. The following is a
+skeleton for us to implement a new listener:
+
+.. code-block:: python
+
+ from airflow.plugins_manager import AirflowPlugin
+
+ # This is the listener file created where custom code to monitor is added over hookimpl
+ import listener
+
+
+ class MetadataCollectionPlugin(AirflowPlugin):
+ name = "MetadataCollectionPlugin"
+ listeners = [listener]
+
+
+Next, we can check code added into ``listener`` and see implementation
+methods for each of those listeners. After the implementation, the listener part
+gets executed during all the task execution across all the DAGs
+
+For reference, here's the plugin code within ``listener.py`` class that shows list of tables in the database:
+
+This example listens when the task instance is in running state
+
+.. exampleinclude:: ../../../airflow/example_dags/plugins/event_listener.py
+ :language: python
+ :start-after: [START howto_listen_ti_running_task]
+ :end-before: [END howto_listen_ti_running_task]
+
+Similarly, code to listen after task_instance success and failure can be implemented.
+
+This example listens when the dag run is change to failed state
+
+.. exampleinclude:: ../../../airflow/example_dags/plugins/event_listener.py
+ :language: python
+ :start-after: [START howto_listen_dagrun_failure_task]
+ :end-before: [END howto_listen_dagrun_failure_task]
+
+Similarly, code to listen after dag_run success and during running state can be implemented.
+
+The listener plugin files required to add the listener implementation is added as part of the
+Airflow plugin into ``$AIRFLOW_HOME/plugins/`` folder and loaded during Airflow startup.
diff --git a/docs/spelling_wordlist.txt b/docs/spelling_wordlist.txt
index 734723894a..5a62b70939 100644
--- a/docs/spelling_wordlist.txt
+++ b/docs/spelling_wordlist.txt
@@ -123,6 +123,7 @@ backfill
backfillable
backfilled
backfilling
+backfilljob
BackfillJobTest
Backfills
backfills
[airflow] 19/37: Be more selective when adopting pods with KubernetesExecutor (#28899)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 5a8ddaedd80638f9066f5af30c5695ba43ee9214
Author: Jed Cunningham <66...@users.noreply.github.com>
AuthorDate: Wed Jan 18 14:05:50 2023 -0600
Be more selective when adopting pods with KubernetesExecutor (#28899)
* Be more selective when adopting pods with KubernetesExecutor
When trying to adopt "resettable" TIs from SchedulerJob, we should not
list out all the pods to compare against, only those that didn't
succeed. This means we will get any pods that are still starting,
running, or failed (meaning the TI wasn't moved to a terminal state
there, and will be in out "adoptable" list).
This avoids the scenario where a dead scheduler has both a completed,
successful worker, and a still running worker, causing log lines
like these about the successful one:
ERROR - attempting to adopt taskinstance which was not specified by
database: TaskInstanceKey(...)
This also makes sure we only find pods with the
`kubernetes_executor=True` label for extra safety.
Closes #28071
* Also ignore done pods
(cherry picked from commit f64ac5978fb3dfa9e40a0e5190ef88e9f9615824)
---
airflow/executors/kubernetes_executor.py | 32 +++++++++++++++++++++--------
tests/executors/test_kubernetes_executor.py | 20 ++++++++++++++----
2 files changed, 39 insertions(+), 13 deletions(-)
diff --git a/airflow/executors/kubernetes_executor.py b/airflow/executors/kubernetes_executor.py
index 34204539fa..412caea5c5 100644
--- a/airflow/executors/kubernetes_executor.py
+++ b/airflow/executors/kubernetes_executor.py
@@ -468,13 +468,16 @@ class KubernetesExecutor(BaseExecutor):
@provide_session
def clear_not_launched_queued_tasks(self, session=None) -> None:
"""
- Tasks can end up in a "Queued" state through either the executor being
- abruptly shut down (leaving a non-empty task_queue on this executor)
- or when a rescheduled/deferred operator comes back up for execution
- (with the same try_number) before the pod of its previous incarnation
- has been fully removed (we think).
+ Clear tasks that were not yet launched, but were previously queued.
- This method checks each of those tasks to see if the corresponding pod
+ Tasks can end up in a "Queued" state through when a rescheduled/deferred
+ operator comes back up for execution (with the same try_number) before the
+ pod of its previous incarnation has been fully removed (we think).
+
+ It's also possible when an executor abruptly shuts down (leaving a non-empty
+ task_queue on that executor), but that scenario is handled via normal adoption.
+
+ This method checks each of our queued tasks to see if the corresponding pod
is around, and if not, and there's no matching entry in our own
task_queue, marks it for re-execution.
"""
@@ -747,9 +750,20 @@ class KubernetesExecutor(BaseExecutor):
kube_client: client.CoreV1Api = self.kube_client
for scheduler_job_id in scheduler_job_ids:
scheduler_job_id = pod_generator.make_safe_label_value(str(scheduler_job_id))
- kwargs = {"label_selector": f"airflow-worker={scheduler_job_id}"}
- pod_list = kube_client.list_namespaced_pod(namespace=self.kube_config.kube_namespace, **kwargs)
- for pod in pod_list.items:
+ # We will look for any pods owned by the no-longer-running scheduler,
+ # but will exclude only successful pods, as those TIs will have a terminal state
+ # and not be up for adoption!
+ # Those workers that failed, however, are okay to adopt here as their TI will
+ # still be in queued.
+ query_kwargs = {
+ "field_selector": "status.phase!=Succeeded",
+ "label_selector": (
+ "kubernetes_executor=True,"
+ f"airflow-worker={scheduler_job_id},{POD_EXECUTOR_DONE_KEY}!=True"
+ ),
+ }
+ pod_list = self._list_pods(query_kwargs)
+ for pod in pod_list:
self.adopt_launched_task(kube_client, pod, pod_ids)
self._adopt_completed_pods(kube_client)
tis_to_flush.extend(pod_ids.values())
diff --git a/tests/executors/test_kubernetes_executor.py b/tests/executors/test_kubernetes_executor.py
index d24f3e8fd9..a56b6c9639 100644
--- a/tests/executors/test_kubernetes_executor.py
+++ b/tests/executors/test_kubernetes_executor.py
@@ -622,7 +622,9 @@ class TestKubernetesExecutor:
# First adoption
reset_tis = executor.try_adopt_task_instances([mock_ti])
mock_kube_client.list_namespaced_pod.assert_called_once_with(
- namespace="default", label_selector="airflow-worker=1"
+ namespace="default",
+ field_selector="status.phase!=Succeeded",
+ label_selector="kubernetes_executor=True,airflow-worker=1,airflow_executor_done!=True",
)
mock_adopt_launched_task.assert_called_once_with(mock_kube_client, pod, {ti_key: mock_ti})
mock_adopt_completed_pods.assert_called_once()
@@ -640,7 +642,9 @@ class TestKubernetesExecutor:
reset_tis = executor.try_adopt_task_instances([mock_ti])
mock_kube_client.list_namespaced_pod.assert_called_once_with(
- namespace="default", label_selector="airflow-worker=10"
+ namespace="default",
+ field_selector="status.phase!=Succeeded",
+ label_selector="kubernetes_executor=True,airflow-worker=10,airflow_executor_done!=True",
)
mock_adopt_launched_task.assert_called_once() # Won't check args this time around as they get mutated
mock_adopt_completed_pods.assert_called_once()
@@ -663,8 +667,16 @@ class TestKubernetesExecutor:
assert mock_kube_client.list_namespaced_pod.call_count == 2
mock_kube_client.list_namespaced_pod.assert_has_calls(
[
- mock.call(namespace="default", label_selector="airflow-worker=10"),
- mock.call(namespace="default", label_selector="airflow-worker=40"),
+ mock.call(
+ namespace="default",
+ field_selector="status.phase!=Succeeded",
+ label_selector="kubernetes_executor=True,airflow-worker=10,airflow_executor_done!=True",
+ ),
+ mock.call(
+ namespace="default",
+ field_selector="status.phase!=Succeeded",
+ label_selector="kubernetes_executor=True,airflow-worker=40,airflow_executor_done!=True",
+ ),
],
any_order=True,
)
[airflow] 12/37: Annotate KubernetesExecutor pods that we don't delete (#28844)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 9ad5779a54e3a2a25449336e1e1f91b0048ef17c
Author: Jed Cunningham <66...@users.noreply.github.com>
AuthorDate: Wed Jan 11 15:15:21 2023 -0600
Annotate KubernetesExecutor pods that we don't delete (#28844)
We weren't keeping track of which pods we'd finished with yet, so if you
had `[kubernetes_executor] delete_worker_pods` false, your
KubeExecutor would adopt every single remaining pod when starting up.
Every time.
We now annotate them with `airflow_executor_done` when processing a pods event
from the watcher, so we can ignore the pod when doing adoption.
(cherry picked from commit 72da8bff12e3133045b61936952599a84d3f53a2)
---
airflow/executors/kubernetes_executor.py | 22 +++++++++++++++++++++-
tests/executors/test_kubernetes_executor.py | 18 ++++++++++++------
2 files changed, 33 insertions(+), 7 deletions(-)
diff --git a/airflow/executors/kubernetes_executor.py b/airflow/executors/kubernetes_executor.py
index 7dcba12968..8098486ded 100644
--- a/airflow/executors/kubernetes_executor.py
+++ b/airflow/executors/kubernetes_executor.py
@@ -52,6 +52,8 @@ from airflow.utils.log.logging_mixin import LoggingMixin
from airflow.utils.session import provide_session
from airflow.utils.state import State
+POD_EXECUTOR_DONE_KEY = "airflow_executor_done"
+
# TaskInstance key, command, configuration, pod_template_file
KubernetesJobType = Tuple[TaskInstanceKey, CommandType, Any, Optional[str]]
@@ -360,6 +362,18 @@ class AirflowKubernetesScheduler(LoggingMixin):
if e.status != 404:
raise
+ def patch_pod_executor_done(self, *, pod_id: str, namespace: str):
+ """Add a "done" annotation to ensure we don't continually adopt pods"""
+ self.log.debug("Patching pod %s in namespace %s to mark it as done", pod_id, namespace)
+ try:
+ self.kube_client.patch_namespaced_pod(
+ name=pod_id,
+ namespace=namespace,
+ body={"metadata": {"labels": {POD_EXECUTOR_DONE_KEY: "True"}}},
+ )
+ except ApiException as e:
+ self.log.info("Failed to patch pod %s with done annotation. Reason: %s", pod_id, e)
+
def sync(self) -> None:
"""
The sync function checks the status of all currently running kubernetes jobs.
@@ -710,6 +724,9 @@ class KubernetesExecutor(BaseExecutor):
if state != State.FAILED or self.kube_config.delete_worker_pods_on_failure:
self.kube_scheduler.delete_pod(pod_id, namespace)
self.log.info("Deleted pod: %s in namespace %s", str(key), str(namespace))
+ else:
+ self.kube_scheduler.patch_pod_executor_done(pod_id=pod_id, namespace=namespace)
+ self.log.info("Patched pod %s in namespace %s to mark it as done", str(key), str(namespace))
try:
self.running.remove(key)
except KeyError:
@@ -776,7 +793,10 @@ class KubernetesExecutor(BaseExecutor):
new_worker_id_label = pod_generator.make_safe_label_value(self.scheduler_job_id)
kwargs = {
"field_selector": "status.phase=Succeeded",
- "label_selector": f"kubernetes_executor=True,airflow-worker!={new_worker_id_label}",
+ "label_selector": (
+ "kubernetes_executor=True,"
+ f"airflow-worker!={new_worker_id_label},{POD_EXECUTOR_DONE_KEY}!=True"
+ ),
}
pod_list = kube_client.list_namespaced_pod(namespace=self.kube_config.kube_namespace, **kwargs)
for pod in pod_list.items:
diff --git a/tests/executors/test_kubernetes_executor.py b/tests/executors/test_kubernetes_executor.py
index d1210765c0..9f5304ba4a 100644
--- a/tests/executors/test_kubernetes_executor.py
+++ b/tests/executors/test_kubernetes_executor.py
@@ -549,10 +549,12 @@ class TestKubernetesExecutor:
@mock.patch("airflow.executors.kubernetes_executor.KubernetesJobWatcher")
@mock.patch("airflow.executors.kubernetes_executor.get_kube_client")
- @mock.patch("airflow.executors.kubernetes_executor.AirflowKubernetesScheduler.delete_pod")
+ @mock.patch("airflow.executors.kubernetes_executor.AirflowKubernetesScheduler")
def test_change_state_skip_pod_deletion(
- self, mock_delete_pod, mock_get_kube_client, mock_kubernetes_job_watcher
+ self, mock_kubescheduler, mock_get_kube_client, mock_kubernetes_job_watcher
):
+ mock_delete_pod = mock_kubescheduler.return_value.delete_pod
+ mock_patch_pod = mock_kubescheduler.return_value.patch_pod_executor_done
executor = self.kubernetes_executor
executor.kube_config.delete_worker_pods = False
executor.kube_config.delete_worker_pods_on_failure = False
@@ -560,18 +562,21 @@ class TestKubernetesExecutor:
executor.start()
try:
key = ("dag_id", "task_id", "run_id", "try_number2")
- executor._change_state(key, State.SUCCESS, "pod_id", "default")
+ executor._change_state(key, State.SUCCESS, "pod_id", "test-namespace")
assert executor.event_buffer[key][0] == State.SUCCESS
mock_delete_pod.assert_not_called()
+ mock_patch_pod.assert_called_once_with(pod_id="pod_id", namespace="test-namespace")
finally:
executor.end()
@mock.patch("airflow.executors.kubernetes_executor.KubernetesJobWatcher")
@mock.patch("airflow.executors.kubernetes_executor.get_kube_client")
- @mock.patch("airflow.executors.kubernetes_executor.AirflowKubernetesScheduler.delete_pod")
+ @mock.patch("airflow.executors.kubernetes_executor.AirflowKubernetesScheduler")
def test_change_state_failed_pod_deletion(
- self, mock_delete_pod, mock_get_kube_client, mock_kubernetes_job_watcher
+ self, mock_kubescheduler, mock_get_kube_client, mock_kubernetes_job_watcher
):
+ mock_delete_pod = mock_kubescheduler.return_value.delete_pod
+ mock_patch_pod = mock_kubescheduler.return_value.patch_pod_executor_done
executor = self.kubernetes_executor
executor.kube_config.delete_worker_pods_on_failure = True
@@ -581,6 +586,7 @@ class TestKubernetesExecutor:
executor._change_state(key, State.FAILED, "pod_id", "test-namespace")
assert executor.event_buffer[key][0] == State.FAILED
mock_delete_pod.assert_called_once_with("pod_id", "test-namespace")
+ mock_patch_pod.assert_not_called()
finally:
executor.end()
@@ -743,7 +749,7 @@ class TestKubernetesExecutor:
mock_kube_client.list_namespaced_pod.assert_called_once_with(
namespace="somens",
field_selector="status.phase=Succeeded",
- label_selector="kubernetes_executor=True,airflow-worker!=modified",
+ label_selector="kubernetes_executor=True,airflow-worker!=modified,airflow_executor_done!=True",
)
assert len(pod_names) == mock_kube_client.patch_namespaced_pod.call_count
mock_kube_client.patch_namespaced_pod.assert_has_calls(
[airflow] 18/37: Don't get ES log template from airflow local settings unless necessary (#28882)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 0cb731c3944b46e670e9edbb74d333ac275da70b
Author: Daniel Standish <15...@users.noreply.github.com>
AuthorDate: Fri Jan 13 13:40:11 2023 -0800
Don't get ES log template from airflow local settings unless necessary (#28882)
This creates an unnecessary warning.
(cherry picked from commit c5ee4b8a3a2266ef98b379ee28ed68ff1b59ac5f)
---
airflow/config_templates/airflow_local_settings.py | 2 --
1 file changed, 2 deletions(-)
diff --git a/airflow/config_templates/airflow_local_settings.py b/airflow/config_templates/airflow_local_settings.py
index 01edea7520..b606a00018 100644
--- a/airflow/config_templates/airflow_local_settings.py
+++ b/airflow/config_templates/airflow_local_settings.py
@@ -287,7 +287,6 @@ if REMOTE_LOGGING:
}
DEFAULT_LOGGING_CONFIG["handlers"].update(OSS_REMOTE_HANDLERS)
elif ELASTICSEARCH_HOST:
- ELASTICSEARCH_LOG_ID_TEMPLATE: str = conf.get_mandatory_value("elasticsearch", "LOG_ID_TEMPLATE")
ELASTICSEARCH_END_OF_LOG_MARK: str = conf.get_mandatory_value("elasticsearch", "END_OF_LOG_MARK")
ELASTICSEARCH_FRONTEND: str = conf.get_mandatory_value("elasticsearch", "frontend")
ELASTICSEARCH_WRITE_STDOUT: bool = conf.getboolean("elasticsearch", "WRITE_STDOUT")
@@ -301,7 +300,6 @@ if REMOTE_LOGGING:
"class": "airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler",
"formatter": "airflow",
"base_log_folder": str(os.path.expanduser(BASE_LOG_FOLDER)),
- "log_id_template": ELASTICSEARCH_LOG_ID_TEMPLATE,
"filename_template": FILENAME_TEMPLATE,
"end_of_log_mark": ELASTICSEARCH_END_OF_LOG_MARK,
"host": ELASTICSEARCH_HOST,
[airflow] 33/37: Remove erroneous TODO comment (#29015)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 8fe9d7e4779ac88fd324367e724e4043d9cdcfd6
Author: Ephraim Anierobi <sp...@gmail.com>
AuthorDate: Wed Jan 18 18:02:04 2023 +0100
Remove erroneous TODO comment (#29015)
The issue that this comment was referring to was closed as won't fix.
This PR removes the comment.
Issue: https://issues.apache.org/jira/browse/AIRFLOW-1455
(cherry picked from commit e3a6bd817f3c32ec5e780449090ad96782267d80)
---
airflow/config_templates/airflow_local_settings.py | 4 ----
1 file changed, 4 deletions(-)
diff --git a/airflow/config_templates/airflow_local_settings.py b/airflow/config_templates/airflow_local_settings.py
index b606a00018..39d809e516 100644
--- a/airflow/config_templates/airflow_local_settings.py
+++ b/airflow/config_templates/airflow_local_settings.py
@@ -26,10 +26,6 @@ from urllib.parse import urlsplit
from airflow.configuration import conf
from airflow.exceptions import AirflowException
-# TODO: Logging format and level should be configured
-# in this file instead of from airflow.cfg. Currently
-# there are other log format and level configurations in
-# settings.py and cli.py. Please see AIRFLOW-1455.
LOG_LEVEL: str = conf.get_mandatory_value("logging", "LOGGING_LEVEL").upper()
[airflow] 01/37: Update doc for API clients release policy (#28521)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 7fe2ceaad6acf61bcf75c373af852fda66367a50
Author: Pierre Jeambrun <pi...@gmail.com>
AuthorDate: Mon Jan 9 18:56:10 2023 +0100
Update doc for API clients release policy (#28521)
* Update doc for API clients release policy
* Update README.md
Co-authored-by: Xiaodong DENG <xd...@gmail.com>
* Update airflow release process
* Clarify patch release for clients, add process link
* Update dev/README_RELEASE_AIRFLOW.md
Co-authored-by: Ephraim Anierobi <sp...@gmail.com>
Co-authored-by: Xiaodong DENG <xd...@gmail.com>
Co-authored-by: Ephraim Anierobi <sp...@gmail.com>
(cherry picked from commit 654242b54638d7a682246d9dbea394bbb59d8923)
---
README.md | 8 ++++--
dev/README_RELEASE_AIRFLOW.md | 64 ++++++++++++++++++++++++++++++++++++++-----
2 files changed, 63 insertions(+), 9 deletions(-)
diff --git a/README.md b/README.md
index 3b5e704e5f..d6fbad7e69 100644
--- a/README.md
+++ b/README.md
@@ -259,8 +259,12 @@ packages:
Chart to depend on minimal Airflow version.
* **Airflow API clients**: SemVer MAJOR and MINOR versions follow MAJOR and MINOR versions of Airflow.
The first MAJOR or MINOR X.Y.0 release of Airflow should always be followed by X.Y.0 release of
- all clients. The clients then can release their own PATCH releases with bugfixes,
- independently of Airflow PATCH releases.
+ all clients. An airflow PATCH X.Y.Z release can be followed by a PATCH release of API clients, only
+ if this PATCH is relevant to the clients.
+ The clients then can release their own PATCH releases with bugfixes, independently of Airflow PATCH releases.
+ As a consequence, each API client will have its own PATCH version that may or may not be in sync with the Airflow
+ PATCH version. For a specific MAJOR/MINOR Airflow version, users should favor the latest PATCH version of clients
+ independently of their Airflow PATCH version.
## Version Life Cycle
diff --git a/dev/README_RELEASE_AIRFLOW.md b/dev/README_RELEASE_AIRFLOW.md
index 1fd37283b7..0e905cf706 100644
--- a/dev/README_RELEASE_AIRFLOW.md
+++ b/dev/README_RELEASE_AIRFLOW.md
@@ -29,6 +29,7 @@
- [Prepare new release branches and cache - optional when first minor version is released](#prepare-new-release-branches-and-cache---optional-when-first-minor-version-is-released)
- [Prepare PyPI convenience "snapshot" packages](#prepare-pypi-convenience-snapshot-packages)
- [Prepare production Docker Image RC](#prepare-production-docker-image-rc)
+ - [Prepare API clients RC packages](#prepare-api-clients-rc-packages)
- [Prepare issue for testing status of rc](#prepare-issue-for-testing-status-of-rc)
- [Prepare Vote email on the Apache Airflow release candidate](#prepare-vote-email-on-the-apache-airflow-release-candidate)
- [Verify the release candidate by PMCs](#verify-the-release-candidate-by-pmcs)
@@ -41,6 +42,7 @@
- [Summarize the voting for the Apache Airflow release](#summarize-the-voting-for-the-apache-airflow-release)
- [Publish release to SVN](#publish-release-to-svn)
- [Prepare PyPI "release" packages](#prepare-pypi-release-packages)
+ - [Manually release API clients](#manually-release-api-clients)
- [Manually prepare production Docker Image](#manually-prepare-production-docker-image)
- [Verify production images](#verify-production-images)
- [Publish documentation](#publish-documentation)
@@ -221,8 +223,8 @@ The Release Candidate artifacts we vote upon should be the exact ones we vote ag
git reset --hard origin/v${VERSION_BRANCH}-test
```
-- Set your version in `setup.py` (without the RC tag)
-- Add supported Airflow version to `./scripts/ci/pre_commit/pre_commit_supported_versions.py` and let pre-commit do the job
+- Set your version in `setup.py` and `airflow/api_connexion/openapi/v1.yaml` (without the RC tag).
+- Add supported Airflow version to `./scripts/ci/pre_commit/pre_commit_supported_versions.py` and let pre-commit do the job.
- Replace the version in `README.md` and verify that installation instructions work fine.
- Build the release notes:
@@ -522,6 +524,43 @@ When you trigger it you need to pass:
The manual building is described in [MANUALLY_BUILDING_IMAGES.md](MANUALLY_BUILDING_IMAGES.md).
+
+## Prepare API clients RC packages
+
+### API Clients versioning policy
+
+For major/minor version release, always release new versions of the API clients.
+
+- [Python client](https://github.com/apache/airflow-client-python)
+- [Go client](https://github.com/apache/airflow-client-go)
+
+For patch version release, you can also release patch versions of clients **only** if the patch is relevant to the clients.
+A patch is considered relevant to the clients if it updates the [openapi specification](https://github.com/apache/airflow/blob/main/airflow/api_connexion/openapi/v1.yaml).
+There are other external reasons for which we might want to release a patch version for clients only, but they are not
+tied to an airflow release and therefore out of scope.
+
+> The patch version of each API client is not necessarily in sync with the patch that you are releasing. You need to check for
+> each client what is the next patch version to be released.
+
+### Prepare the vote artifacts
+
+If API clients are to be released in this airflow version:
+
+- Set environment variables (useful for the rest of the process)
+
+ ```shell script
+ # Set Version
+ export GO_API_CLIENT_VERSION=2.1.3
+ export PYTHON_API_CLIENT_VERSION=2.1.1
+ ```
+
+- Follow the specific release process of each API client to generate the artifacts and push to PyPI a
+ release candidate client package:
+
+ - [Python client](https://github.com/apache/airflow-client-python#release-process)
+ - [Go client](https://github.com/apache/airflow-client-go#release-process)
+
+
## Prepare issue for testing status of rc
For now this part works for bugfix releases only, for major/minor ones we will experiment and
@@ -586,6 +625,11 @@ https://dist.apache.org/repos/dist/dev/airflow/$VERSION/
Public keys are available at:
https://dist.apache.org/repos/dist/release/airflow/KEYS
+TODO:REMOVE PARAGRAPH IF NOT RELEVANT
+New API clients were generated:
+- Python ${PYTHON_API_CLIENT_VERSION}
+- GO ${GO_API_CLIENT_VERSION}
+
Please vote accordingly:
[ ] +1 approve
@@ -964,6 +1008,12 @@ At this point we release an official package:
git push origin tag ${VERSION}
```
+## Manually release API clients
+
+If API clients are part of the release, publish new versions of the clients to PyPI without the rc suffix
+based on the vote artifacts.
+
+
## Manually prepare production Docker Image
Building the image is triggered by running the
@@ -1144,12 +1194,12 @@ EOF
This includes:
-- Modify `./scripts/ci/pre_commit/pre_commit_supported_versions.py` and let pre-commit do the job
-- For major/minor release, Update version in `setup.py` and `docs/docker-stack/` to the next likely minor version release.
+- Modify `./scripts/ci/pre_commit/pre_commit_supported_versions.py` and let pre-commit do the job.
+- For major/minor release, update version in `setup.py`, `docs/docker-stack/` and `airflow/api_connexion/openapi/v1.yaml` to the next likely minor version release.
- Update the `REVISION_HEADS_MAP` at airflow/utils/db.py to include the revision head of the release even if there are no migrations.
-- Sync `RELEASE_NOTES.rst` (including deleting relevant `newsfragments`) and `README.md` changes
-- Updating `airflow_bug_report.yml` issue template in `.github/ISSUE_TEMPLATE/` with the new version
-- Updating `Dockerfile` with the new version
+- Sync `RELEASE_NOTES.rst` (including deleting relevant `newsfragments`) and `README.md` changes.
+- Updating `airflow_bug_report.yml` issue template in `.github/ISSUE_TEMPLATE/` with the new version.
+- Updating `Dockerfile` with the new version.
## Update default Airflow version in the helm chart
[airflow] 31/37: Write action log to DB when DAG run is trigged via API (#28998)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 2a8e414364304b8e57eded3c02ea565250503003
Author: Andrey Komrakov <ak...@gmail.com>
AuthorDate: Fri Feb 3 11:52:58 2023 +0300
Write action log to DB when DAG run is trigged via API (#28998)
(cherry picked from commit edc2e0b118a77c143b1a5d1eb82f1137148af633)
---
airflow/api_connexion/endpoints/dag_run_endpoint.py | 10 ++++++++++
tests/api_connexion/endpoints/test_dag_run_endpoint.py | 4 +++-
2 files changed, 13 insertions(+), 1 deletion(-)
diff --git a/airflow/api_connexion/endpoints/dag_run_endpoint.py b/airflow/api_connexion/endpoints/dag_run_endpoint.py
index 2b3ee16b2f..7018fcb8a8 100644
--- a/airflow/api_connexion/endpoints/dag_run_endpoint.py
+++ b/airflow/api_connexion/endpoints/dag_run_endpoint.py
@@ -56,9 +56,13 @@ from airflow.api_connexion.types import APIResponse
from airflow.models import DagModel, DagRun
from airflow.security import permissions
from airflow.utils.airflow_flask_app import get_airflow_app
+from airflow.utils.log.action_logger import action_event_from_permission
from airflow.utils.session import NEW_SESSION, provide_session
from airflow.utils.state import DagRunState
from airflow.utils.types import DagRunType
+from airflow.www.decorators import action_logging
+
+RESOURCE_EVENT_PREFIX = "dag_run"
@security.requires_access(
@@ -281,6 +285,12 @@ def get_dag_runs_batch(*, session: Session = NEW_SESSION) -> APIResponse:
],
)
@provide_session
+@action_logging(
+ event=action_event_from_permission(
+ prefix=RESOURCE_EVENT_PREFIX,
+ permission=permissions.ACTION_CAN_CREATE,
+ ),
+)
def post_dag_run(*, dag_id: str, session: Session = NEW_SESSION) -> APIResponse:
"""Trigger a DAG."""
dm = session.query(DagModel).filter(DagModel.dag_id == dag_id).first()
diff --git a/tests/api_connexion/endpoints/test_dag_run_endpoint.py b/tests/api_connexion/endpoints/test_dag_run_endpoint.py
index b645e601e9..1ea4b8d1fe 100644
--- a/tests/api_connexion/endpoints/test_dag_run_endpoint.py
+++ b/tests/api_connexion/endpoints/test_dag_run_endpoint.py
@@ -36,6 +36,7 @@ from airflow.utils.types import DagRunType
from tests.test_utils.api_connexion_utils import assert_401, create_user, delete_roles, delete_user
from tests.test_utils.config import conf_vars
from tests.test_utils.db import clear_db_dags, clear_db_runs, clear_db_serialized_dags
+from tests.test_utils.www import _check_last_log
@pytest.fixture(scope="module")
@@ -1025,7 +1026,7 @@ class TestPostDagRun(TestDagRunEndpoint):
pytest.param(None, None, None, id="all-missing"),
],
)
- def test_should_respond_200(self, logical_date_field_name, dag_run_id, logical_date, note):
+ def test_should_respond_200(self, session, logical_date_field_name, dag_run_id, logical_date, note):
self._create_dag("TEST_DAG_ID")
# We'll patch airflow.utils.timezone.utcnow to always return this so we
@@ -1070,6 +1071,7 @@ class TestPostDagRun(TestDagRunEndpoint):
"run_type": "manual",
"note": note,
}
+ _check_last_log(session, dag_id="TEST_DAG_ID", event="dag_run.create", execution_date=None)
def test_should_respond_400_if_a_dag_has_import_errors(self, session):
"""Test that if a dagmodel has import errors, dags won't be triggered"""
[airflow] 13/37: Email Config docs more explicit env var examples (#28845)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 9f3fecbdbc0a0e8b212a0a42a86839328b2afdb1
Author: Pavel T <pa...@gmail.com>
AuthorDate: Mon Feb 20 13:40:34 2023 -0500
Email Config docs more explicit env var examples (#28845)
Co-authored-by: Jarek Potiuk <ja...@potiuk.com>
Co-authored-by: Bas Harenslak <Ba...@users.noreply.github.com>
Co-authored-by: Tzu-ping Chung <ur...@gmail.com>
(cherry picked from commit d306067369d432dc81129947e730fc13de035f76)
---
docs/apache-airflow/howto/email-config.rst | 22 +++++++++++++++++++++-
1 file changed, 21 insertions(+), 1 deletion(-)
diff --git a/docs/apache-airflow/howto/email-config.rst b/docs/apache-airflow/howto/email-config.rst
index b66224aee5..014a2e2464 100644
--- a/docs/apache-airflow/howto/email-config.rst
+++ b/docs/apache-airflow/howto/email-config.rst
@@ -29,7 +29,27 @@ in the ``[email]`` section.
subject_template = /path/to/my_subject_template_file
html_content_template = /path/to/my_html_content_template_file
-You can configure sender's email address by setting ``from_email`` in the ``[email]`` section.
+Equivalent environment variables look like:
+
+.. code-block:: sh
+
+ AIRFLOW__EMAIL__EMAIL_BACKEND=airflow.utils.email.send_email_smtp
+ AIRFLOW__EMAIL__SUBJECT_TEMPLATE=/path/to/my_subject_template_file
+ AIRFLOW__EMAIL__HTML_CONTENT_TEMPLATE=/path/to/my_html_content_template_file
+
+You can configure a sender's email address by setting ``from_email`` in the ``[email]`` section like:
+
+.. code-block:: ini
+
+ [email]
+ from_email = "John Doe <jo...@example.com>"
+
+Equivalent environment variables look like:
+
+.. code-block:: sh
+
+ AIRFLOW__EMAIL__FROM_EMAIL="John Doe <jo...@example.com>"
+
To configure SMTP settings, checkout the :ref:`SMTP <config:smtp>` section in the standard configuration.
If you do not want to store the SMTP credentials in the config or in the environment variables, you can create a
[airflow] 08/37: introduce dag processor job (#28799)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 13ce6725d464debc1397589e99b1ebc9e72d439d
Author: Farhan Syakir <fa...@gmail.com>
AuthorDate: Tue Feb 21 11:54:52 2023 +0200
introduce dag processor job (#28799)
DagFileProcessorManager was not creating jobs in the metadata DB, so the livenessProbe was not valid.
A new is created for the Standalone DAG Processor.
By doing that, the airflow jobs check --hostname command would work correctly and the livenessProbe wouldn't fail
(cherry picked from commit 0018b94a4a5f846fc87457e9393ca953ba0b5ec6)
---
airflow/cli/commands/dag_processor_command.py | 18 +++----
airflow/jobs/dag_processor_job.py | 69 ++++++++++++++++++++++++
tests/cli/commands/test_dag_processor_command.py | 13 ++---
3 files changed, 83 insertions(+), 17 deletions(-)
diff --git a/airflow/cli/commands/dag_processor_command.py b/airflow/cli/commands/dag_processor_command.py
index f8ce65663b..d96fb4f06f 100644
--- a/airflow/cli/commands/dag_processor_command.py
+++ b/airflow/cli/commands/dag_processor_command.py
@@ -25,21 +25,21 @@ from daemon.pidfile import TimeoutPIDLockFile
from airflow import settings
from airflow.configuration import conf
-from airflow.dag_processing.manager import DagFileProcessorManager
+from airflow.jobs.dag_processor_job import DagProcessorJob
from airflow.utils import cli as cli_utils
from airflow.utils.cli import setup_locations, setup_logging
log = logging.getLogger(__name__)
-def _create_dag_processor_manager(args) -> DagFileProcessorManager:
+def _create_dag_processor_job(args) -> DagProcessorJob:
"""Creates DagFileProcessorProcess instance."""
processor_timeout_seconds: int = conf.getint("core", "dag_file_processor_timeout")
processor_timeout = timedelta(seconds=processor_timeout_seconds)
- return DagFileProcessorManager(
+ return DagProcessorJob(
+ processor_timeout=processor_timeout,
dag_directory=args.subdir,
max_runs=args.num_runs,
- processor_timeout=processor_timeout,
dag_ids=[],
pickle_dags=args.do_pickle,
)
@@ -55,7 +55,7 @@ def dag_processor(args):
if sql_conn.startswith("sqlite"):
raise SystemExit("Standalone DagProcessor is not supported when using sqlite.")
- manager = _create_dag_processor_manager(args)
+ job = _create_dag_processor_job(args)
if args.daemon:
pid, stdout, stderr, log_file = setup_locations(
@@ -74,10 +74,6 @@ def dag_processor(args):
umask=int(settings.DAEMON_UMASK, 8),
)
with ctx:
- try:
- manager.start()
- finally:
- manager.terminate()
- manager.end()
+ job.run()
else:
- manager.start()
+ job.run()
diff --git a/airflow/jobs/dag_processor_job.py b/airflow/jobs/dag_processor_job.py
new file mode 100644
index 0000000000..70690a78db
--- /dev/null
+++ b/airflow/jobs/dag_processor_job.py
@@ -0,0 +1,69 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import os
+from datetime import timedelta
+
+from airflow.dag_processing.manager import DagFileProcessorManager
+from airflow.jobs.base_job import BaseJob
+
+
+class DagProcessorJob(BaseJob):
+ """
+ :param dag_directory: Directory where DAG definitions are kept. All
+ files in file_paths should be under this directory
+ :param max_runs: The number of times to parse and schedule each file. -1
+ for unlimited.
+ :param processor_timeout: How long to wait before timing out a DAG file processor
+ :param dag_ids: if specified, only schedule tasks with these DAG IDs
+ :param pickle_dags: whether to pickle DAGs.
+ :param async_mode: Whether to start agent in async mode
+ """
+
+ __mapper_args__ = {"polymorphic_identity": "DagProcessorJob"}
+
+ def __init__(
+ self,
+ dag_directory: os.PathLike,
+ max_runs: int,
+ processor_timeout: timedelta,
+ dag_ids: list[str] | None,
+ pickle_dags: bool,
+ *args,
+ **kwargs,
+ ):
+ self.processor = DagFileProcessorManager(
+ dag_directory=dag_directory,
+ max_runs=max_runs,
+ processor_timeout=processor_timeout,
+ dag_ids=dag_ids,
+ pickle_dags=pickle_dags,
+ )
+ super().__init__(*args, **kwargs)
+
+ def _execute(self) -> None:
+ self.log.info("Starting the Dag Processor Job")
+ try:
+ self.processor.start()
+ except Exception:
+ self.log.exception("Exception when executing DagProcessorJob")
+ raise
+ finally:
+ self.processor.terminate()
+ self.processor.end()
diff --git a/tests/cli/commands/test_dag_processor_command.py b/tests/cli/commands/test_dag_processor_command.py
index b3ff84531f..69d0dcdb02 100644
--- a/tests/cli/commands/test_dag_processor_command.py
+++ b/tests/cli/commands/test_dag_processor_command.py
@@ -42,16 +42,17 @@ class TestDagProcessorCommand:
("core", "load_examples"): "False",
}
)
- @mock.patch("airflow.cli.commands.dag_processor_command.DagFileProcessorManager")
+ @mock.patch("airflow.cli.commands.dag_processor_command.DagProcessorJob")
@pytest.mark.skipif(
conf.get_mandatory_value("database", "sql_alchemy_conn").lower().startswith("sqlite"),
reason="Standalone Dag Processor doesn't support sqlite.",
)
- def test_start_manager(
+ def test_start_job(
self,
- mock_dag_manager,
+ mock_dag_job,
):
"""Ensure that DagFileProcessorManager is started"""
- args = self.parser.parse_args(["dag-processor"])
- dag_processor_command.dag_processor(args)
- mock_dag_manager.return_value.start.assert_called()
+ with conf_vars({("scheduler", "standalone_dag_processor"): "True"}):
+ args = self.parser.parse_args(["dag-processor"])
+ dag_processor_command.dag_processor(args)
+ mock_dag_job.return_value.run.assert_called()
[airflow] 21/37: Allow URI without authority and host blocks in `airflow connections add` (#28922)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 31bffa87131906f50b8b433bcc3e909dc9ffdd90
Author: Andrey Anshin <An...@taragol.is>
AuthorDate: Sat Jan 14 01:41:11 2023 +0400
Allow URI without authority and host blocks in `airflow connections add` (#28922)
(cherry picked from commit d8b84ce0e6d36850cd61b1ce37840c80aaec0116)
---
airflow/cli/commands/connection_command.py | 5 +-
tests/cli/commands/test_connection_command.py | 67 +++++++++++++++++++++++----
2 files changed, 59 insertions(+), 13 deletions(-)
diff --git a/airflow/cli/commands/connection_command.py b/airflow/cli/commands/connection_command.py
index 7737f4fa2c..c19490da93 100644
--- a/airflow/cli/commands/connection_command.py
+++ b/airflow/cli/commands/connection_command.py
@@ -132,9 +132,8 @@ def _is_stdout(fileio: io.TextIOWrapper) -> bool:
def _valid_uri(uri: str) -> bool:
- """Check if a URI is valid, by checking if both scheme and netloc are available."""
- uri_parts = urlsplit(uri)
- return uri_parts.scheme != "" and uri_parts.netloc != ""
+ """Check if a URI is valid, by checking if scheme (conn_type) provided."""
+ return urlsplit(uri).scheme != ""
@cache
diff --git a/tests/cli/commands/test_connection_command.py b/tests/cli/commands/test_connection_command.py
index 05d8417c16..b5748e7a1d 100644
--- a/tests/cli/commands/test_connection_command.py
+++ b/tests/cli/commands/test_connection_command.py
@@ -19,6 +19,7 @@ from __future__ import annotations
import io
import json
import re
+import shlex
import warnings
from contextlib import redirect_stdout
from unittest import mock
@@ -353,7 +354,7 @@ class TestCliAddConnections:
@pytest.mark.parametrize(
"cmd, expected_output, expected_conn",
[
- (
+ pytest.param(
[
"connections",
"add",
@@ -371,8 +372,9 @@ class TestCliAddConnections:
"port": 5432,
"schema": "airflow",
},
+ id="json-connection",
),
- (
+ pytest.param(
[
"connections",
"add",
@@ -391,8 +393,9 @@ class TestCliAddConnections:
"port": 5432,
"schema": "airflow",
},
+ id="uri-connection-with-description",
),
- (
+ pytest.param(
[
"connections",
"add",
@@ -411,8 +414,9 @@ class TestCliAddConnections:
"port": 5432,
"schema": "airflow",
},
+ id="uri-connection-with-description-2",
),
- (
+ pytest.param(
[
"connections",
"add",
@@ -432,8 +436,9 @@ class TestCliAddConnections:
"port": 5432,
"schema": "airflow",
},
+ id="uri-connection-with-extra",
),
- (
+ pytest.param(
[
"connections",
"add",
@@ -455,8 +460,9 @@ class TestCliAddConnections:
"port": 5432,
"schema": "airflow",
},
+ id="uri-connection-with-extra-and-description",
),
- (
+ pytest.param(
[
"connections",
"add",
@@ -480,8 +486,9 @@ class TestCliAddConnections:
"port": 9083,
"schema": "airflow",
},
+ id="individual-parts",
),
- (
+ pytest.param(
[
"connections",
"add",
@@ -504,6 +511,37 @@ class TestCliAddConnections:
"port": None,
"schema": None,
},
+ id="empty-uri-with-conn-type-and-extra",
+ ),
+ pytest.param(
+ ["connections", "add", "new6", "--conn-uri", "aws://?region_name=foo-bar-1"],
+ "Successfully added `conn_id`=new6 : aws://?region_name=foo-bar-1",
+ {
+ "conn_type": "aws",
+ "description": None,
+ "host": "",
+ "is_encrypted": False,
+ "is_extra_encrypted": True,
+ "login": None,
+ "port": None,
+ "schema": "",
+ },
+ id="uri-without-authority-and-host-blocks",
+ ),
+ pytest.param(
+ ["connections", "add", "new7", "--conn-uri", "aws://@/?region_name=foo-bar-1"],
+ "Successfully added `conn_id`=new7 : aws://@/?region_name=foo-bar-1",
+ {
+ "conn_type": "aws",
+ "description": None,
+ "host": "",
+ "is_encrypted": False,
+ "is_extra_encrypted": True,
+ "login": "",
+ "port": None,
+ "schema": "",
+ },
+ id="uri-with-@-instead-authority-and-host-blocks",
),
],
)
@@ -572,11 +610,20 @@ class TestCliAddConnections:
)
)
- def test_cli_connections_add_invalid_uri(self):
+ @pytest.mark.parametrize(
+ "invalid_uri",
+ [
+ pytest.param("nonsense_uri", id="word"),
+ pytest.param("://password:type@host:42/schema", id="missing-conn-type"),
+ ],
+ )
+ def test_cli_connections_add_invalid_uri(self, invalid_uri):
# Attempt to add with invalid uri
- with pytest.raises(SystemExit, match=r"The URI provided to --conn-uri is invalid: nonsense_uri"):
+ with pytest.raises(SystemExit, match=r"The URI provided to --conn-uri is invalid: .*"):
connection_command.connections_add(
- self.parser.parse_args(["connections", "add", "new1", f"--conn-uri={'nonsense_uri'}"])
+ self.parser.parse_args(
+ ["connections", "add", "new1", f"--conn-uri={shlex.quote(invalid_uri)}"]
+ )
)
def test_cli_connections_add_invalid_type(self):
[airflow] 35/37: Only skip provider integration tests for non-main builds (#29022)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 94bce2b76396da717de8fc7445dc2d5367e47b34
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sun Jan 22 10:47:08 2023 +0100
Only skip provider integration tests for non-main builds (#29022)
Previously we skipped all integration tests in non-main builds (i.e.
builds that were running in v2-* branches where we test and prepare
releases. This had two effects:
* the non-provider integration tests were not run before release, which
could have revealed some problems
* constraint generation was skipped for those runs because constraint
generation depended on integration tests
This is now fixed:
* integration tests are run on non-main builds
* but all provider tests are skipped there
(cherry picked from commit c04cfec6de68858d3372f16749244c34021310df)
---
.github/workflows/ci.yml | 6 ++++--
Dockerfile.ci | 13 ++++++++++++-
.../src/airflow_breeze/commands/testing_commands.py | 9 +++++++++
dev/breeze/src/airflow_breeze/params/shell_params.py | 1 +
images/breeze/output-commands-hash.txt | 4 ++--
images/breeze/output_testing_integration-tests.svg | 18 +++++++++++-------
scripts/ci/docker-compose/_docker.env | 1 +
scripts/ci/docker-compose/base.yml | 1 +
scripts/docker/entrypoint_ci.sh | 13 ++++++++++++-
9 files changed, 53 insertions(+), 13 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 86440754e3..e83e83af94 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -1007,7 +1007,8 @@ jobs:
BACKEND_VERSION: "${{needs.build-info.outputs.default-python-version}}"
JOB_ID: "integration"
COVERAGE: "${{needs.build-info.outputs.run-coverage}}"
- if: needs.build-info.outputs.run-tests == 'true' && needs.build-info.outputs.default-branch == 'main'
+ SKIP_PROVIDER_TESTS: "${{needs.build-info.outputs.default-branch != 'main'}}"
+ if: needs.build-info.outputs.run-tests == 'true'
steps:
- name: Cleanup repo
shell: bash
@@ -1066,7 +1067,8 @@ jobs:
BACKEND_VERSION: "${{needs.build-info.outputs.default-python-version}}"
JOB_ID: "integration"
COVERAGE: "${{needs.build-info.outputs.run-coverage}}"
- if: needs.build-info.outputs.run-tests == 'true' && needs.build-info.outputs.default-branch == 'main'
+ SKIP_PROVIDER_TESTS: "${{needs.build-info.outputs.default-branch != 'main'}}"
+ if: needs.build-info.outputs.run-tests == 'true'
steps:
- name: Cleanup repo
shell: bash
diff --git a/Dockerfile.ci b/Dockerfile.ci
index f3d9bff8aa..6f12c1251b 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -926,6 +926,13 @@ else
"${WWW_TESTS[@]}"
)
+ NO_PROVIDERS_INTEGRATION_TESTS=(
+ "tests/integration/api"
+ "tests/integration/cli"
+ "tests/integration/executors"
+ "tests/integration/security"
+ )
+
if [[ ${TEST_TYPE:=""} == "CLI" ]]; then
SELECTED_TESTS=("${CLI_TESTS[@]}")
elif [[ ${TEST_TYPE:=""} == "API" ]]; then
@@ -941,7 +948,11 @@ else
elif [[ ${TEST_TYPE:=""} == "Helm" ]]; then
SELECTED_TESTS=("${HELM_CHART_TESTS[@]}")
elif [[ ${TEST_TYPE:=""} == "Integration" ]]; then
- SELECTED_TESTS=("${INTEGRATION_TESTS[@]}")
+ if [[ ${SKIP_PROVIDER_TESTS:=""} == "true" ]]; then
+ SELECTED_TESTS=("${NO_PROVIDERS_INTEGRATION_TESTS[@]}")
+ else
+ SELECTED_TESTS=("${INTEGRATION_TESTS[@]}")
+ fi
elif [[ ${TEST_TYPE:=""} == "Other" ]]; then
find_all_other_tests
SELECTED_TESTS=("${ALL_OTHER_TESTS[@]}")
diff --git a/dev/breeze/src/airflow_breeze/commands/testing_commands.py b/dev/breeze/src/airflow_breeze/commands/testing_commands.py
index 1dcfa1196d..74c3069417 100644
--- a/dev/breeze/src/airflow_breeze/commands/testing_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/testing_commands.py
@@ -129,6 +129,7 @@ def _run_test(
env_variables["DB_RESET"] = "true"
perform_environment_checks()
env_variables["TEST_TYPE"] = exec_shell_params.test_type
+ env_variables["SKIP_PROVIDER_TESTS"] = str(exec_shell_params.skip_provider_tests).lower()
if "[" in exec_shell_params.test_type and not exec_shell_params.test_type.startswith("Providers"):
get_console(output=output).print(
"[error]Only 'Providers' test type can specify actual tests with \\[\\][/]"
@@ -442,6 +443,12 @@ def tests(
type=IntRange(min=0),
show_default=True,
)
+@click.option(
+ "--skip-provider-tests",
+ help="Skip provider tests",
+ is_flag=True,
+ envvar="SKIP_PROVIDER_TESTS",
+)
@option_db_reset
@option_verbose
@option_dry_run
@@ -454,6 +461,7 @@ def integration_tests(
mssql_version: str,
integration: tuple,
test_timeout: int,
+ skip_provider_tests: bool,
db_reset: bool,
image_tag: str | None,
mount_sources: str,
@@ -472,6 +480,7 @@ def integration_tests(
mount_sources=mount_sources,
forward_ports=False,
test_type="Integration",
+ skip_provider_tests=skip_provider_tests,
)
cleanup_python_generated_files()
returncode, _ = _run_test(
diff --git a/dev/breeze/src/airflow_breeze/params/shell_params.py b/dev/breeze/src/airflow_breeze/params/shell_params.py
index f69a950389..fad96e81f5 100644
--- a/dev/breeze/src/airflow_breeze/params/shell_params.py
+++ b/dev/breeze/src/airflow_breeze/params/shell_params.py
@@ -105,6 +105,7 @@ class ShellParams:
skip_constraints: bool = False
start_airflow: str = "false"
test_type: str | None = None
+ skip_provider_tests: bool = False
use_airflow_version: str | None = None
use_packages_from_dist: bool = False
version_suffix_for_pypi: str = ""
diff --git a/images/breeze/output-commands-hash.txt b/images/breeze/output-commands-hash.txt
index 5fa6522464..e580f65d33 100644
--- a/images/breeze/output-commands-hash.txt
+++ b/images/breeze/output-commands-hash.txt
@@ -59,6 +59,6 @@ static-checks:f45ad432bdc47a2256fdb0277b19d816
stop:8969537ccdd799f692ccb8600a7bbed6
testing:docker-compose-tests:b86c044b24138af0659a05ed6331576c
testing:helm-tests:94a442e7f3f63b34c4831a84d165690a
-testing:integration-tests:4c8321c60b2112b12c866beebeb6c61b
+testing:integration-tests:585da1e636f710be9c9de36a71586963
testing:tests:1b67704a3f8eae3a6db8d7c8d6cc162a
-testing:0614ca49c1b4c2b0b1d92c31278dc4f9
+testing:f07eb24d62e6628c0899d643ae3fe353
diff --git a/images/breeze/output_testing_integration-tests.svg b/images/breeze/output_testing_integration-tests.svg
index 42fe4cc6bb..9ebb331636 100644
--- a/images/breeze/output_testing_integration-tests.svg
+++ b/images/breeze/output_testing_integration-tests.svg
@@ -1,4 +1,4 @@
-<svg class="rich-terminal" viewBox="0 0 1482 806.4" xmlns="http://www.w3.org/2000/svg">
+<svg class="rich-terminal" viewBox="0 0 1482 830.8" xmlns="http://www.w3.org/2000/svg">
<!-- Generated with Rich https://www.textualize.io -->
<style>
@@ -43,7 +43,7 @@
<defs>
<clipPath id="breeze-testing-integration-tests-clip-terminal">
- <rect x="0" y="0" width="1463.0" height="755.4" />
+ <rect x="0" y="0" width="1463.0" height="779.8" />
</clipPath>
<clipPath id="breeze-testing-integration-tests-line-0">
<rect x="0" y="1.5" width="1464" height="24.65"/>
@@ -135,9 +135,12 @@
<clipPath id="breeze-testing-integration-tests-line-29">
<rect x="0" y="709.1" width="1464" height="24.65"/>
</clipPath>
+<clipPath id="breeze-testing-integration-tests-line-30">
+ <rect x="0" y="733.5" width="1464" height="24.65"/>
+ </clipPath>
</defs>
- <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="804.4" rx="8"/><text class="breeze-testing-integration-tests-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: testing integration-tests</text>
+ <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="828.8" rx="8"/><text class="breeze-testing-integration-tests-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: testing integration-tests</text>
<g transform="translate(26,22)">
<circle cx="0" cy="0" r="7" fill="#ff5f57"/>
<circle cx="22" cy="0" r="7" fill="#febc2e"/>
@@ -174,10 +177,11 @@
</text><text class="breeze-testing-integration-tests-r5" x="0" y="605.6" textLength="12.2" clip-path="url(#breeze-testing-integration-tests-line-24)">│</text><text class="breeze-testing-integration-tests-r5" x="280.6" y="605.6" textLength="1159" clip-path="url(#breeze-testing-integration-tests-line-24)">[default: selected]                           & [...]
</text><text class="breeze-testing-integration-tests-r5" x="0" y="630" textLength="1464" clip-path="url(#breeze-testing-integration-tests-line-25)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-testing-integration-tests-r2" x="1464" y="630" textLength="12.2" clip-path="url(#breeze-testing-integration-tests-line-25)">
</text><text class="breeze-testing-integration-tests-r5" x="0" y="654.4" textLength="24.4" clip-path="url(#breeze-testing-integration-tests-line-26)">╭─</text><text class="breeze-testing-integration-tests-r5" x="24.4" y="654.4" textLength="195.2" clip-path="url(#breeze-testing-integration-tests-line-26)"> Common options </text><text class="breeze-testing-integration-tests-r5" x="219.6" y="654.4" textLength="1220" clip-path="url(#breeze-testing-integration-tests-line-26)">─ [...]
-</text><text class="breeze-testing-integration-tests-r5" x="0" y="678.8" textLength="12.2" clip-path="url(#breeze-testing-integration-tests-line-27)">│</text><text class="breeze-testing-integration-tests-r4" x="24.4" y="678.8" textLength="12.2" clip-path="url(#breeze-testing-integration-tests-line-27)">-</text><text class="breeze-testing-integration-tests-r4" x="36.6" y="678.8" textLength="97.6" clip-path="url(#breeze-testing-integration-tests-line-27)">-verbose</text><text class="breeze [...]
-</text><text class="breeze-testing-integration-tests-r5" x="0" y="703.2" textLength="12.2" clip-path="url(#breeze-testing-integration-tests-line-28)">│</text><text class="breeze-testing-integration-tests-r4" x="24.4" y="703.2" textLength="12.2" clip-path="url(#breeze-testing-integration-tests-line-28)">-</text><text class="breeze-testing-integration-tests-r4" x="36.6" y="703.2" textLength="48.8" clip-path="url(#breeze-testing-integration-tests-line-28)">-dry</text><text class="breeze-tes [...]
-</text><text class="breeze-testing-integration-tests-r5" x="0" y="727.6" textLength="12.2" clip-path="url(#breeze-testing-integration-tests-line-29)">│</text><text class="breeze-testing-integration-tests-r4" x="24.4" y="727.6" textLength="12.2" clip-path="url(#breeze-testing-integration-tests-line-29)">-</text><text class="breeze-testing-integration-tests-r4" x="36.6" y="727.6" textLength="61" clip-path="url(#breeze-testing-integration-tests-line-29)">-help</text><text class="breeze-test [...]
-</text><text class="breeze-testing-integration-tests-r5" x="0" y="752" textLength="1464" clip-path="url(#breeze-testing-integration-tests-line-30)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-testing-integration-tests-r2" x="1464" y="752" textLength="12.2" clip-path="url(#breeze-testing-integration-tests-line-30)">
+</text><text class="breeze-testing-integration-tests-r5" x="0" y="678.8" textLength="12.2" clip-path="url(#breeze-testing-integration-tests-line-27)">│</text><text class="breeze-testing-integration-tests-r4" x="24.4" y="678.8" textLength="12.2" clip-path="url(#breeze-testing-integration-tests-line-27)">-</text><text class="breeze-testing-integration-tests-r4" x="36.6" y="678.8" textLength="61" clip-path="url(#breeze-testing-integration-tests-line-27)">-skip</text><text class="breeze-test [...]
+</text><text class="breeze-testing-integration-tests-r5" x="0" y="703.2" textLength="12.2" clip-path="url(#breeze-testing-integration-tests-line-28)">│</text><text class="breeze-testing-integration-tests-r4" x="24.4" y="703.2" textLength="12.2" clip-path="url(#breeze-testing-integration-tests-line-28)">-</text><text class="breeze-testing-integration-tests-r4" x="36.6" y="703.2" textLength="97.6" clip-path="url(#breeze-testing-integration-tests-line-28)">-verbose</text><text class="breeze [...]
+</text><text class="breeze-testing-integration-tests-r5" x="0" y="727.6" textLength="12.2" clip-path="url(#breeze-testing-integration-tests-line-29)">│</text><text class="breeze-testing-integration-tests-r4" x="24.4" y="727.6" textLength="12.2" clip-path="url(#breeze-testing-integration-tests-line-29)">-</text><text class="breeze-testing-integration-tests-r4" x="36.6" y="727.6" textLength="48.8" clip-path="url(#breeze-testing-integration-tests-line-29)">-dry</text><text class="breeze-tes [...]
+</text><text class="breeze-testing-integration-tests-r5" x="0" y="752" textLength="12.2" clip-path="url(#breeze-testing-integration-tests-line-30)">│</text><text class="breeze-testing-integration-tests-r4" x="24.4" y="752" textLength="12.2" clip-path="url(#breeze-testing-integration-tests-line-30)">-</text><text class="breeze-testing-integration-tests-r4" x="36.6" y="752" textLength="61" clip-path="url(#breeze-testing-integration-tests-line-30)">-help</text><text class="breeze-testing-in [...]
+</text><text class="breeze-testing-integration-tests-r5" x="0" y="776.4" textLength="1464" clip-path="url(#breeze-testing-integration-tests-line-31)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-testing-integration-tests-r2" x="1464" y="776.4" textLength="12.2" clip-path="url(#breeze-testing-integration-tests-line-31)">
</text>
</g>
</g>
diff --git a/scripts/ci/docker-compose/_docker.env b/scripts/ci/docker-compose/_docker.env
index 34b9381f81..dbea935eea 100644
--- a/scripts/ci/docker-compose/_docker.env
+++ b/scripts/ci/docker-compose/_docker.env
@@ -59,6 +59,7 @@ RUN_SYSTEM_TESTS
START_AIRFLOW
SKIP_CONSTRAINTS
SKIP_ENVIRONMENT_INITIALIZATION
+SKIP_PROVIDER_TESTS
SKIP_SSH_SETUP
TEST_TIMEOUT
TEST_TYPE
diff --git a/scripts/ci/docker-compose/base.yml b/scripts/ci/docker-compose/base.yml
index ea95ede5f5..7f11966b30 100644
--- a/scripts/ci/docker-compose/base.yml
+++ b/scripts/ci/docker-compose/base.yml
@@ -72,6 +72,7 @@ services:
- START_AIRFLOW=${START_AIRFLOW}
- SKIP_CONSTRAINTS=${SKIP_CONSTRAINTS}
- SKIP_ENVIRONMENT_INITIALIZATION=${SKIP_ENVIRONMENT_INITIALIZATION}
+ - SKIP_PROVIDER_TESTS=${SKIP_PROVIDER_TESTS}
- SKIP_SSH_SETUP=${SKIP_SSH_SETUP}
- TEST_TYPE=${TEST_TYPE}
- TEST_TIMEOUT=${TEST_TIMEOUT}
diff --git a/scripts/docker/entrypoint_ci.sh b/scripts/docker/entrypoint_ci.sh
index 190995590d..76ccbdeab7 100755
--- a/scripts/docker/entrypoint_ci.sh
+++ b/scripts/docker/entrypoint_ci.sh
@@ -378,6 +378,13 @@ else
"${WWW_TESTS[@]}"
)
+ NO_PROVIDERS_INTEGRATION_TESTS=(
+ "tests/integration/api"
+ "tests/integration/cli"
+ "tests/integration/executors"
+ "tests/integration/security"
+ )
+
if [[ ${TEST_TYPE:=""} == "CLI" ]]; then
SELECTED_TESTS=("${CLI_TESTS[@]}")
elif [[ ${TEST_TYPE:=""} == "API" ]]; then
@@ -393,7 +400,11 @@ else
elif [[ ${TEST_TYPE:=""} == "Helm" ]]; then
SELECTED_TESTS=("${HELM_CHART_TESTS[@]}")
elif [[ ${TEST_TYPE:=""} == "Integration" ]]; then
- SELECTED_TESTS=("${INTEGRATION_TESTS[@]}")
+ if [[ ${SKIP_PROVIDER_TESTS:=""} == "true" ]]; then
+ SELECTED_TESTS=("${NO_PROVIDERS_INTEGRATION_TESTS[@]}")
+ else
+ SELECTED_TESTS=("${INTEGRATION_TESTS[@]}")
+ fi
elif [[ ${TEST_TYPE:=""} == "Other" ]]; then
find_all_other_tests
SELECTED_TESTS=("${ALL_OTHER_TESTS[@]}")
[airflow] 10/37: Bump swagger-ui-dist from 3.52.0 to 4.1.3 in /airflow/www (#28824)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit a833dfb72008ffbe49b1fd8d3d91402b8aaafa66
Author: Dependabot [bot] <49...@users.noreply.github.com>
AuthorDate: Tue Jan 10 22:21:44 2023 +0100
Bump swagger-ui-dist from 3.52.0 to 4.1.3 in /airflow/www (#28824)
Bumps [swagger-ui-dist](https://github.com/swagger-api/swagger-ui) from 3.52.0 to 4.1.3.
- [Release notes](https://github.com/swagger-api/swagger-ui/releases)
- [Commits](https://github.com/swagger-api/swagger-ui/compare/v3.52.0...v4.1.3)
---
updated-dependencies:
- dependency-name: swagger-ui-dist
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <su...@github.com>
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
(cherry picked from commit e1e3a8e94dd29630a8912aab5c1bf0522c2ec9aa)
---
airflow/www/package.json | 252 +++++++++++++++++++++++------------------------
airflow/www/yarn.lock | 8 +-
2 files changed, 130 insertions(+), 130 deletions(-)
diff --git a/airflow/www/package.json b/airflow/www/package.json
index 37565483c2..a9bee3c1eb 100644
--- a/airflow/www/package.json
+++ b/airflow/www/package.json
@@ -1,128 +1,128 @@
{
- "name": "airflow-www",
- "version": "1.0.0",
- "description": "Apache Airflow is a platform to programmatically author, schedule and monitor workflows.",
- "scripts": {
- "test": "jest",
- "dev": "NODE_ENV=development webpack --watch --progress --devtool eval-cheap-source-map --mode development",
- "prod": "NODE_ENV=production node --max_old_space_size=4096 ./node_modules/webpack/bin/webpack.js --mode production --progress",
- "build": "NODE_ENV=production webpack --progress --mode production",
- "lint": "eslint --ignore-path=.eslintignore --ext .js,.jsx,.ts,.tsx . && tsc",
- "lint:fix": "eslint --fix --ignore-path=.eslintignore --ext .js,.jsx,.ts,.tsx . && tsc",
- "generate-api-types": "npx openapi-typescript \"../api_connexion/openapi/v1.yaml\" --output static/js/types/api-generated.ts && node alias-rest-types.js static/js/types/api-generated.ts"
- },
- "author": "Apache",
- "license": "Apache-2.0",
- "repository": {
- "type": "git",
- "url": "git+https://github.com/apache/airflow.git"
- },
- "homepage": "https://airflow.apache.org/",
- "keywords": [
- "big",
- "data",
- "workflow",
- "airflow",
- "d3",
- "nerds",
- "database",
- "flask"
- ],
- "devDependencies": {
- "@babel/core": "^7.18.5",
- "@babel/eslint-parser": "^7.18.2",
- "@babel/plugin-transform-runtime": "^7.16.0",
- "@babel/preset-env": "^7.16.0",
- "@babel/preset-react": "^7.16.0",
- "@babel/preset-typescript": "^7.17.12",
- "@testing-library/jest-dom": "^5.16.0",
- "@testing-library/react": "^13.0.0",
- "@types/color": "^3.0.3",
- "@types/react": "^18.0.12",
- "@types/react-dom": "^18.0.5",
- "@types/react-table": "^7.7.12",
- "@typescript-eslint/eslint-plugin": "^5.13.0",
- "@typescript-eslint/parser": "^5.0.0",
- "babel-jest": "^27.3.1",
- "babel-loader": "^8.1.0",
- "clean-webpack-plugin": "^3.0.0",
- "copy-webpack-plugin": "^6.0.3",
- "css-loader": "5.2.7",
- "css-minimizer-webpack-plugin": "^4.0.0",
- "eslint": "^8.6.0",
- "eslint-config-airbnb": "^19.0.4",
- "eslint-config-airbnb-typescript": "^17.0.0",
- "eslint-plugin-html": "^6.0.2",
- "eslint-plugin-import": "^2.25.3",
- "eslint-plugin-jsx-a11y": "^6.5.0",
- "eslint-plugin-node": "^11.1.0",
- "eslint-plugin-promise": "^4.2.1",
- "eslint-plugin-react": "^7.30.0",
- "eslint-plugin-react-hooks": "^4.5.0",
- "eslint-plugin-standard": "^4.0.1",
- "file-loader": "^6.0.0",
- "imports-loader": "^1.1.0",
- "jest": "^27.3.1",
- "mini-css-extract-plugin": "^1.6.2",
- "moment": "^2.29.4",
- "moment-locales-webpack-plugin": "^1.2.0",
- "nock": "^13.2.4",
- "openapi-typescript": "^5.4.1",
- "style-loader": "^1.2.1",
- "stylelint": "^13.6.1",
- "stylelint-config-standard": "^20.0.0",
- "terser-webpack-plugin": "<5.0.0",
- "typescript": "^4.6.3",
- "url-loader": "4.1.0",
- "web-worker": "^1.2.0",
- "webpack": "^5.73.0",
- "webpack-cli": "^4.0.0",
- "webpack-license-plugin": "^4.2.1",
- "webpack-manifest-plugin": "^4.0.0"
- },
- "dependencies": {
- "@chakra-ui/react": "^2.2.0",
- "@emotion/cache": "^11.9.3",
- "@emotion/react": "^11.9.3",
- "@emotion/styled": "^11",
- "@visx/group": "^2.10.0",
- "@visx/marker": "^2.12.2",
- "@visx/shape": "^2.12.2",
- "@visx/zoom": "^2.10.0",
- "axios": "^0.26.0",
- "bootstrap-3-typeahead": "^4.0.2",
- "camelcase-keys": "^7.0.0",
- "chakra-react-select": "^4.0.0",
- "codemirror": "^5.59.1",
- "color": "^4.2.3",
- "d3": "^3.4.4",
- "d3-shape": "^2.1.0",
- "d3-tip": "^0.9.1",
- "dagre-d3": "^0.6.4",
- "datatables.net": "^1.11.4",
- "datatables.net-bs": "^1.11.4",
- "elkjs": "^0.7.1",
- "eonasdan-bootstrap-datetimepicker": "^4.17.47",
- "framer-motion": "^6.0.0",
- "jquery": ">=3.5.0",
- "jshint": "^2.13.4",
- "lodash": "^4.17.21",
- "moment-timezone": "^0.5.35",
- "nvd3": "^1.8.6",
- "react": "^18.0.0",
- "react-dom": "^18.0.0",
- "react-icons": "^4.3.1",
- "react-json-view": "^1.21.3",
- "react-query": "^3.39.1",
- "react-router-dom": "^6.3.0",
- "react-table": "^7.8.0",
- "react-textarea-autosize": "^8.3.4",
- "redoc": "^2.0.0-rc.72",
- "swagger-ui-dist": "3.52.0",
- "type-fest": "^2.17.0",
- "url-search-params-polyfill": "^8.1.0"
- },
- "resolutions": {
- "d3-color": "^3.1.0"
- }
+ "name": "airflow-www",
+ "version": "1.0.0",
+ "description": "Apache Airflow is a platform to programmatically author, schedule and monitor workflows.",
+ "scripts": {
+ "test": "jest",
+ "dev": "NODE_ENV=development webpack --watch --progress --devtool eval-cheap-source-map --mode development",
+ "prod": "NODE_ENV=production node --max_old_space_size=4096 ./node_modules/webpack/bin/webpack.js --mode production --progress",
+ "build": "NODE_ENV=production webpack --progress --mode production",
+ "lint": "eslint --ignore-path=.eslintignore --ext .js,.jsx,.ts,.tsx . && tsc",
+ "lint:fix": "eslint --fix --ignore-path=.eslintignore --ext .js,.jsx,.ts,.tsx . && tsc",
+ "generate-api-types": "npx openapi-typescript \"../api_connexion/openapi/v1.yaml\" --output static/js/types/api-generated.ts && node alias-rest-types.js static/js/types/api-generated.ts"
+ },
+ "author": "Apache",
+ "license": "Apache-2.0",
+ "repository": {
+ "type": "git",
+ "url": "git+https://github.com/apache/airflow.git"
+ },
+ "homepage": "https://airflow.apache.org/",
+ "keywords": [
+ "big",
+ "data",
+ "workflow",
+ "airflow",
+ "d3",
+ "nerds",
+ "database",
+ "flask"
+ ],
+ "devDependencies": {
+ "@babel/core": "^7.18.5",
+ "@babel/eslint-parser": "^7.18.2",
+ "@babel/plugin-transform-runtime": "^7.16.0",
+ "@babel/preset-env": "^7.16.0",
+ "@babel/preset-react": "^7.16.0",
+ "@babel/preset-typescript": "^7.17.12",
+ "@testing-library/jest-dom": "^5.16.0",
+ "@testing-library/react": "^13.0.0",
+ "@types/color": "^3.0.3",
+ "@types/react": "^18.0.12",
+ "@types/react-dom": "^18.0.5",
+ "@types/react-table": "^7.7.12",
+ "@typescript-eslint/eslint-plugin": "^5.13.0",
+ "@typescript-eslint/parser": "^5.0.0",
+ "babel-jest": "^27.3.1",
+ "babel-loader": "^8.1.0",
+ "clean-webpack-plugin": "^3.0.0",
+ "copy-webpack-plugin": "^6.0.3",
+ "css-loader": "5.2.7",
+ "css-minimizer-webpack-plugin": "^4.0.0",
+ "eslint": "^8.6.0",
+ "eslint-config-airbnb": "^19.0.4",
+ "eslint-config-airbnb-typescript": "^17.0.0",
+ "eslint-plugin-html": "^6.0.2",
+ "eslint-plugin-import": "^2.25.3",
+ "eslint-plugin-jsx-a11y": "^6.5.0",
+ "eslint-plugin-node": "^11.1.0",
+ "eslint-plugin-promise": "^4.2.1",
+ "eslint-plugin-react": "^7.30.0",
+ "eslint-plugin-react-hooks": "^4.5.0",
+ "eslint-plugin-standard": "^4.0.1",
+ "file-loader": "^6.0.0",
+ "imports-loader": "^1.1.0",
+ "jest": "^27.3.1",
+ "mini-css-extract-plugin": "^1.6.2",
+ "moment": "^2.29.4",
+ "moment-locales-webpack-plugin": "^1.2.0",
+ "nock": "^13.2.4",
+ "openapi-typescript": "^5.4.1",
+ "style-loader": "^1.2.1",
+ "stylelint": "^13.6.1",
+ "stylelint-config-standard": "^20.0.0",
+ "terser-webpack-plugin": "<5.0.0",
+ "typescript": "^4.6.3",
+ "url-loader": "4.1.0",
+ "web-worker": "^1.2.0",
+ "webpack": "^5.73.0",
+ "webpack-cli": "^4.0.0",
+ "webpack-license-plugin": "^4.2.1",
+ "webpack-manifest-plugin": "^4.0.0"
+ },
+ "dependencies": {
+ "@chakra-ui/react": "^2.2.0",
+ "@emotion/cache": "^11.9.3",
+ "@emotion/react": "^11.9.3",
+ "@emotion/styled": "^11",
+ "@visx/group": "^2.10.0",
+ "@visx/marker": "^2.12.2",
+ "@visx/shape": "^2.12.2",
+ "@visx/zoom": "^2.10.0",
+ "axios": "^0.26.0",
+ "bootstrap-3-typeahead": "^4.0.2",
+ "camelcase-keys": "^7.0.0",
+ "chakra-react-select": "^4.0.0",
+ "codemirror": "^5.59.1",
+ "color": "^4.2.3",
+ "d3": "^3.4.4",
+ "d3-shape": "^2.1.0",
+ "d3-tip": "^0.9.1",
+ "dagre-d3": "^0.6.4",
+ "datatables.net": "^1.11.4",
+ "datatables.net-bs": "^1.11.4",
+ "elkjs": "^0.7.1",
+ "eonasdan-bootstrap-datetimepicker": "^4.17.47",
+ "framer-motion": "^6.0.0",
+ "jquery": ">=3.5.0",
+ "jshint": "^2.13.4",
+ "lodash": "^4.17.21",
+ "moment-timezone": "^0.5.35",
+ "nvd3": "^1.8.6",
+ "react": "^18.0.0",
+ "react-dom": "^18.0.0",
+ "react-icons": "^4.3.1",
+ "react-json-view": "^1.21.3",
+ "react-query": "^3.39.1",
+ "react-router-dom": "^6.3.0",
+ "react-table": "^7.8.0",
+ "react-textarea-autosize": "^8.3.4",
+ "redoc": "^2.0.0-rc.72",
+ "swagger-ui-dist": "4.1.3",
+ "type-fest": "^2.17.0",
+ "url-search-params-polyfill": "^8.1.0"
+ },
+ "resolutions": {
+ "d3-color": "^3.1.0"
+ }
}
diff --git a/airflow/www/yarn.lock b/airflow/www/yarn.lock
index f25e2647bd..1241de45bc 100644
--- a/airflow/www/yarn.lock
+++ b/airflow/www/yarn.lock
@@ -9880,10 +9880,10 @@ svgo@^2.7.0:
picocolors "^1.0.0"
stable "^0.1.8"
-swagger-ui-dist@3.52.0:
- version "3.52.0"
- resolved "https://registry.yarnpkg.com/swagger-ui-dist/-/swagger-ui-dist-3.52.0.tgz#cb16ec6dcdf134ff47cbfe57cba7be7748429142"
- integrity sha512-SGfhW8FCih00QG59PphdeAUtTNw7HS5k3iPqDZowerPw9mcbhKchUb12kbROk99c1X6RTWW1gB1kqgfnYGuCSg==
+swagger-ui-dist@4.1.3:
+ version "4.1.3"
+ resolved "https://registry.yarnpkg.com/swagger-ui-dist/-/swagger-ui-dist-4.1.3.tgz#2be9f9de9b5c19132fa4a5e40933058c151563dc"
+ integrity sha512-WvfPSfAAMlE/sKS6YkW47nX/hA7StmhYnAHc6wWCXNL0oclwLj6UXv0hQCkLnDgvebi0MEV40SJJpVjKUgH1IQ==
swagger2openapi@^7.0.6:
version "7.0.6"
[airflow] 30/37: Fix dag run trigger with a note. (#29228)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 9891eef2abdade85c2956f99e29405fa7ecc960b
Author: Vedant Lodha <ve...@hotmail.com>
AuthorDate: Thu Feb 2 01:07:39 2023 +0530
Fix dag run trigger with a note. (#29228)
* Fix dag run trigger with a note.
Currently, triggering dag run with note gives 400. This PR fixes it.
Closes: #28825
* make flask_login.current_user a global import
* precommit error fixes.
* add test coverage for the fix
(cherry picked from commit b94f36bf563f5c8372086cec63b74eadef638ef8)
---
airflow/api_connexion/endpoints/dag_run_endpoint.py | 7 +++++--
airflow/api_connexion/schemas/dag_run_schema.py | 2 +-
tests/api_connexion/endpoints/test_dag_run_endpoint.py | 14 +++++++-------
3 files changed, 13 insertions(+), 10 deletions(-)
diff --git a/airflow/api_connexion/endpoints/dag_run_endpoint.py b/airflow/api_connexion/endpoints/dag_run_endpoint.py
index b566ba2df7..2b3ee16b2f 100644
--- a/airflow/api_connexion/endpoints/dag_run_endpoint.py
+++ b/airflow/api_connexion/endpoints/dag_run_endpoint.py
@@ -21,6 +21,7 @@ from http import HTTPStatus
import pendulum
from connexion import NoContent
from flask import g
+from flask_login import current_user
from marshmallow import ValidationError
from sqlalchemy import or_
from sqlalchemy.orm import Query, Session
@@ -319,6 +320,10 @@ def post_dag_run(*, dag_id: str, session: Session = NEW_SESSION) -> APIResponse:
dag_hash=get_airflow_app().dag_bag.dags_hash.get(dag_id),
session=session,
)
+ dag_run_note = post_body.get("note")
+ if dag_run_note:
+ current_user_id = getattr(current_user, "id", None)
+ dag_run.note = (dag_run_note, current_user_id)
return dagrun_schema.dump(dag_run)
except ValueError as ve:
raise BadRequest(detail=str(ve))
@@ -438,8 +443,6 @@ def set_dag_run_note(*, dag_id: str, dag_run_id: str, session: Session = NEW_SES
except ValidationError as err:
raise BadRequest(detail=str(err))
- from flask_login import current_user
-
current_user_id = getattr(current_user, "id", None)
if dag_run.dag_run_note is None:
dag_run.note = (new_note, current_user_id)
diff --git a/airflow/api_connexion/schemas/dag_run_schema.py b/airflow/api_connexion/schemas/dag_run_schema.py
index 7ca857951b..999b533728 100644
--- a/airflow/api_connexion/schemas/dag_run_schema.py
+++ b/airflow/api_connexion/schemas/dag_run_schema.py
@@ -73,7 +73,7 @@ class DAGRunSchema(SQLAlchemySchema):
data_interval_end = auto_field(dump_only=True)
last_scheduling_decision = auto_field(dump_only=True)
run_type = auto_field(dump_only=True)
- note = auto_field(dump_only=True)
+ note = auto_field(dump_only=False)
@pre_load
def autogenerate(self, data, **kwargs):
diff --git a/tests/api_connexion/endpoints/test_dag_run_endpoint.py b/tests/api_connexion/endpoints/test_dag_run_endpoint.py
index 8b02bb321f..b645e601e9 100644
--- a/tests/api_connexion/endpoints/test_dag_run_endpoint.py
+++ b/tests/api_connexion/endpoints/test_dag_run_endpoint.py
@@ -1018,14 +1018,14 @@ class TestGetDagRunBatchDateFilters(TestDagRunEndpoint):
class TestPostDagRun(TestDagRunEndpoint):
@pytest.mark.parametrize("logical_date_field_name", ["execution_date", "logical_date"])
@pytest.mark.parametrize(
- "dag_run_id, logical_date",
+ "dag_run_id, logical_date, note",
[
- pytest.param("TEST_DAG_RUN", "2020-06-11T18:00:00+00:00", id="both-present"),
- pytest.param(None, "2020-06-11T18:00:00+00:00", id="only-date"),
- pytest.param(None, None, id="both-missing"),
+ pytest.param("TEST_DAG_RUN", "2020-06-11T18:00:00+00:00", "test-note", id="all-present"),
+ pytest.param(None, "2020-06-11T18:00:00+00:00", None, id="only-date"),
+ pytest.param(None, None, None, id="all-missing"),
],
)
- def test_should_respond_200(self, logical_date_field_name, dag_run_id, logical_date):
+ def test_should_respond_200(self, logical_date_field_name, dag_run_id, logical_date, note):
self._create_dag("TEST_DAG_ID")
# We'll patch airflow.utils.timezone.utcnow to always return this so we
@@ -1037,7 +1037,7 @@ class TestPostDagRun(TestDagRunEndpoint):
request_json[logical_date_field_name] = logical_date
if dag_run_id is not None:
request_json["dag_run_id"] = dag_run_id
-
+ request_json["note"] = note
with mock.patch("airflow.utils.timezone.utcnow", lambda: fixed_now):
response = self.client.post(
"api/v1/dags/TEST_DAG_ID/dagRuns",
@@ -1068,7 +1068,7 @@ class TestPostDagRun(TestDagRunEndpoint):
"data_interval_start": expected_logical_date,
"last_scheduling_decision": None,
"run_type": "manual",
- "note": None,
+ "note": note,
}
def test_should_respond_400_if_a_dag_has_import_errors(self, session):
[airflow] 16/37: Add dep context description for better log message (#28875)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 8719079355fa27b56329a90b153b9acbf53c6eff
Author: Daniel Standish <15...@users.noreply.github.com>
AuthorDate: Wed Jan 11 23:56:49 2023 -0800
Add dep context description for better log message (#28875)
Otherwise, it appears that there is a duplicate log record.
(cherry picked from commit 1ca94ee6ba767ed6851858db24319aa1008562eb)
---
airflow/models/taskinstance.py | 4 +++-
airflow/ti_deps/dep_context.py | 1 +
2 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/airflow/models/taskinstance.py b/airflow/models/taskinstance.py
index 9708d73124..f237fe35e3 100644
--- a/airflow/models/taskinstance.py
+++ b/airflow/models/taskinstance.py
@@ -1080,7 +1080,7 @@ class TaskInstance(Base, LoggingMixin):
if failed:
return False
- verbose_aware_logger("Dependencies all met for %s", self)
+ verbose_aware_logger("Dependencies all met for dep_context=%s ti=%s", dep_context.description, self)
return True
@provide_session
@@ -1230,6 +1230,7 @@ class TaskInstance(Base, LoggingMixin):
ignore_ti_state=ignore_ti_state,
ignore_depends_on_past=ignore_depends_on_past,
ignore_task_deps=ignore_task_deps,
+ description="non-requeueable deps",
)
if not self.are_dependencies_met(
dep_context=non_requeueable_dep_context, session=session, verbose=True
@@ -1258,6 +1259,7 @@ class TaskInstance(Base, LoggingMixin):
ignore_depends_on_past=ignore_depends_on_past,
ignore_task_deps=ignore_task_deps,
ignore_ti_state=ignore_ti_state,
+ description="requeueable deps",
)
if not self.are_dependencies_met(dep_context=dep_context, session=session, verbose=True):
self.state = State.NONE
diff --git a/airflow/ti_deps/dep_context.py b/airflow/ti_deps/dep_context.py
index 829e396417..5ddecb21c7 100644
--- a/airflow/ti_deps/dep_context.py
+++ b/airflow/ti_deps/dep_context.py
@@ -73,6 +73,7 @@ class DepContext:
ignore_ti_state: bool = False
ignore_unmapped_tasks: bool = False
finished_tis: list[TaskInstance] | None = None
+ description: str | None = None
have_changed_ti_states: bool = False
"""Have any of the TIs state's been changed as a result of evaluating dependencies"""
[airflow] 34/37: change context example to no longer us as variable (#29021)
Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 0bab483f43d761293b863fbad1bd288f6efeda84
Author: Frank Cash <fc...@astronomer.io>
AuthorDate: Wed Jan 18 15:18:48 2023 -0500
change context example to no longer us as variable (#29021)
https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html#dags-used-in-a-context-manager-no-longer-need-to-be-assigned-to-a-module-variable-23592
(cherry picked from commit bc5cecc0db27cb8684c238b36ad12c7217d0c3ca)
---
docs/apache-airflow/core-concepts/dags.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/apache-airflow/core-concepts/dags.rst b/docs/apache-airflow/core-concepts/dags.rst
index 91bcde51ca..527965b9e6 100644
--- a/docs/apache-airflow/core-concepts/dags.rst
+++ b/docs/apache-airflow/core-concepts/dags.rst
@@ -40,7 +40,7 @@ which will add the DAG to anything inside it implicitly::
with DAG(
"my_dag_name", start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
schedule="@daily", catchup=False
- ) as dag:
+ ):
op = EmptyOperator(task_id="task")
Or, you can use a standard constructor, passing the dag into any