You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ep...@apache.org on 2023/08/09 07:35:42 UTC
[airflow] branch v2-7-test updated (b37fadf139 -> 9c36dd0b9e)
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a change to branch v2-7-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
discard b37fadf139 Update RELEASE_NOTES.rst
new af98c60554 Fix mistakenly used "release-mangement build-docs" command (#33195)
new 39e590c04e Add graph screenshots in setup-teardown howto doc (#33244)
new 9c36dd0b9e Update RELEASE_NOTES.rst
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (b37fadf139)
\
N -- N -- N refs/heads/v2-7-test (9c36dd0b9e)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../publish_provider_documentation.sh | 3 +-
docs/apache-airflow/howto/setup-and-teardown.rst | 71 ++++++++++++++++-----
docs/apache-airflow/img/setup-teardown-complex.png | Bin 0 -> 114476 bytes
docs/apache-airflow/img/setup-teardown-group.png | Bin 0 -> 48789 bytes
docs/apache-airflow/img/setup-teardown-nesting.png | Bin 0 -> 83934 bytes
.../apache-airflow/img/setup-teardown-parallel.png | Bin 0 -> 43571 bytes
docs/apache-airflow/img/setup-teardown-scope.png | Bin 0 -> 62951 bytes
.../img/setup-teardown-setup-group.png | Bin 0 -> 76759 bytes
docs/apache-airflow/img/setup-teardown-simple.png | Bin 0 -> 29606 bytes
9 files changed, 57 insertions(+), 17 deletions(-)
create mode 100644 docs/apache-airflow/img/setup-teardown-complex.png
create mode 100644 docs/apache-airflow/img/setup-teardown-group.png
create mode 100644 docs/apache-airflow/img/setup-teardown-nesting.png
create mode 100644 docs/apache-airflow/img/setup-teardown-parallel.png
create mode 100644 docs/apache-airflow/img/setup-teardown-scope.png
create mode 100644 docs/apache-airflow/img/setup-teardown-setup-group.png
create mode 100644 docs/apache-airflow/img/setup-teardown-simple.png
[airflow] 02/03: Add graph screenshots in setup-teardown howto doc (#33244)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-7-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 39e590c04ed6da076e148572dbc346d6fa5efef5
Author: Daniel Standish <15...@users.noreply.github.com>
AuthorDate: Tue Aug 8 22:46:10 2023 -0700
Add graph screenshots in setup-teardown howto doc (#33244)
(cherry picked from commit d45781fca224074c9e8c298816b7ff4411f9c846)
---
docs/apache-airflow/howto/setup-and-teardown.rst | 71 ++++++++++++++++-----
docs/apache-airflow/img/setup-teardown-complex.png | Bin 0 -> 114476 bytes
docs/apache-airflow/img/setup-teardown-group.png | Bin 0 -> 48789 bytes
docs/apache-airflow/img/setup-teardown-nesting.png | Bin 0 -> 83934 bytes
.../apache-airflow/img/setup-teardown-parallel.png | Bin 0 -> 43571 bytes
docs/apache-airflow/img/setup-teardown-scope.png | Bin 0 -> 62951 bytes
.../img/setup-teardown-setup-group.png | Bin 0 -> 76759 bytes
docs/apache-airflow/img/setup-teardown-simple.png | Bin 0 -> 29606 bytes
8 files changed, 56 insertions(+), 15 deletions(-)
diff --git a/docs/apache-airflow/howto/setup-and-teardown.rst b/docs/apache-airflow/howto/setup-and-teardown.rst
index 08eb4157ef..355442a299 100644
--- a/docs/apache-airflow/howto/setup-and-teardown.rst
+++ b/docs/apache-airflow/howto/setup-and-teardown.rst
@@ -52,6 +52,10 @@ For convenience we can do this in one line by passing ``create_cluster`` to the
create_cluster >> run_query >> delete_cluster.as_teardown(setups=create_cluster)
+Here's the graph for this dag:
+
+.. image:: ../img/setup-teardown-simple.png
+
Observations:
* If you clear ``run_query`` to run it again, then both ``create_cluster`` and ``delete_cluster`` will be cleared.
@@ -62,12 +66,16 @@ Additionally, if we have multiple tasks to wrap, we can use the teardown as a co
.. code-block:: python
- with delete_cluster.as_teardown(setups=create_cluster):
+ with delete_cluster().as_teardown(setups=create_cluster()):
[RunQueryOne(), RunQueryTwo()] >> DoSomeOtherStuff()
WorkOne() >> [do_this_stuff(), do_other_stuff()]
This will set create_cluster to run before the tasks in the context, and delete_cluster after them.
+Here it is, shown in the graph:
+
+.. image:: ../img/setup-teardown-complex.png
+
Note that if you are attempting to add an already-instantiated task to a setup context you need to do it explicitly:
.. code-block:: python
@@ -87,6 +95,10 @@ Let's look at an example:
s1 >> w1 >> w2 >> t1.as_teardown(setups=s1) >> w3
w2 >> w4
+And the graph:
+
+.. image:: ../img/setup-teardown-scope.png
+
In the above example, ``w1`` and ``w2`` are "between" ``s1`` and ``t1`` and therefore are assumed to require ``s1``. Thus if ``w1`` or ``w2`` is cleared, so too will be ``s1`` and ``t1``. But if ``w3`` or ``w4`` is cleared, neither ``s1`` nor ``t1`` will be cleared.
You can have multiple setup tasks wired to a single teardown. The teardown will run if at least one of the setups completed successfully.
@@ -97,7 +109,7 @@ You can have a setup without a teardown:
create_cluster >> run_query >> other_task
-In this case, everything downstream of create_cluster is assumed to require it. So if you clear query_two, it will also clear create_cluster. Suppose that we add a teardown for create_cluster after run_query:
+In this case, everything downstream of create_cluster is assumed to require it. So if you clear other_task, it will also clear create_cluster. Suppose that we add a teardown for create_cluster after run_query:
.. code-block:: python
@@ -119,7 +131,7 @@ Controlling dag run state
Another feature of setup / teardown tasks is you can choose whether or not the teardown task should have an impact on dag run state. Perhaps you don't care if the "cleanup" work performed by your teardown task fails, and you only consider the dag run a failure if the "work" tasks fail. By default, teardown tasks are not considered for dag run state.
-Continuing with the example above, if you want the run's success to depend on ``delete_cluster``, then set``on_failure_fail_dagrun=True`` when setting ``delete_cluster`` as teardown. For example:
+Continuing with the example above, if you want the run's success to depend on ``delete_cluster``, then set ``on_failure_fail_dagrun=True`` when setting ``delete_cluster`` as teardown. For example:
.. code-block:: python
@@ -128,20 +140,24 @@ Continuing with the example above, if you want the run's success to depend on ``
Authoring with task groups
""""""""""""""""""""""""""
-When arrowing from task group to task group, or from task group to task, we ignore teardowns. This allows teardowns to run in parallel, and allows dag execution to proceed even if teardown tasks fail.
+When arrowing from task group to task group, or from task group to *task*, we ignore teardowns. This allows teardowns to run in parallel, and allows dag execution to proceed even if teardown tasks fail.
Consider this example:
.. code-block:: python
with TaskGroup("my_group") as tg:
- s1 = my_setup()
- w1 = my_work()
- t1 = my_teardown()
+ s1 = s1()
+ w1 = w1()
+ t1 = t1()
s1 >> w1 >> t1.as_teardown(setups=s1)
- w2 = other_work()
+ w2 = w2()
tg >> w2
+Graph:
+
+.. image:: ../img/setup-teardown-group.png
+
If ``t1`` were not a teardown task, then this dag would effectively be ``s1 >> w1 >> t1 >> w2``. But since we have marked ``t1`` as a teardown, it's ignored in ``tg >> w2``. So the dag is equivalent to the following:
.. code-block:: python
@@ -153,15 +169,19 @@ Now let's consider an example with nesting:
.. code-block:: python
with TaskGroup("my_group") as tg:
- s1 = my_setup()
- w1 = my_work()
- t1 = my_teardown()
+ s1 = s1()
+ w1 = w1()
+ t1 = t1()
s1 >> w1 >> t1.as_teardown(setups=s1)
- w2 = other_work()
+ w2 = w2()
tg >> w2
- dag_s1 = dag_setup1()
- dag_t1 = dag_teardown1()
- dag_s1 >> [tg, w2] >> dag_t1.as_teardown(dag_s1)
+ dag_s1 = dag_s1()
+ dag_t1 = dag_t1()
+ dag_s1 >> [tg, w2] >> dag_t1.as_teardown(setups=dag_s1)
+
+Graph:
+
+.. image:: ../img/setup-teardown-nesting.png
In this example ``s1`` is downstream of ``dag_s1``, so it must wait for ``dag_s1`` to complete successfully. But ``t1`` and ``dag_t1`` can run concurrently, because ``t1`` is ignored in the expression ``tg >> dag_t1``. If you clear ``w2``, it will clear ``dag_s1`` and ``dag_t1``, but not anything in the task group.
@@ -178,6 +198,27 @@ You can run setup tasks in parallel:
>> [delete_cluster.as_teardown(setups=create_cluster), delete_bucket.as_teardown(setups=create_bucket)]
)
+Graph:
+
+.. image:: ../img/setup-teardown-parallel.png
+
+It can be nice visually to put them in a group:
+
+.. code-block:: python
+
+ with TaskGroup("setup") as tg_s:
+ create_cluster = create_cluster()
+ create_bucket = create_bucket()
+ run_query = run_query()
+ with TaskGroup("teardown") as tg_t:
+ delete_cluster = delete_cluster().as_teardown(setups=create_cluster)
+ delete_bucket = delete_bucket().as_teardown(setups=create_bucket)
+ tg_s >> run_query >> tg_t
+
+And the graph:
+
+.. image:: ../img/setup-teardown-setup-group.png
+
Trigger rule behavior for teardowns
"""""""""""""""""""""""""""""""""""
diff --git a/docs/apache-airflow/img/setup-teardown-complex.png b/docs/apache-airflow/img/setup-teardown-complex.png
new file mode 100644
index 0000000000..dbc19de2a5
Binary files /dev/null and b/docs/apache-airflow/img/setup-teardown-complex.png differ
diff --git a/docs/apache-airflow/img/setup-teardown-group.png b/docs/apache-airflow/img/setup-teardown-group.png
new file mode 100644
index 0000000000..a298b0dffd
Binary files /dev/null and b/docs/apache-airflow/img/setup-teardown-group.png differ
diff --git a/docs/apache-airflow/img/setup-teardown-nesting.png b/docs/apache-airflow/img/setup-teardown-nesting.png
new file mode 100644
index 0000000000..17338236ca
Binary files /dev/null and b/docs/apache-airflow/img/setup-teardown-nesting.png differ
diff --git a/docs/apache-airflow/img/setup-teardown-parallel.png b/docs/apache-airflow/img/setup-teardown-parallel.png
new file mode 100644
index 0000000000..7a55e64a41
Binary files /dev/null and b/docs/apache-airflow/img/setup-teardown-parallel.png differ
diff --git a/docs/apache-airflow/img/setup-teardown-scope.png b/docs/apache-airflow/img/setup-teardown-scope.png
new file mode 100644
index 0000000000..d57af239d4
Binary files /dev/null and b/docs/apache-airflow/img/setup-teardown-scope.png differ
diff --git a/docs/apache-airflow/img/setup-teardown-setup-group.png b/docs/apache-airflow/img/setup-teardown-setup-group.png
new file mode 100644
index 0000000000..87a5d150c3
Binary files /dev/null and b/docs/apache-airflow/img/setup-teardown-setup-group.png differ
diff --git a/docs/apache-airflow/img/setup-teardown-simple.png b/docs/apache-airflow/img/setup-teardown-simple.png
new file mode 100644
index 0000000000..1c44217f81
Binary files /dev/null and b/docs/apache-airflow/img/setup-teardown-simple.png differ
[airflow] 03/03: Update RELEASE_NOTES.rst
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-7-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 9c36dd0b9ec224e6916200af5e411769b539fb92
Author: Ephraim Anierobi <sp...@gmail.com>
AuthorDate: Wed Aug 9 07:55:58 2023 +0100
Update RELEASE_NOTES.rst
---
RELEASE_NOTES.rst | 22 ++++++++++++++++++++--
1 file changed, 20 insertions(+), 2 deletions(-)
diff --git a/RELEASE_NOTES.rst b/RELEASE_NOTES.rst
index 24c927764f..c15e11c01f 100644
--- a/RELEASE_NOTES.rst
+++ b/RELEASE_NOTES.rst
@@ -165,11 +165,14 @@ New Features
Improvements
""""""""""""
+- Configurable health check threshold for triggerer (#33089)
+- Add documentation generation for CLI commands from executors (#33081, #33143, #33081)
+- add dag_run_ids and task_ids filter for the batch task instance API endpoint (#32705)
- Ensure DAG-level references are filled on unmap (#33083)
- Add support for arrays of different data types in the Trigger Form UI (#32734)
- Always show gantt and code tabs (#33029)
- Move listener success hook to after SQLAlchemy commit (#32988)
-- Rename ``db upgrade`` to ``db migrate`` and add ``connections create-default-connections`` (#32810)
+- Rename ``db upgrade`` to ``db migrate`` and add ``connections create-default-connections`` (#32810, #33136)
- Remove old gantt chart and redirect to grid views gantt tab (#32908)
- Adjust graph zoom based on selected task (#32792)
- Call listener on_task_instance_running after rendering templates (#32716)
@@ -214,6 +217,12 @@ Improvements
Bug Fixes
"""""""""
+- Fix ``virtualenv`` detection for Python ``virtualenv`` operator (#33223)
+- Correctly log when there are problems trying to ``chmod`` ``airflow.cfg`` (#33118)
+- Pass app context to webserver_config.py (#32759)
+- Skip served logs for non-running task try (#32561)
+- Fix reload gunicorn workers (#32102)
+- Fix future DagRun rarely triggered by race conditions when ``max_active_runs`` reached its upper limit. (#31414)
- Fix BaseOperator ``get_task_instances`` query (#33054)
- Fix issue with using the various state enum value in logs (#33065)
- Use string concatenation to prepend base URL for log_url (#33063)
@@ -262,6 +271,14 @@ Bug Fixes
Misc/Internal
"""""""""""""
+- For now limit ``Pydantic`` to ``< 2.0.0`` (#33235)
+- Refactor: Simplify code in models (#33181)
+- Add elasticsearch group to pre-2.7 defaults (#33166)
+- Refactor: Simplify dict manipulation in airflow/cli (#33159)
+- Remove redundant dict.keys() call (#33158)
+- Upgrade ruff to latest 0.0.282 version in pre-commits (#33152)
+- Move openlineage configuration to provider (#33124)
+- Replace State by TaskInstanceState in Airflow executors (#32627)
- Get rid of Python 2 numeric relics (#33050)
- Remove legacy dag code (#33058)
- Remove legacy task instance modal (#33060)
@@ -304,7 +321,8 @@ Misc/Internal
Docs only changes
"""""""""""""""""
-- Documentation change to enhance readability (#33006)
+- Add links to ``DAGRun / DAG / Task`` in templates-ref.rst (#33013)
+- Add example for list-import-errors as a CI check (#32811)
- Clean-up of our new security page (#32951)
- Cleans up Extras reference page (#32954)
- Update Dag trigger API and command docs (#32696)
[airflow] 01/03: Fix mistakenly used "release-mangement build-docs" command (#33195)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-7-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit af98c605540e5b9f654e1102e46ccff5d9499462
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Tue Aug 8 11:00:27 2023 +0200
Fix mistakenly used "release-mangement build-docs" command (#33195)
The #32495 mistakeny replaced the "build" script with a
"release-management build-docs" command where there is no such
command - there is just "build-docs"
(cherry picked from commit 199c604d3841797b97bd4b813b36379910a532a8)
---
dev/provider_packages/publish_provider_documentation.sh | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/dev/provider_packages/publish_provider_documentation.sh b/dev/provider_packages/publish_provider_documentation.sh
index cd7ef4433c..bf8eaca1fb 100755
--- a/dev/provider_packages/publish_provider_documentation.sh
+++ b/dev/provider_packages/publish_provider_documentation.sh
@@ -28,7 +28,6 @@ do
provider_filters+=("--package-filter" "apache-airflow-providers-${provider//./-}")
done
-breeze release-management build-docs \
- "${provider_filters[@]}"
+breeze build-docs "${provider_filters[@]}"
cd "${AIRFLOW_SITE_DIRECTORY}"