You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ds...@apache.org on 2023/08/09 05:46:17 UTC
[airflow] branch main updated: Add graph screenshots in setup-teardown howto doc (#33244)
This is an automated email from the ASF dual-hosted git repository.
dstandish pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new d45781fca2 Add graph screenshots in setup-teardown howto doc (#33244)
d45781fca2 is described below
commit d45781fca224074c9e8c298816b7ff4411f9c846
Author: Daniel Standish <15...@users.noreply.github.com>
AuthorDate: Tue Aug 8 22:46:10 2023 -0700
Add graph screenshots in setup-teardown howto doc (#33244)
---
docs/apache-airflow/howto/setup-and-teardown.rst | 71 ++++++++++++++++-----
docs/apache-airflow/img/setup-teardown-complex.png | Bin 0 -> 114476 bytes
docs/apache-airflow/img/setup-teardown-group.png | Bin 0 -> 48789 bytes
docs/apache-airflow/img/setup-teardown-nesting.png | Bin 0 -> 83934 bytes
.../apache-airflow/img/setup-teardown-parallel.png | Bin 0 -> 43571 bytes
docs/apache-airflow/img/setup-teardown-scope.png | Bin 0 -> 62951 bytes
.../img/setup-teardown-setup-group.png | Bin 0 -> 76759 bytes
docs/apache-airflow/img/setup-teardown-simple.png | Bin 0 -> 29606 bytes
8 files changed, 56 insertions(+), 15 deletions(-)
diff --git a/docs/apache-airflow/howto/setup-and-teardown.rst b/docs/apache-airflow/howto/setup-and-teardown.rst
index 08eb4157ef..355442a299 100644
--- a/docs/apache-airflow/howto/setup-and-teardown.rst
+++ b/docs/apache-airflow/howto/setup-and-teardown.rst
@@ -52,6 +52,10 @@ For convenience we can do this in one line by passing ``create_cluster`` to the
create_cluster >> run_query >> delete_cluster.as_teardown(setups=create_cluster)
+Here's the graph for this dag:
+
+.. image:: ../img/setup-teardown-simple.png
+
Observations:
* If you clear ``run_query`` to run it again, then both ``create_cluster`` and ``delete_cluster`` will be cleared.
@@ -62,12 +66,16 @@ Additionally, if we have multiple tasks to wrap, we can use the teardown as a co
.. code-block:: python
- with delete_cluster.as_teardown(setups=create_cluster):
+ with delete_cluster().as_teardown(setups=create_cluster()):
[RunQueryOne(), RunQueryTwo()] >> DoSomeOtherStuff()
WorkOne() >> [do_this_stuff(), do_other_stuff()]
This will set create_cluster to run before the tasks in the context, and delete_cluster after them.
+Here it is, shown in the graph:
+
+.. image:: ../img/setup-teardown-complex.png
+
Note that if you are attempting to add an already-instantiated task to a setup context you need to do it explicitly:
.. code-block:: python
@@ -87,6 +95,10 @@ Let's look at an example:
s1 >> w1 >> w2 >> t1.as_teardown(setups=s1) >> w3
w2 >> w4
+And the graph:
+
+.. image:: ../img/setup-teardown-scope.png
+
In the above example, ``w1`` and ``w2`` are "between" ``s1`` and ``t1`` and therefore are assumed to require ``s1``. Thus if ``w1`` or ``w2`` is cleared, so too will be ``s1`` and ``t1``. But if ``w3`` or ``w4`` is cleared, neither ``s1`` nor ``t1`` will be cleared.
You can have multiple setup tasks wired to a single teardown. The teardown will run if at least one of the setups completed successfully.
@@ -97,7 +109,7 @@ You can have a setup without a teardown:
create_cluster >> run_query >> other_task
-In this case, everything downstream of create_cluster is assumed to require it. So if you clear query_two, it will also clear create_cluster. Suppose that we add a teardown for create_cluster after run_query:
+In this case, everything downstream of create_cluster is assumed to require it. So if you clear other_task, it will also clear create_cluster. Suppose that we add a teardown for create_cluster after run_query:
.. code-block:: python
@@ -119,7 +131,7 @@ Controlling dag run state
Another feature of setup / teardown tasks is you can choose whether or not the teardown task should have an impact on dag run state. Perhaps you don't care if the "cleanup" work performed by your teardown task fails, and you only consider the dag run a failure if the "work" tasks fail. By default, teardown tasks are not considered for dag run state.
-Continuing with the example above, if you want the run's success to depend on ``delete_cluster``, then set``on_failure_fail_dagrun=True`` when setting ``delete_cluster`` as teardown. For example:
+Continuing with the example above, if you want the run's success to depend on ``delete_cluster``, then set ``on_failure_fail_dagrun=True`` when setting ``delete_cluster`` as teardown. For example:
.. code-block:: python
@@ -128,20 +140,24 @@ Continuing with the example above, if you want the run's success to depend on ``
Authoring with task groups
""""""""""""""""""""""""""
-When arrowing from task group to task group, or from task group to task, we ignore teardowns. This allows teardowns to run in parallel, and allows dag execution to proceed even if teardown tasks fail.
+When arrowing from task group to task group, or from task group to *task*, we ignore teardowns. This allows teardowns to run in parallel, and allows dag execution to proceed even if teardown tasks fail.
Consider this example:
.. code-block:: python
with TaskGroup("my_group") as tg:
- s1 = my_setup()
- w1 = my_work()
- t1 = my_teardown()
+ s1 = s1()
+ w1 = w1()
+ t1 = t1()
s1 >> w1 >> t1.as_teardown(setups=s1)
- w2 = other_work()
+ w2 = w2()
tg >> w2
+Graph:
+
+.. image:: ../img/setup-teardown-group.png
+
If ``t1`` were not a teardown task, then this dag would effectively be ``s1 >> w1 >> t1 >> w2``. But since we have marked ``t1`` as a teardown, it's ignored in ``tg >> w2``. So the dag is equivalent to the following:
.. code-block:: python
@@ -153,15 +169,19 @@ Now let's consider an example with nesting:
.. code-block:: python
with TaskGroup("my_group") as tg:
- s1 = my_setup()
- w1 = my_work()
- t1 = my_teardown()
+ s1 = s1()
+ w1 = w1()
+ t1 = t1()
s1 >> w1 >> t1.as_teardown(setups=s1)
- w2 = other_work()
+ w2 = w2()
tg >> w2
- dag_s1 = dag_setup1()
- dag_t1 = dag_teardown1()
- dag_s1 >> [tg, w2] >> dag_t1.as_teardown(dag_s1)
+ dag_s1 = dag_s1()
+ dag_t1 = dag_t1()
+ dag_s1 >> [tg, w2] >> dag_t1.as_teardown(setups=dag_s1)
+
+Graph:
+
+.. image:: ../img/setup-teardown-nesting.png
In this example ``s1`` is downstream of ``dag_s1``, so it must wait for ``dag_s1`` to complete successfully. But ``t1`` and ``dag_t1`` can run concurrently, because ``t1`` is ignored in the expression ``tg >> dag_t1``. If you clear ``w2``, it will clear ``dag_s1`` and ``dag_t1``, but not anything in the task group.
@@ -178,6 +198,27 @@ You can run setup tasks in parallel:
>> [delete_cluster.as_teardown(setups=create_cluster), delete_bucket.as_teardown(setups=create_bucket)]
)
+Graph:
+
+.. image:: ../img/setup-teardown-parallel.png
+
+It can be nice visually to put them in a group:
+
+.. code-block:: python
+
+ with TaskGroup("setup") as tg_s:
+ create_cluster = create_cluster()
+ create_bucket = create_bucket()
+ run_query = run_query()
+ with TaskGroup("teardown") as tg_t:
+ delete_cluster = delete_cluster().as_teardown(setups=create_cluster)
+ delete_bucket = delete_bucket().as_teardown(setups=create_bucket)
+ tg_s >> run_query >> tg_t
+
+And the graph:
+
+.. image:: ../img/setup-teardown-setup-group.png
+
Trigger rule behavior for teardowns
"""""""""""""""""""""""""""""""""""
diff --git a/docs/apache-airflow/img/setup-teardown-complex.png b/docs/apache-airflow/img/setup-teardown-complex.png
new file mode 100644
index 0000000000..dbc19de2a5
Binary files /dev/null and b/docs/apache-airflow/img/setup-teardown-complex.png differ
diff --git a/docs/apache-airflow/img/setup-teardown-group.png b/docs/apache-airflow/img/setup-teardown-group.png
new file mode 100644
index 0000000000..a298b0dffd
Binary files /dev/null and b/docs/apache-airflow/img/setup-teardown-group.png differ
diff --git a/docs/apache-airflow/img/setup-teardown-nesting.png b/docs/apache-airflow/img/setup-teardown-nesting.png
new file mode 100644
index 0000000000..17338236ca
Binary files /dev/null and b/docs/apache-airflow/img/setup-teardown-nesting.png differ
diff --git a/docs/apache-airflow/img/setup-teardown-parallel.png b/docs/apache-airflow/img/setup-teardown-parallel.png
new file mode 100644
index 0000000000..7a55e64a41
Binary files /dev/null and b/docs/apache-airflow/img/setup-teardown-parallel.png differ
diff --git a/docs/apache-airflow/img/setup-teardown-scope.png b/docs/apache-airflow/img/setup-teardown-scope.png
new file mode 100644
index 0000000000..d57af239d4
Binary files /dev/null and b/docs/apache-airflow/img/setup-teardown-scope.png differ
diff --git a/docs/apache-airflow/img/setup-teardown-setup-group.png b/docs/apache-airflow/img/setup-teardown-setup-group.png
new file mode 100644
index 0000000000..87a5d150c3
Binary files /dev/null and b/docs/apache-airflow/img/setup-teardown-setup-group.png differ
diff --git a/docs/apache-airflow/img/setup-teardown-simple.png b/docs/apache-airflow/img/setup-teardown-simple.png
new file mode 100644
index 0000000000..1c44217f81
Binary files /dev/null and b/docs/apache-airflow/img/setup-teardown-simple.png differ