You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/04/14 15:21:38 UTC
[GitHub] [airflow] josh-fell commented on a diff in pull request #22941: Allow re-use of decorated tasks
josh-fell commented on code in PR #22941:
URL: https://github.com/apache/airflow/pull/22941#discussion_r850544082
##########
docs/apache-airflow/tutorial_taskflow_api.rst:
##########
@@ -160,6 +160,65 @@ the dependencies as shown below.
:start-after: [START main_flow]
:end-before: [END main_flow]
+
+Re-using a decorated task
+-------------------------
+
+Decorated tasks are flexible. You can re-use a decorated task in multiple DAGs, overriding the operator
+parameters such as the ``task_id``, ``queue``, ``pool``, etc, as well as the task's arguments.
+
+Below is an example of how you can re-use a decorated task in multiple DAGs:
Review Comment:
```suggestion
Reusing a decorated task
-------------------------
Decorated tasks are flexible. You can reuse a decorated task in multiple DAGs, overriding the operator
parameters such as the ``task_id``, ``queue``, ``pool``, etc., as well as the task's arguments.
Below is an example of how you can reuse a decorated task in multiple DAGs:
```
##########
docs/apache-airflow/tutorial_taskflow_api.rst:
##########
@@ -160,6 +160,65 @@ the dependencies as shown below.
:start-after: [START main_flow]
:end-before: [END main_flow]
+
+Re-using a decorated task
+-------------------------
+
+Decorated tasks are flexible. You can re-use a decorated task in multiple DAGs, overriding the operator
+parameters such as the ``task_id``, ``queue``, ``pool``, etc, as well as the task's arguments.
+
+Below is an example of how you can re-use a decorated task in multiple DAGs:
+
+.. code-block:: python
+ from airflow.decorators import task, dag
+ from datetime import datetime
+
+
+ @task
+ def add_task(x, y):
+ print(f"Task args: x={x}, y={y}")
+ return x + y
+
+
+ @dag(start_date=datetime(2022, 1, 1))
+ def mydag():
+ start = add_task.override(task_id="start")(1, 2)
+ for i in range(3):
+ start >> add_task.override(task_id=f"add_start_{i}")(start, i)
+
+
+ @dag(start_date=datetime(2022, 1, 1))
+ def mydag2():
+ start = add_task(1, 2)
+ for i in range(3):
+ start >> add_task.override(task_id=f"new_add_task_{i}")(start, i)
+
+
+ first_dag = mydag()
+ second_dag = mydag2()
+
+You can also import the above ``add_task`` and use it in another DAG file.
+Suppose the code above lives in a file called ``common.py``. You can do this:
+
+.. code-block:: python
+
+ from common import add_task
+ from airflow.decorators import dag
+ from datetime import datetime
+
+
+ @dag(start_date=datetime(2022, 1, 1))
+ def use_add_task():
+ start = add_task.override(priority_weight=3)(1, 2)
+ for i in range(3):
+ start >> add_task.override(task_id=f"new_add_task_{i}", pool="default_pool")(
Review Comment:
Should we use another `pool` value that's not the default for an override?
##########
docs/apache-airflow/tutorial_taskflow_api.rst:
##########
@@ -160,6 +160,65 @@ the dependencies as shown below.
:start-after: [START main_flow]
:end-before: [END main_flow]
+
+Re-using a decorated task
+-------------------------
+
+Decorated tasks are flexible. You can re-use a decorated task in multiple DAGs, overriding the operator
+parameters such as the ``task_id``, ``queue``, ``pool``, etc, as well as the task's arguments.
+
+Below is an example of how you can re-use a decorated task in multiple DAGs:
Review Comment:
Should we link to the [list of operator params](https://airflow.apache.org/docs/apache-airflow/stable/tutorial.html#default-arguments) we have in the docs so users have some more reference to what can be overwritten albeit not exhaustive? WDYT?
##########
docs/apache-airflow/tutorial_taskflow_api.rst:
##########
@@ -160,6 +160,65 @@ the dependencies as shown below.
:start-after: [START main_flow]
:end-before: [END main_flow]
+
+Re-using a decorated task
+-------------------------
+
+Decorated tasks are flexible. You can re-use a decorated task in multiple DAGs, overriding the operator
+parameters such as the ``task_id``, ``queue``, ``pool``, etc, as well as the task's arguments.
+
+Below is an example of how you can re-use a decorated task in multiple DAGs:
+
+.. code-block:: python
+ from airflow.decorators import task, dag
+ from datetime import datetime
+
+
+ @task
+ def add_task(x, y):
+ print(f"Task args: x={x}, y={y}")
+ return x + y
+
+
+ @dag(start_date=datetime(2022, 1, 1))
+ def mydag():
+ start = add_task.override(task_id="start")(1, 2)
+ for i in range(3):
+ start >> add_task.override(task_id=f"add_start_{i}")(start, i)
+
+
+ @dag(start_date=datetime(2022, 1, 1))
+ def mydag2():
+ start = add_task(1, 2)
+ for i in range(3):
+ start >> add_task.override(task_id=f"new_add_task_{i}")(start, i)
+
+
+ first_dag = mydag()
+ second_dag = mydag2()
+
+You can also import the above ``add_task`` and use it in another DAG file.
+Suppose the code above lives in a file called ``common.py``. You can do this:
Review Comment:
```suggestion
Suppose the ``add_task`` code lives in a file called ``common.py``. You can do this:
```
##########
docs/apache-airflow/tutorial_taskflow_api.rst:
##########
@@ -160,6 +160,65 @@ the dependencies as shown below.
:start-after: [START main_flow]
:end-before: [END main_flow]
+
+Re-using a decorated task
+-------------------------
+
+Decorated tasks are flexible. You can re-use a decorated task in multiple DAGs, overriding the operator
+parameters such as the ``task_id``, ``queue``, ``pool``, etc, as well as the task's arguments.
+
+Below is an example of how you can re-use a decorated task in multiple DAGs:
Review Comment:
>as well as the task's arguments.
We're just overriding the operator parameters, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org