You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/10/26 02:06:37 UTC
[GitHub] [airflow] josh-fell opened a new pull request #19214: Expanding ``.output`` operator property information in TaskFlow tutorial doc
josh-fell opened a new pull request #19214:
URL: https://github.com/apache/airflow/pull/19214
Closes: #9415
Adding more color and information about the ``.output`` property of operators in the TaskFlow tutorial doc. There are also small changes to keep spelling and descriptions consistent.
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on a change in pull request #19214: Expanding ``.output`` operator property information in TaskFlow tutorial doc
Posted by GitBox <gi...@apache.org>.
josh-fell commented on a change in pull request #19214:
URL: https://github.com/apache/airflow/pull/19214#discussion_r736515521
##########
File path: docs/apache-airflow/tutorial_taskflow_api.rst
##########
@@ -279,18 +302,43 @@ Building this dependency is shown in the code below:
return json.loads(api_results)
- parsed_results = parsed_results(get_api_results_task.output)
+ parsed_results = parsed_results(api_results=get_api_results_task.output)
+
+Not only can you use traditional operator outputs as inputs for TaskFlow functions, but also as inputs to
+other traditional operators. In the example below, the output from the :class:`~airflow.providers.amazon.aws.transfers.salesforce_to_s3.SalesforceToS3Operator`
+task (which is an S3 URI for a destination file location) is used an input for the :class:`~airflow.providers.amazon.aws.operators.s3_copy_object.S3CopyObjectOperator`
+task to copy the same file to a date-partitioned storage location in S3 for long-term storage in a data lake.
Review comment:
IMO the TaskFlow API (I guess more broadly AIP-31) is about a more functional DAG authoring style + passing data between tasks via `XComArgs`. The `.output` property is a means to access `XComArgs` for TaskFlow functions and traditional tasks alike. Separating this example to another place in the documentation might be a little disjointed with providing this information. Having answers for "what can I do with the TaskFlow API and a functional DAG style?"-type questions in this doc seems useful.
Although I'm not opposed to adding this information to the `XComs` concepts page as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a change in pull request #19214: Expanding ``.output`` operator property information in TaskFlow tutorial doc
Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #19214:
URL: https://github.com/apache/airflow/pull/19214#discussion_r736126450
##########
File path: docs/apache-airflow/tutorial_taskflow_api.rst
##########
@@ -224,11 +224,11 @@ is automatically set to true.
Note, If you manually set the ``multiple_outputs`` parameter the inference is disabled and
the parameter value is used.
-Adding dependencies to decorated tasks from regular tasks
----------------------------------------------------------
-The above tutorial shows how to create dependencies between python-based tasks. However, it is
-quite possible while writing a DAG to have some pre-existing tasks such as :class:`~airflow.operators.bash.BashOperator` or :class:`~airflow.sensors.filesystem.FileSensor`
-based tasks which need to be run first before a python-based task is run.
+Adding dependencies to decorated tasks from traditional tasks
Review comment:
```suggestion
Adding dependencies between decorated tasks and traditional tasks
```
the dependency is mutual not exactly _from_ one _to_ another
##########
File path: docs/apache-airflow/tutorial_taskflow_api.rst
##########
@@ -224,11 +224,11 @@ is automatically set to true.
Note, If you manually set the ``multiple_outputs`` parameter the inference is disabled and
the parameter value is used.
-Adding dependencies to decorated tasks from regular tasks
----------------------------------------------------------
-The above tutorial shows how to create dependencies between python-based tasks. However, it is
-quite possible while writing a DAG to have some pre-existing tasks such as :class:`~airflow.operators.bash.BashOperator` or :class:`~airflow.sensors.filesystem.FileSensor`
-based tasks which need to be run first before a python-based task is run.
+Adding dependencies to decorated tasks from traditional tasks
+-------------------------------------------------------------
+The above tutorial shows how to create dependencies between TaskFlow functions. However, it is
+quite possible while writing a DAG to have some traditional tasks (such as :class:`~airflow.operators.bash.BashOperator` or :class:`~airflow.sensors.filesystem.FileSensor`)
+which need to be run first before a TaskFlow function is run.
Review comment:
```suggestion
TaskFlow functions may be combined with traditional tasks in a dag.
```
This may be venturing into nit pickery here, but seems a bit verbose
##########
File path: docs/apache-airflow/tutorial_taskflow_api.rst
##########
@@ -251,18 +251,41 @@ Building this dependency is shown in the code below:
file_task >> order_data
-In the above code block, a new python-based task is defined as ``extract_from_file`` which
+In the above code block, a new TaskFlow function is defined as ``extract_from_file`` which
reads the data from a known file location.
In the main DAG, a new ``FileSensor`` task is defined to check for this file. Please note
that this is a Sensor task which waits for the file.
-Finally, a dependency between this Sensor task and the python-based task is specified.
+Finally, a dependency between this Sensor task and the TaskFlow function is specified.
-Consuming XCOMs with decorated tasks from regular tasks
----------------------------------------------------------
-You may additionally find it necessary to consume an XCOM from a pre-existing task as an input into python-based tasks.
+Consuming XComs from traditional tasks as inputs
+------------------------------------------------
+You may find it necessary to consume an XCom from traditional task, either pushed within the task's execution
+or via its return value, as an input into downstream tasks. You can access the pushed XCom (also known as an
+``XComArg``) by simply utilizing the ``.output`` property exposed for all operators.
Review comment:
```suggestion
``XComArg``) with the ``.output`` property exposed for all operators.
```
##########
File path: docs/apache-airflow/tutorial_taskflow_api.rst
##########
@@ -251,18 +251,41 @@ Building this dependency is shown in the code below:
file_task >> order_data
-In the above code block, a new python-based task is defined as ``extract_from_file`` which
+In the above code block, a new TaskFlow function is defined as ``extract_from_file`` which
reads the data from a known file location.
In the main DAG, a new ``FileSensor`` task is defined to check for this file. Please note
that this is a Sensor task which waits for the file.
-Finally, a dependency between this Sensor task and the python-based task is specified.
+Finally, a dependency between this Sensor task and the TaskFlow function is specified.
-Consuming XCOMs with decorated tasks from regular tasks
----------------------------------------------------------
-You may additionally find it necessary to consume an XCOM from a pre-existing task as an input into python-based tasks.
+Consuming XComs from traditional tasks as inputs
+------------------------------------------------
+You may find it necessary to consume an XCom from traditional task, either pushed within the task's execution
+or via its return value, as an input into downstream tasks. You can access the pushed XCom (also known as an
+``XComArg``) by simply utilizing the ``.output`` property exposed for all operators.
-Building this dependency is shown in the code below:
+By default, using the ``.output`` property to retrieve an XCom result is the equivalent of:
+
+.. code-block:: python
+
+ task_instance.xcom_pull(task_ids="my_task_id", key="return_value")
+
+To retrieve an XCom result for a key other than ``return_value``, you can use:
+
+.. code-block:: python
+
+ my_op = MyOperator(...)
+ my_op_output = my_op.output["some_other_xcom_key"]
+ # OR
+ my_op_output = my_op.output.get("some_other_xcom_key")
+
+.. note::
+ Using the ``.output`` property as an input to another task is supported only for operator parameters
+ listed as a ``template_field``.
+
+In the code example below, a :class:`~airflow.providers.http.operators.http.SimpleHttpOperator` result
+is captured via :doc:`XComs </concepts/xcoms>`. This XCom result, which is the task output, is then passed
+to a TaskFlow function to parse the response as JSON.
Review comment:
```suggestion
to a TaskFlow function which parses the response as JSON.
```
##########
File path: docs/apache-airflow/tutorial_taskflow_api.rst
##########
@@ -279,18 +302,43 @@ Building this dependency is shown in the code below:
return json.loads(api_results)
- parsed_results = parsed_results(get_api_results_task.output)
+ parsed_results = parsed_results(api_results=get_api_results_task.output)
+
+Not only can you use traditional operator outputs as inputs for TaskFlow functions, but also as inputs to
+other traditional operators. In the example below, the output from the :class:`~airflow.providers.amazon.aws.transfers.salesforce_to_s3.SalesforceToS3Operator`
+task (which is an S3 URI for a destination file location) is used an input for the :class:`~airflow.providers.amazon.aws.operators.s3_copy_object.S3CopyObjectOperator`
+task to copy the same file to a date-partitioned storage location in S3 for long-term storage in a data lake.
Review comment:
does this particular example move out of the realm of TaskFlow and back into simply XCom (in which case maybe it belongs in an xcom doc)? Or is `.output` considered taskflow-specific? I thought taskflow was really about decorated tasks?
##########
File path: docs/apache-airflow/tutorial_taskflow_api.rst
##########
@@ -251,18 +251,41 @@ Building this dependency is shown in the code below:
file_task >> order_data
-In the above code block, a new python-based task is defined as ``extract_from_file`` which
+In the above code block, a new TaskFlow function is defined as ``extract_from_file`` which
reads the data from a known file location.
In the main DAG, a new ``FileSensor`` task is defined to check for this file. Please note
that this is a Sensor task which waits for the file.
-Finally, a dependency between this Sensor task and the python-based task is specified.
+Finally, a dependency between this Sensor task and the TaskFlow function is specified.
-Consuming XCOMs with decorated tasks from regular tasks
----------------------------------------------------------
-You may additionally find it necessary to consume an XCOM from a pre-existing task as an input into python-based tasks.
+Consuming XComs from traditional tasks as inputs
+------------------------------------------------
+You may find it necessary to consume an XCom from traditional task, either pushed within the task's execution
+or via its return value, as an input into downstream tasks. You can access the pushed XCom (also known as an
Review comment:
```suggestion
A TaskFlow function can consume an XCom from a traditional task. You can access the pushed XCom (also known as an
```
a little more direct this way.
##########
File path: docs/apache-airflow/tutorial_taskflow_api.rst
##########
@@ -224,11 +224,11 @@ is automatically set to true.
Note, If you manually set the ``multiple_outputs`` parameter the inference is disabled and
the parameter value is used.
-Adding dependencies to decorated tasks from regular tasks
----------------------------------------------------------
-The above tutorial shows how to create dependencies between python-based tasks. However, it is
-quite possible while writing a DAG to have some pre-existing tasks such as :class:`~airflow.operators.bash.BashOperator` or :class:`~airflow.sensors.filesystem.FileSensor`
-based tasks which need to be run first before a python-based task is run.
+Adding dependencies to decorated tasks from traditional tasks
Review comment:
another thought:
```suggestion
Adding dependencies between ``TaskFlow`` tasks and traditional tasks
```
##########
File path: docs/apache-airflow/tutorial_taskflow_api.rst
##########
@@ -224,11 +224,11 @@ is automatically set to true.
Note, If you manually set the ``multiple_outputs`` parameter the inference is disabled and
the parameter value is used.
-Adding dependencies to decorated tasks from regular tasks
----------------------------------------------------------
-The above tutorial shows how to create dependencies between python-based tasks. However, it is
-quite possible while writing a DAG to have some pre-existing tasks such as :class:`~airflow.operators.bash.BashOperator` or :class:`~airflow.sensors.filesystem.FileSensor`
-based tasks which need to be run first before a python-based task is run.
+Adding dependencies to decorated tasks from traditional tasks
+-------------------------------------------------------------
+The above tutorial shows how to create dependencies between TaskFlow functions. However, it is
Review comment:
my suggestion was `TaskFlow functions may be combined with traditional tasks in a dag` replacing the whole paragraph. but anyway this is sort of tangential to your PR just figured since we're looking at it... but i can always followup with language cleanup later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a change in pull request #19214: Expanding ``.output`` operator property information in TaskFlow tutorial doc
Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #19214:
URL: https://github.com/apache/airflow/pull/19214#discussion_r736703287
##########
File path: docs/apache-airflow/tutorial_taskflow_api.rst
##########
@@ -224,11 +224,11 @@ is automatically set to true.
Note, If you manually set the ``multiple_outputs`` parameter the inference is disabled and
the parameter value is used.
-Adding dependencies to decorated tasks from regular tasks
----------------------------------------------------------
-The above tutorial shows how to create dependencies between python-based tasks. However, it is
-quite possible while writing a DAG to have some pre-existing tasks such as :class:`~airflow.operators.bash.BashOperator` or :class:`~airflow.sensors.filesystem.FileSensor`
-based tasks which need to be run first before a python-based task is run.
+Adding dependencies to decorated tasks from traditional tasks
Review comment:
> Although, do we want to use "TaskFlow functions" instead of "TaskFlow tasks" to be consistent?
Fine with me...
And it certainly doesn't need to be resolved here but personally I am not sure how i feel about the language `TaskFlow functions` when we're talking about the tasks they produce. Maybe the only thing we should refer to as a "taskflow function" is the function. And once it is called, the returned value is now a task, and should be referred to as a task. wdyt?
> Yeah, fair. I think the original section heading was chosen because the example is specifically from TaskFlow function to a traditional task but I think folks could understand that's the concept is bidirectional
yeah i mean it totally works either way, (and certainly no dying on hills here either :) )... just some caffeinated thoughts on it...
i guess the reason the language seemed awkward to me is because for example t1 >> t2 is the same as t2 << t1 -- i.e. you could say the "dependency _relationship_" is symmetrical.
but on further reflection i see you could read "dependency" as only one direction -- this task _depends_ on that one, i.e. t1 must run before t2. but still, that particular phrasing still seems to me to be a bit ambiguous. e.g. if you say "there's a dependency from task `hello` to task `hola`" it doesn't really make it clear which one is upstream and which is downstream.
another way one could think of it, the example illustrates how to add dependencies between tasks created in these two ways _in general_. i.e. true, in the example one is upstream of the other but it doesn't really matter because the technique is the same no matter the directionality. so we don't need to describe it as the special case of `traditional >> TF`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a change in pull request #19214: Expanding ``.output`` operator property information in TaskFlow tutorial doc
Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #19214:
URL: https://github.com/apache/airflow/pull/19214#discussion_r736650144
##########
File path: docs/apache-airflow/tutorial_taskflow_api.rst
##########
@@ -251,18 +251,41 @@ Building this dependency is shown in the code below:
file_task >> order_data
-In the above code block, a new python-based task is defined as ``extract_from_file`` which
+In the above code block, a new TaskFlow function is defined as ``extract_from_file`` which
reads the data from a known file location.
In the main DAG, a new ``FileSensor`` task is defined to check for this file. Please note
that this is a Sensor task which waits for the file.
-Finally, a dependency between this Sensor task and the python-based task is specified.
+Finally, a dependency between this Sensor task and the TaskFlow function is specified.
-Consuming XCOMs with decorated tasks from regular tasks
----------------------------------------------------------
-You may additionally find it necessary to consume an XCOM from a pre-existing task as an input into python-based tasks.
+Consuming XComs from traditional tasks as inputs
+------------------------------------------------
+You may find it necessary to consume an XCom from traditional task, either pushed within the task's execution
+or via its return value, as an input into downstream tasks. You can access the pushed XCom (also known as an
Review comment:
> but not a hill I'm going to die on
i enjoy this expression :)
yeah follow your heart!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish merged pull request #19214: Expanding ``.output`` operator property information in TaskFlow tutorial doc
Posted by GitBox <gi...@apache.org>.
dstandish merged pull request #19214:
URL: https://github.com/apache/airflow/pull/19214
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a change in pull request #19214: Expanding ``.output`` operator property information in TaskFlow tutorial doc
Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #19214:
URL: https://github.com/apache/airflow/pull/19214#discussion_r736703287
##########
File path: docs/apache-airflow/tutorial_taskflow_api.rst
##########
@@ -224,11 +224,11 @@ is automatically set to true.
Note, If you manually set the ``multiple_outputs`` parameter the inference is disabled and
the parameter value is used.
-Adding dependencies to decorated tasks from regular tasks
----------------------------------------------------------
-The above tutorial shows how to create dependencies between python-based tasks. However, it is
-quite possible while writing a DAG to have some pre-existing tasks such as :class:`~airflow.operators.bash.BashOperator` or :class:`~airflow.sensors.filesystem.FileSensor`
-based tasks which need to be run first before a python-based task is run.
+Adding dependencies to decorated tasks from traditional tasks
Review comment:
> Although, do we want to use "TaskFlow functions" instead of "TaskFlow tasks" to be consistent?
Personally i am not sure how i feel about the language `TaskFlow functions` because in my mind taskflow is just another way of writing a `task`.
In fact maybe that's it. Maybe the only thing we should refer to as a "taskflow function" is the function. And once it is called, the returned value is now a task, and should be referred to as a task. wdyt?
> Yeah, fair. I think the original section heading was chosen because the example is specifically from TaskFlow function to a traditional task but I think folks could understand that's the concept is bidirectional
yeah i mean it totally works either way, (and certainly no dying on hills here either :) )... just some caffeinated thoughts on it...
i guess the reason the language seemed awkward to me is because for example t1 >> t2 is the same as t2 << t1 -- i.e. you could say the "dependency _relationship_" is symmetrical.
but on further reflection i see you could read "dependency" as only one direction -- this task _depends_ on that one, i.e. t1 must run before t2. but still, that particular phrasing still seems to me to be a bit ambiguous. e.g. if you say "there's a dependency from task `hello` to task `hola`" it doesn't really make it clear which one is upstream and which is downstream.
another way one could think of it, the example illustrates how to add dependencies between tasks created in these two ways _in general_. i.e. true, in the example one is upstream of the other but it doesn't really matter because the technique is the same no matter the directionality. so we don't need to describe it as the special case of `traditional >> TF`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19214: Expanding ``.output`` operator property information in TaskFlow tutorial doc
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #19214:
URL: https://github.com/apache/airflow/pull/19214#issuecomment-951522268
The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on a change in pull request #19214: Expanding ``.output`` operator property information in TaskFlow tutorial doc
Posted by GitBox <gi...@apache.org>.
josh-fell commented on a change in pull request #19214:
URL: https://github.com/apache/airflow/pull/19214#discussion_r736498077
##########
File path: docs/apache-airflow/tutorial_taskflow_api.rst
##########
@@ -224,11 +224,11 @@ is automatically set to true.
Note, If you manually set the ``multiple_outputs`` parameter the inference is disabled and
the parameter value is used.
-Adding dependencies to decorated tasks from regular tasks
----------------------------------------------------------
-The above tutorial shows how to create dependencies between python-based tasks. However, it is
-quite possible while writing a DAG to have some pre-existing tasks such as :class:`~airflow.operators.bash.BashOperator` or :class:`~airflow.sensors.filesystem.FileSensor`
-based tasks which need to be run first before a python-based task is run.
+Adding dependencies to decorated tasks from traditional tasks
Review comment:
Yeah, fair. I think the original section heading was chosen because the example is specifically _from_ TaskFlow function _to_ a traditional task but I think folks could understand that's the concept is bidirectional. Although, do we want to use "TaskFlow functions" instead of "TaskFlow tasks" to be consistent?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on a change in pull request #19214: Expanding ``.output`` operator property information in TaskFlow tutorial doc
Posted by GitBox <gi...@apache.org>.
josh-fell commented on a change in pull request #19214:
URL: https://github.com/apache/airflow/pull/19214#discussion_r736087527
##########
File path: docs/apache-airflow/tutorial_taskflow_api.rst
##########
@@ -224,11 +224,11 @@ is automatically set to true.
Note, If you manually set the ``multiple_outputs`` parameter the inference is disabled and
the parameter value is used.
-Adding dependencies to decorated tasks from regular tasks
----------------------------------------------------------
-The above tutorial shows how to create dependencies between python-based tasks. However, it is
-quite possible while writing a DAG to have some pre-existing tasks such as :class:`~airflow.operators.bash.BashOperator` or :class:`~airflow.sensors.filesystem.FileSensor`
-based tasks which need to be run first before a python-based task is run.
+Adding dependencies to decorated tasks from traditional tasks
+-------------------------------------------------------------
+The above tutorial shows how to create dependencies between TaskFlow functions. However, it is
Review comment:
Using the description "TaskFlow function" to be consistent with the description in the Concepts: TaskFlow doc. Let me know if there is a more succinct description we could use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a change in pull request #19214: Expanding ``.output`` operator property information in TaskFlow tutorial doc
Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #19214:
URL: https://github.com/apache/airflow/pull/19214#discussion_r736703287
##########
File path: docs/apache-airflow/tutorial_taskflow_api.rst
##########
@@ -224,11 +224,11 @@ is automatically set to true.
Note, If you manually set the ``multiple_outputs`` parameter the inference is disabled and
the parameter value is used.
-Adding dependencies to decorated tasks from regular tasks
----------------------------------------------------------
-The above tutorial shows how to create dependencies between python-based tasks. However, it is
-quite possible while writing a DAG to have some pre-existing tasks such as :class:`~airflow.operators.bash.BashOperator` or :class:`~airflow.sensors.filesystem.FileSensor`
-based tasks which need to be run first before a python-based task is run.
+Adding dependencies to decorated tasks from traditional tasks
Review comment:
> Although, do we want to use "TaskFlow functions" instead of "TaskFlow tasks" to be consistent?
Personally i am not sure how i feel about the language `TaskFlow functions` because in my mind taskflow is just another way of writing a `task`.
In fact maybe that's it. Maybe the only thing we should refer to as a "taskflow function" is the function that is _decorated_. And once it is called, the returned value is now a task, and should be referred to as a task.
> Yeah, fair. I think the original section heading was chosen because the example is specifically from TaskFlow function to a traditional task but I think folks could understand that's the concept is bidirectional
yeah i mean it totally works either way, (and certainly no dying on hills here either :) )... just some caffeinated thoughts on it...
i guess the reason the language seemed awkward to me is because for example t1 >> t2 is the same as t2 << t1 -- i.e. you could say the "dependency _relationship_" is symmetrical.
but on further reflection i see you could read "dependency" as only one direction -- this task _depends_ on that one, i.e. t1 must run before t2. but still, that particular phrasing still seems to me to be a bit ambiguous. e.g. if you say "there's a dependency from task `hello` to task `hola`" it doesn't really make it clear which one is upstream and which is downstream.
another way one could think of it, the example illustrates how to add dependencies between tasks created in these two ways _in general_. i.e. true, in the example one is upstream of the other but it doesn't really matter because the technique is the same no matter the directionality. so we don't need to describe it as the special case of `traditional >> TF`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on a change in pull request #19214: Expanding ``.output`` operator property information in TaskFlow tutorial doc
Posted by GitBox <gi...@apache.org>.
josh-fell commented on a change in pull request #19214:
URL: https://github.com/apache/airflow/pull/19214#discussion_r736501487
##########
File path: docs/apache-airflow/tutorial_taskflow_api.rst
##########
@@ -251,18 +251,41 @@ Building this dependency is shown in the code below:
file_task >> order_data
-In the above code block, a new python-based task is defined as ``extract_from_file`` which
+In the above code block, a new TaskFlow function is defined as ``extract_from_file`` which
reads the data from a known file location.
In the main DAG, a new ``FileSensor`` task is defined to check for this file. Please note
that this is a Sensor task which waits for the file.
-Finally, a dependency between this Sensor task and the python-based task is specified.
+Finally, a dependency between this Sensor task and the TaskFlow function is specified.
-Consuming XCOMs with decorated tasks from regular tasks
----------------------------------------------------------
-You may additionally find it necessary to consume an XCOM from a pre-existing task as an input into python-based tasks.
+Consuming XComs from traditional tasks as inputs
+------------------------------------------------
+You may find it necessary to consume an XCom from traditional task, either pushed within the task's execution
+or via its return value, as an input into downstream tasks. You can access the pushed XCom (also known as an
Review comment:
Agreed it's more direct but interacting with users in Slack and StackOverflow it doesn't seem particularly clear, especially to new users, that `XComs` can be pushed via those 2 methods: manually pushed in a task + a return value. Being a little bit verbose here might be beneficial in this case but not a hill I'm going to die on 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on a change in pull request #19214: Expanding ``.output`` operator property information in TaskFlow tutorial doc
Posted by GitBox <gi...@apache.org>.
josh-fell commented on a change in pull request #19214:
URL: https://github.com/apache/airflow/pull/19214#discussion_r736501487
##########
File path: docs/apache-airflow/tutorial_taskflow_api.rst
##########
@@ -251,18 +251,41 @@ Building this dependency is shown in the code below:
file_task >> order_data
-In the above code block, a new python-based task is defined as ``extract_from_file`` which
+In the above code block, a new TaskFlow function is defined as ``extract_from_file`` which
reads the data from a known file location.
In the main DAG, a new ``FileSensor`` task is defined to check for this file. Please note
that this is a Sensor task which waits for the file.
-Finally, a dependency between this Sensor task and the python-based task is specified.
+Finally, a dependency between this Sensor task and the TaskFlow function is specified.
-Consuming XCOMs with decorated tasks from regular tasks
----------------------------------------------------------
-You may additionally find it necessary to consume an XCOM from a pre-existing task as an input into python-based tasks.
+Consuming XComs from traditional tasks as inputs
+------------------------------------------------
+You may find it necessary to consume an XCom from traditional task, either pushed within the task's execution
+or via its return value, as an input into downstream tasks. You can access the pushed XCom (also known as an
Review comment:
Agreed it's more direct but interacting with users in Slack and StackOverflow it doesn't seem particularly clear, especially to new users, that `XComs` can be pushed via those 2 methods: manually pushed in a task + a return value relative to the TaskFlow API. Being a little bit verbose here might be beneficial in this case but not a hill I'm going to die on 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org