You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/08/31 06:41:35 UTC
[GitHub] [airflow] ee987 opened a new pull request #17927: Add example of TaskFlow to declare task dependencies
ee987 opened a new pull request #17927:
URL: https://github.com/apache/airflow/pull/17927
As a new user of Airflow it is not obvious to realize TaskFlow exists and
one might think declaring dependencies between tasks is very verbose.
Having more examples of such would help in promoting this style of
writing DAGs and speed up on-boarding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a change in pull request #17927: Add example of TaskFlow to declare task dependencies
Posted by GitBox <gi...@apache.org>.
eladkal commented on a change in pull request #17927:
URL: https://github.com/apache/airflow/pull/17927#discussion_r723567582
##########
File path: docs/apache-airflow/concepts/tasks.rst
##########
@@ -36,16 +36,30 @@ Relationships
The key part of using Tasks is defining how they relate to each other - their *dependencies*, or as we say in Airflow, their *upstream* and *downstream* tasks. You declare your Tasks first, and then you declare their dependencies second.
-There are two ways of declaring dependencies - using the ``>>`` and ``<<`` (bitshift) operators::
+There are three ways of declaring dependencies:
+
+Option 1: using the ``>>`` and ``<<`` (bitshift) operators::
first_task >> second_task >> [third_task, fourth_task]
-Or the more explicit ``set_upstream`` and ``set_downstream`` methods::
+Option 2: The more explicit ``set_upstream`` and ``set_downstream`` methods::
first_task.set_downstream(second_task)
third_task.set_upstream(second_task)
-
-These both do exactly the same thing, but in general we recommend you use the bitshift operators, as they are easier to read in most cases.
+
+Option 3: The @task decorator provided by TaskFlow API, see :ref:`TaskFlow <concepts:taskflow>`::
+
+ @task
+ def second_task(some_parameter):
+ ...
+ @task
+ def first_task():
+ ...
+
+ second_task(first_task())
+
Review comment:
```suggestion
Option 3: The @task decorator provided by TaskFlow API, see :ref:`TaskFlow <concepts:taskflow>`::
@task
def second_task(some_parameter):
...
@task
def first_task():
...
second_task(first_task())
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #17927: Add example of TaskFlow to declare task dependencies
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #17927:
URL: https://github.com/apache/airflow/pull/17927#issuecomment-908957111
The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #17927: Add example of TaskFlow to declare task dependencies
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #17927:
URL: https://github.com/apache/airflow/pull/17927#discussion_r723765288
##########
File path: docs/apache-airflow/concepts/tasks.rst
##########
@@ -36,16 +36,30 @@ Relationships
The key part of using Tasks is defining how they relate to each other - their *dependencies*, or as we say in Airflow, their *upstream* and *downstream* tasks. You declare your Tasks first, and then you declare their dependencies second.
-There are two ways of declaring dependencies - using the ``>>`` and ``<<`` (bitshift) operators::
+There are three ways of declaring dependencies:
+
+Option 1: using the ``>>`` and ``<<`` (bitshift) operators::
first_task >> second_task >> [third_task, fourth_task]
-Or the more explicit ``set_upstream`` and ``set_downstream`` methods::
+Option 2: The more explicit ``set_upstream`` and ``set_downstream`` methods::
first_task.set_downstream(second_task)
third_task.set_upstream(second_task)
-These both do exactly the same thing, but in general we recommend you use the bitshift operators, as they are easier to read in most cases.
+Option 3: The @task decorator provided by TaskFlow API, see :ref:`TaskFlow <concepts:taskflow>`::
+
+ @task
+ def second_task(some_parameter):
+ ...
+ @task
+ def first_task():
+ ...
+
+ second_task(first_task())
+
+
+The former two do exactly the same thing and work on any type of Operator, but in general we recommend you use the bitshift operators, as they are easier to read in most cases. The TaskFlow API likewise connects dependencies between tasks, but is tailored to python functions written with the @task-decorator, note that the functions here are not executed inline, rather they are converted to tasks to be scheduled and sent to Executors, for more information see :ref:`TaskFlow <concepts:taskflow>`.
Review comment:
```suggestion
The former two do exactly the same thing and work on any type of Operator, but in general we recommend you use the bitshift operators, as they are easier to read in most cases. The TaskFlow API likewise connects dependencies between tasks, but is tailored to python functions written with the ``@task-decorator``, note that the functions here are not executed inline, rather they are converted to tasks to be scheduled and sent to Executors, for more information see :ref:`TaskFlow <concepts:taskflow>`.
```
Also the CI says ``concepts:taskflow` is not a defined reference. Where do you intend to link this to? (There's another occurrence of this above.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #17927: Add example of TaskFlow to declare task dependencies
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #17927:
URL: https://github.com/apache/airflow/pull/17927#issuecomment-908957111
The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #17927: Add example of TaskFlow to declare task dependencies
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #17927:
URL: https://github.com/apache/airflow/pull/17927#discussion_r723764445
##########
File path: docs/apache-airflow/concepts/tasks.rst
##########
@@ -36,16 +36,30 @@ Relationships
The key part of using Tasks is defining how they relate to each other - their *dependencies*, or as we say in Airflow, their *upstream* and *downstream* tasks. You declare your Tasks first, and then you declare their dependencies second.
-There are two ways of declaring dependencies - using the ``>>`` and ``<<`` (bitshift) operators::
+There are three ways of declaring dependencies:
+
+Option 1: using the ``>>`` and ``<<`` (bitshift) operators::
first_task >> second_task >> [third_task, fourth_task]
-Or the more explicit ``set_upstream`` and ``set_downstream`` methods::
+Option 2: The more explicit ``set_upstream`` and ``set_downstream`` methods::
first_task.set_downstream(second_task)
third_task.set_upstream(second_task)
-These both do exactly the same thing, but in general we recommend you use the bitshift operators, as they are easier to read in most cases.
+Option 3: The @task decorator provided by TaskFlow API, see :ref:`TaskFlow <concepts:taskflow>`::
+
+ @task
+ def second_task(some_parameter):
+ ...
+ @task
+ def first_task():
+ ...
Review comment:
```suggestion
@task
def second_task(some_parameter):
...
@task
def first_task():
...
```
(I hope the linter won't complain this needs two empty lines.)
##########
File path: docs/apache-airflow/concepts/tasks.rst
##########
@@ -36,16 +36,30 @@ Relationships
The key part of using Tasks is defining how they relate to each other - their *dependencies*, or as we say in Airflow, their *upstream* and *downstream* tasks. You declare your Tasks first, and then you declare their dependencies second.
-There are two ways of declaring dependencies - using the ``>>`` and ``<<`` (bitshift) operators::
+There are three ways of declaring dependencies:
+
+Option 1: using the ``>>`` and ``<<`` (bitshift) operators::
first_task >> second_task >> [third_task, fourth_task]
-Or the more explicit ``set_upstream`` and ``set_downstream`` methods::
+Option 2: The more explicit ``set_upstream`` and ``set_downstream`` methods::
first_task.set_downstream(second_task)
third_task.set_upstream(second_task)
-These both do exactly the same thing, but in general we recommend you use the bitshift operators, as they are easier to read in most cases.
+Option 3: The @task decorator provided by TaskFlow API, see :ref:`TaskFlow <concepts:taskflow>`::
+
+ @task
+ def second_task(some_parameter):
+ ...
+ @task
+ def first_task():
+ ...
+
+ second_task(first_task())
+
+
+The former two do exactly the same thing and work on any type of Operator, but in general we recommend you use the bitshift operators, as they are easier to read in most cases. The TaskFlow API likewise connects dependencies between tasks, but is tailored to python functions written with the @task-decorator, note that the functions here are not executed inline, rather they are converted to tasks to be scheduled and sent to Executors, for more information see :ref:`TaskFlow <concepts:taskflow>`.
Review comment:
```suggestion
The former two do exactly the same thing and work on any type of Operator, but in general we recommend you use the bitshift operators, as they are easier to read in most cases. The TaskFlow API likewise connects dependencies between tasks, but is tailored to python functions written with the ``@task-decorator``, note that the functions here are not executed inline, rather they are converted to tasks to be scheduled and sent to Executors, for more information see :ref:`TaskFlow <concepts:taskflow>`.
```
Also the CI says ``concepts:taskflow` is not a defined reference. Where do you intend to link this to? (There's another occurrence of this above.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #17927: Add example of TaskFlow to declare task dependencies
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #17927:
URL: https://github.com/apache/airflow/pull/17927#discussion_r699040163
##########
File path: docs/apache-airflow/concepts/tasks.rst
##########
@@ -36,16 +36,30 @@ Relationships
The key part of using Tasks is defining how they relate to each other - their *dependencies*, or as we say in Airflow, their *upstream* and *downstream* tasks. You declare your Tasks first, and then you declare their dependencies second.
-There are two ways of declaring dependencies - using the ``>>`` and ``<<`` (bitshift) operators::
+There are three ways of declaring dependencies:
+
+Option 1: using the ``>>`` and ``<<`` (bitshift) operators::
first_task >> second_task >> [third_task, fourth_task]
-Or the more explicit ``set_upstream`` and ``set_downstream`` methods::
+Option 2: The more explicit ``set_upstream`` and ``set_downstream`` methods::
first_task.set_downstream(second_task)
third_task.set_upstream(second_task)
-
-These both do exactly the same thing, but in general we recommend you use the bitshift operators, as they are easier to read in most cases.
+
+Option 3: The @task decorator provided by TaskFlow API, see :ref:`concepts:taskflow`::
+
+ @task
+ def second_task(some_parameter):
+ ...
+ @task
+ def first_task():
+ ...
+
+ second_task(first_task()))
+
+
+The former two do exactly the same thing and work on any type of Operator, but in general we recommend you use the bitshift operators, as they are easier to read in most cases. The TaskFlow API likewise connects dependencies between tasks, but is tailored to python functions written with the @task-decorator, note that the functions here are not executed inline, rather they are converted to tasks to be scheduled and sent to Executors, for more information see see :ref:`concepts:taskflow`.
Review comment:
```suggestion
The former two do exactly the same thing and work on any type of Operator, but in general we recommend you use the bitshift operators, as they are easier to read in most cases. The TaskFlow API likewise connects dependencies between tasks, but is tailored to python functions written with the @task-decorator, note that the functions here are not executed inline, rather they are converted to tasks to be scheduled and sent to Executors, for more information see see :ref:`TaskFlow <concepts:taskflow>`.
```
##########
File path: docs/apache-airflow/concepts/tasks.rst
##########
@@ -36,16 +36,30 @@ Relationships
The key part of using Tasks is defining how they relate to each other - their *dependencies*, or as we say in Airflow, their *upstream* and *downstream* tasks. You declare your Tasks first, and then you declare their dependencies second.
-There are two ways of declaring dependencies - using the ``>>`` and ``<<`` (bitshift) operators::
+There are three ways of declaring dependencies:
+
+Option 1: using the ``>>`` and ``<<`` (bitshift) operators::
first_task >> second_task >> [third_task, fourth_task]
-Or the more explicit ``set_upstream`` and ``set_downstream`` methods::
+Option 2: The more explicit ``set_upstream`` and ``set_downstream`` methods::
first_task.set_downstream(second_task)
third_task.set_upstream(second_task)
-
-These both do exactly the same thing, but in general we recommend you use the bitshift operators, as they are easier to read in most cases.
+
+Option 3: The @task decorator provided by TaskFlow API, see :ref:`concepts:taskflow`::
+
+ @task
+ def second_task(some_parameter):
+ ...
+ @task
+ def first_task():
+ ...
+
+ second_task(first_task()))
Review comment:
```suggestion
second_task(first_task())
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #17927: Add example of TaskFlow to declare task dependencies
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #17927:
URL: https://github.com/apache/airflow/pull/17927#discussion_r723764445
##########
File path: docs/apache-airflow/concepts/tasks.rst
##########
@@ -36,16 +36,30 @@ Relationships
The key part of using Tasks is defining how they relate to each other - their *dependencies*, or as we say in Airflow, their *upstream* and *downstream* tasks. You declare your Tasks first, and then you declare their dependencies second.
-There are two ways of declaring dependencies - using the ``>>`` and ``<<`` (bitshift) operators::
+There are three ways of declaring dependencies:
+
+Option 1: using the ``>>`` and ``<<`` (bitshift) operators::
first_task >> second_task >> [third_task, fourth_task]
-Or the more explicit ``set_upstream`` and ``set_downstream`` methods::
+Option 2: The more explicit ``set_upstream`` and ``set_downstream`` methods::
first_task.set_downstream(second_task)
third_task.set_upstream(second_task)
-These both do exactly the same thing, but in general we recommend you use the bitshift operators, as they are easier to read in most cases.
+Option 3: The @task decorator provided by TaskFlow API, see :ref:`TaskFlow <concepts:taskflow>`::
+
+ @task
+ def second_task(some_parameter):
+ ...
+ @task
+ def first_task():
+ ...
Review comment:
```suggestion
@task
def second_task(some_parameter):
...
@task
def first_task():
...
```
(I hope the linter won't complain this needs two empty lines.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #17927: Add example of TaskFlow to declare task dependencies
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #17927:
URL: https://github.com/apache/airflow/pull/17927#issuecomment-908945053
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, itโs a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better ๐.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed pull request #17927: Add example of TaskFlow to declare task dependencies
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #17927:
URL: https://github.com/apache/airflow/pull/17927
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #17927: Add example of TaskFlow to declare task dependencies
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #17927:
URL: https://github.com/apache/airflow/pull/17927#discussion_r701693524
##########
File path: docs/apache-airflow/concepts/tasks.rst
##########
@@ -36,16 +36,30 @@ Relationships
The key part of using Tasks is defining how they relate to each other - their *dependencies*, or as we say in Airflow, their *upstream* and *downstream* tasks. You declare your Tasks first, and then you declare their dependencies second.
-There are two ways of declaring dependencies - using the ``>>`` and ``<<`` (bitshift) operators::
+There are three ways of declaring dependencies:
+
+Option 1: using the ``>>`` and ``<<`` (bitshift) operators::
first_task >> second_task >> [third_task, fourth_task]
-Or the more explicit ``set_upstream`` and ``set_downstream`` methods::
+Option 2: The more explicit ``set_upstream`` and ``set_downstream`` methods::
first_task.set_downstream(second_task)
third_task.set_upstream(second_task)
-
-These both do exactly the same thing, but in general we recommend you use the bitshift operators, as they are easier to read in most cases.
+
+Option 3: The @task decorator provided by TaskFlow API, see :ref:`concepts:taskflow`::
Review comment:
```suggestion
Option 3: The @task decorator provided by TaskFlow API, see :ref:`TaskFlow <concepts:taskflow>`::
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #17927: Add example of TaskFlow to declare task dependencies
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #17927:
URL: https://github.com/apache/airflow/pull/17927#discussion_r701692314
##########
File path: docs/apache-airflow/concepts/tasks.rst
##########
@@ -56,10 +56,10 @@ Option 3: The @task decorator provided by TaskFlow API, see :ref:`concepts:taskf
def first_task():
...
- second_task(first_task()))
+ second_task(first_task())
-The former two do exactly the same thing and work on any type of Operator, but in general we recommend you use the bitshift operators, as they are easier to read in most cases. The TaskFlow API likewise connects dependencies between tasks, but is tailored to python functions written with the @task-decorator, note that the functions here are not executed inline, rather they are converted to tasks to be scheduled and sent to Executors, for more information see see :ref:`concepts:taskflow`.
+The former two do exactly the same thing and work on any type of Operator, but in general we recommend you use the bitshift operators, as they are easier to read in most cases. The TaskFlow API likewise connects dependencies between tasks, but is tailored to python functions written with the @task-decorator, note that the functions here are not executed inline, rather they are converted to tasks to be scheduled and sent to Executors, for more information see see :ref:`TaskFlow <concepts:taskflow>`.
Review comment:
```suggestion
The former two do exactly the same thing and work on any type of Operator, but in general we recommend you use the bitshift operators, as they are easier to read in most cases. The TaskFlow API likewise connects dependencies between tasks, but is tailored to python functions written with the @task-decorator, note that the functions here are not executed inline, rather they are converted to tasks to be scheduled and sent to Executors, for more information see :ref:`TaskFlow <concepts:taskflow>`.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #17927: Add example of TaskFlow to declare task dependencies
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #17927:
URL: https://github.com/apache/airflow/pull/17927#discussion_r699040163
##########
File path: docs/apache-airflow/concepts/tasks.rst
##########
@@ -36,16 +36,30 @@ Relationships
The key part of using Tasks is defining how they relate to each other - their *dependencies*, or as we say in Airflow, their *upstream* and *downstream* tasks. You declare your Tasks first, and then you declare their dependencies second.
-There are two ways of declaring dependencies - using the ``>>`` and ``<<`` (bitshift) operators::
+There are three ways of declaring dependencies:
+
+Option 1: using the ``>>`` and ``<<`` (bitshift) operators::
first_task >> second_task >> [third_task, fourth_task]
-Or the more explicit ``set_upstream`` and ``set_downstream`` methods::
+Option 2: The more explicit ``set_upstream`` and ``set_downstream`` methods::
first_task.set_downstream(second_task)
third_task.set_upstream(second_task)
-
-These both do exactly the same thing, but in general we recommend you use the bitshift operators, as they are easier to read in most cases.
+
+Option 3: The @task decorator provided by TaskFlow API, see :ref:`concepts:taskflow`::
+
+ @task
+ def second_task(some_parameter):
+ ...
+ @task
+ def first_task():
+ ...
+
+ second_task(first_task()))
+
+
+The former two do exactly the same thing and work on any type of Operator, but in general we recommend you use the bitshift operators, as they are easier to read in most cases. The TaskFlow API likewise connects dependencies between tasks, but is tailored to python functions written with the @task-decorator, note that the functions here are not executed inline, rather they are converted to tasks to be scheduled and sent to Executors, for more information see see :ref:`concepts:taskflow`.
Review comment:
```suggestion
The former two do exactly the same thing and work on any type of Operator, but in general we recommend you use the bitshift operators, as they are easier to read in most cases. The TaskFlow API likewise connects dependencies between tasks, but is tailored to python functions written with the @task-decorator, note that the functions here are not executed inline, rather they are converted to tasks to be scheduled and sent to Executors, for more information see see :ref:`TaskFlow <concepts:taskflow>`.
```
##########
File path: docs/apache-airflow/concepts/tasks.rst
##########
@@ -36,16 +36,30 @@ Relationships
The key part of using Tasks is defining how they relate to each other - their *dependencies*, or as we say in Airflow, their *upstream* and *downstream* tasks. You declare your Tasks first, and then you declare their dependencies second.
-There are two ways of declaring dependencies - using the ``>>`` and ``<<`` (bitshift) operators::
+There are three ways of declaring dependencies:
+
+Option 1: using the ``>>`` and ``<<`` (bitshift) operators::
first_task >> second_task >> [third_task, fourth_task]
-Or the more explicit ``set_upstream`` and ``set_downstream`` methods::
+Option 2: The more explicit ``set_upstream`` and ``set_downstream`` methods::
first_task.set_downstream(second_task)
third_task.set_upstream(second_task)
-
-These both do exactly the same thing, but in general we recommend you use the bitshift operators, as they are easier to read in most cases.
+
+Option 3: The @task decorator provided by TaskFlow API, see :ref:`concepts:taskflow`::
+
+ @task
+ def second_task(some_parameter):
+ ...
+ @task
+ def first_task():
+ ...
+
+ second_task(first_task()))
Review comment:
```suggestion
second_task(first_task())
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #17927: Add example of TaskFlow to declare task dependencies
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #17927:
URL: https://github.com/apache/airflow/pull/17927#issuecomment-908945053
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, itโs a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better ๐.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #17927: Add example of TaskFlow to declare task dependencies
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #17927:
URL: https://github.com/apache/airflow/pull/17927#issuecomment-974926170
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #17927: Add example of TaskFlow to declare task dependencies
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #17927:
URL: https://github.com/apache/airflow/pull/17927#discussion_r723576729
##########
File path: docs/apache-airflow/concepts/tasks.rst
##########
@@ -36,16 +36,30 @@ Relationships
The key part of using Tasks is defining how they relate to each other - their *dependencies*, or as we say in Airflow, their *upstream* and *downstream* tasks. You declare your Tasks first, and then you declare their dependencies second.
-There are two ways of declaring dependencies - using the ``>>`` and ``<<`` (bitshift) operators::
+There are three ways of declaring dependencies:
+
+Option 1: using the ``>>`` and ``<<`` (bitshift) operators::
first_task >> second_task >> [third_task, fourth_task]
-Or the more explicit ``set_upstream`` and ``set_downstream`` methods::
+Option 2: The more explicit ``set_upstream`` and ``set_downstream`` methods::
first_task.set_downstream(second_task)
third_task.set_upstream(second_task)
-These both do exactly the same thing, but in general we recommend you use the bitshift operators, as they are easier to read in most cases.
+Option 3: The @task decorator provided by TaskFlow API, see :ref:`TaskFlow <concepts:taskflow>`::
Review comment:
```suggestion
Option 3: The ``@task`` decorator provided by TaskFlow API, see :ref:`TaskFlow <concepts:taskflow>`::
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #17927: Add example of TaskFlow to declare task dependencies
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #17927:
URL: https://github.com/apache/airflow/pull/17927#issuecomment-912366920
@ee987, sorry, wanted to get this merged but there's now a static check failure. Please can you address it, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org