You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/03/28 07:02:59 UTC
[GitHub] [airflow] mingshi-wang opened a new pull request #22562: Adding sensor decorator
mingshi-wang opened a new pull request #22562:
URL: https://github.com/apache/airflow/pull/22562
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
Added the @task.sensor decorator to convert a Python function to an instance of the BaseSensorOperator. Example usage of the decorator is:
```
@task.sensor(poke_interval=60, timeout=3600, mode="poke")
def f():
# implement the condition
condition_met = ...
return condition_met
```
closes: #20323
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mingshi-wang commented on a change in pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
mingshi-wang commented on a change in pull request #22562:
URL: https://github.com/apache/airflow/pull/22562#discussion_r836617849
##########
File path: airflow/decorators/__init__.pyi
##########
@@ -245,5 +247,11 @@ class TaskDecoratorCollection:
:param cap_add: Include container capabilities
"""
# [END decorator_signature]
+ @overload
+ def sensor(self, *, python_callable: Optional[Callable] = None, **kwargs) -> TaskDecorator:
+ """
+ Wraps a Python function into a sensor operator.
+ :param python_callable: decorated function that implements the poke() logics of a sensor operator.
+ """
Review comment:
The only argument useful for the sensor decorator is "python_callable", but please let me know if there are other arguments that need to be explicit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #22562:
URL: https://github.com/apache/airflow/pull/22562#discussion_r836271081
##########
File path: airflow/decorators/__init__.pyi
##########
@@ -245,5 +247,11 @@ class TaskDecoratorCollection:
:param cap_add: Include container capabilities
"""
# [END decorator_signature]
+ @overload
+ def sensor(self, *, python_callable: Optional[Callable] = None, **kwargs) -> TaskDecorator:
+ """
+ Wraps a Python function into a sensor operator.
+ :param python_callable: decorated function that implements the poke() logics of a sensor operator.
+ """
Review comment:
Individual arguments should be spelled out explicitly (instead of using `kwargs`) since this file is providing editor autocompletion, and `kwargs` is mostly useless for autocompletion.
Also, does this make sense?
```python
@sensor # No arguments at all!
def func():
return PokeReturnValue(...)
```
If not, `python_callable` should be removed since that argument is there to specifically support this use case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #22562:
URL: https://github.com/apache/airflow/pull/22562#discussion_r839659895
##########
File path: docs/apache-airflow/tutorial_taskflow_api.rst
##########
@@ -208,6 +208,18 @@ Python version to run your function.
These two options should allow for far greater flexibility for users who wish to keep their workflows more simple
and Pythonic.
+Using the TaskFlow API with Sensor operators
+--------------------------------------------
+You can apply the ``@task.sensor`` decorator to convert a regular Python function to an instance of the BaseSensorOperator
+class. The Python function implements the poke logic and returns a Boolean value just as the ``poke()`` method in the
+BaseSensorOperator does.
Review comment:
so :) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1084908222
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mingshi-wang commented on a change in pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
mingshi-wang commented on a change in pull request #22562:
URL: https://github.com/apache/airflow/pull/22562#discussion_r839864384
##########
File path: airflow/decorators/__init__.pyi
##########
@@ -245,5 +247,11 @@ class TaskDecoratorCollection:
:param cap_add: Include container capabilities
"""
# [END decorator_signature]
+ @overload
+ def sensor(self, *, python_callable: Optional[Callable] = None, **kwargs) -> TaskDecorator:
+ """
+ Wraps a Python function into a sensor operator.
+ :param python_callable: decorated function that implements the poke() logics of a sensor operator.
+ """
Review comment:
@uranusjr I need another +1 to this PR. Could you help take a look again? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #22562:
URL: https://github.com/apache/airflow/pull/22562#discussion_r836271081
##########
File path: airflow/decorators/__init__.pyi
##########
@@ -245,5 +247,11 @@ class TaskDecoratorCollection:
:param cap_add: Include container capabilities
"""
# [END decorator_signature]
+ @overload
+ def sensor(self, *, python_callable: Optional[Callable] = None, **kwargs) -> TaskDecorator:
+ """
+ Wraps a Python function into a sensor operator.
+ :param python_callable: decorated function that implements the poke() logics of a sensor operator.
+ """
Review comment:
`python_callable` can’t be keyword-only, it needs to be accepted as positional for the decorator to work. Also individual arguments should be spelled out explicitly (instead of using `kwargs`) since this file is providing editor autocompletion, and `kwargs` is mostly useless for autocompletion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mingshi-wang commented on a change in pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
mingshi-wang commented on a change in pull request #22562:
URL: https://github.com/apache/airflow/pull/22562#discussion_r837738958
##########
File path: docs/apache-airflow/tutorial_taskflow_api.rst
##########
@@ -208,6 +208,18 @@ Python version to run your function.
These two options should allow for far greater flexibility for users who wish to keep their workflows more simple
and Pythonic.
+Using the TaskFlow API with Sensor operators
+--------------------------------------------
+You can apply the ``@task.sensor`` decorator to convert a regular Python function to an instance of the BaseSensorOperator
+class. The Python function implements the poke logic and returns a Boolean value just as the ``poke()`` method in the
+BaseSensorOperator does.
Review comment:
Sure!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mingshi-wang commented on a change in pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
mingshi-wang commented on a change in pull request #22562:
URL: https://github.com/apache/airflow/pull/22562#discussion_r836590087
##########
File path: airflow/decorators/__init__.pyi
##########
@@ -245,5 +247,11 @@ class TaskDecoratorCollection:
:param cap_add: Include container capabilities
"""
# [END decorator_signature]
+ @overload
+ def sensor(self, *, python_callable: Optional[Callable] = None, **kwargs) -> TaskDecorator:
+ """
+ Wraps a Python function into a sensor operator.
+ :param python_callable: decorated function that implements the poke() logics of a sensor operator.
+ """
Review comment:
Thanks @uranusjr! I've updated the signature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1084906255
LGTM. But we need one other approve to merge it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mingshi-wang commented on a change in pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
mingshi-wang commented on a change in pull request #22562:
URL: https://github.com/apache/airflow/pull/22562#discussion_r839858311
##########
File path: docs/apache-airflow/tutorial_taskflow_api.rst
##########
@@ -208,6 +208,18 @@ Python version to run your function.
These two options should allow for far greater flexibility for users who wish to keep their workflows more simple
and Pythonic.
+Using the TaskFlow API with Sensor operators
+--------------------------------------------
+You can apply the ``@task.sensor`` decorator to convert a regular Python function to an instance of the BaseSensorOperator
+class. The Python function implements the poke logic and returns a Boolean value just as the ``poke()`` method in the
+BaseSensorOperator does.
Review comment:
Hi @potiuk I have updated the doc. Please have a look again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #22562:
URL: https://github.com/apache/airflow/pull/22562#discussion_r837531537
##########
File path: docs/apache-airflow/tutorial_taskflow_api.rst
##########
@@ -208,6 +208,18 @@ Python version to run your function.
These two options should allow for far greater flexibility for users who wish to keep their workflows more simple
and Pythonic.
+Using the TaskFlow API with Sensor operators
+--------------------------------------------
+You can apply the ``@task.sensor`` decorator to convert a regular Python function to an instance of the BaseSensorOperator
+class. The Python function implements the poke logic and returns a Boolean value just as the ``poke()`` method in the
+BaseSensorOperator does.
Review comment:
I think you shoudl mention `PokeReturnValue` here as well. This is a new addition in Airflow 2.3.0 (I see you have it in type, but the doc should be updated as well).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org