You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/04/14 00:08:12 UTC
[GitHub] [airflow] mingshi-wang opened a new pull request, #22562: Adding sensor decorator
mingshi-wang opened a new pull request, #22562:
URL: https://github.com/apache/airflow/pull/22562
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
Added the @task.sensor decorator to convert a Python function to an instance of the BaseSensorOperator. Example usage of the decorator is:
```
@task.sensor(poke_interval=60, timeout=3600, mode="poke")
def f() -> PokeReturnValue:
# implement the condition
condition_met = ...
operator_return_value = ...
return PokeReturnValue(is_done=condition_met, xcom_value=operator_return_value)
```
closes: #20323
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #22562:
URL: https://github.com/apache/airflow/pull/22562#discussion_r945518601
##########
airflow/decorators/sensor.py:
##########
@@ -0,0 +1,80 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from inspect import signature
+from typing import Any, Callable, Collection, Dict, Mapping, Optional, Sequence
+
+from airflow.decorators.base import TaskDecorator, get_unique_task_id, task_decorator_factory
+from airflow.models.taskinstance import Context
+from airflow.sensors.base import BaseSensorOperator, PokeReturnValue
+
+
+class DecoratedSensorOperator(BaseSensorOperator):
Review Comment:
By not using `PythonSensor`, this implementation looses the ability to pass context variables (such as `ti`) to the callable. The ability of passing in `templates_dict` is also lost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1254873667
We are waiting for @jedcunningham to confirm that he is OK with it - can't merge it without it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1098514038
Fine with me cc: @jedcunningham
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1304990535
π π - finally @mingshi-wang !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1251980668
(BTW. The useful message about lack of disk space was in a "cancelled" K8S executor (cancelled because of running too long while trying to work with missing space). The local one was cancelled by K8s failing (we have "fast fail" in this case). Simply GitHub UI shows such "cancelled" jobs less prominently than those that are "failing".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
potiuk closed pull request #22562: Adding sensor decorator
URL: https://github.com/apache/airflow/pull/22562
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mingshi-wang commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
mingshi-wang commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1257256331
HI @jedcunningham can you please take a look again? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #22562:
URL: https://github.com/apache/airflow/pull/22562#discussion_r1005274709
##########
airflow/decorators/__init__.pyi:
##########
@@ -410,5 +412,12 @@ class TaskDecoratorCollection:
of the target ConfigMap's Data field will represent the key-value
pairs as environment variables. Extends env_from.
"""
+ @overload
+ def sensor(self, **kwargs) -> TaskDecorator:
+ """
+ Wraps a Python function into a sensor operator.
+ """
Review Comment:
Can you list out more arguments and the docstring explicitly (like other functions above)? This is shown in editor autocomplete, and the current format is not useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mingshi-wang commented on a diff in pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
mingshi-wang commented on code in PR #22562:
URL: https://github.com/apache/airflow/pull/22562#discussion_r945511736
##########
airflow/decorators/sensor.py:
##########
@@ -0,0 +1,80 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from inspect import signature
+from typing import Any, Callable, Collection, Dict, Mapping, Optional, Sequence
+
+from airflow.decorators.base import TaskDecorator, get_unique_task_id, task_decorator_factory
+from airflow.models.taskinstance import Context
+from airflow.sensors.base import BaseSensorOperator, PokeReturnValue
+
+
+class DecoratedSensorOperator(BaseSensorOperator):
Review Comment:
I don't see not much benefit of having another layer of indirection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1098280783
@ashb @jedcunningham @ephraimbuddy -> shall we add it to 2.3 ? It's a pretty requested feature and seems ready (but I leave it up to you to decide :).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #22562:
URL: https://github.com/apache/airflow/pull/22562#discussion_r945488678
##########
docs/apache-airflow/tutorial_taskflow_api.rst:
##########
@@ -266,6 +266,20 @@ Python version to run your function.
These two options should allow for far greater flexibility for users who wish to keep their workflows more simple
and Pythonic.
+Using the TaskFlow API with Sensor operators
+--------------------------------------------
+You can apply the ``@task.sensor`` decorator to convert a regular Python function to an instance of the
+BaseSensorOperator class. The Python function implements the poke logic and returns an instance of
+the ``PokeReturnValue`` class as the ``poke()`` method in the BaseSensorOperator does. The ``PokeReturnValue`` is
+a new feature in Airflow 2.3 that allows a sensor operator to push an XCom value as described in
+`Community Providers <https://github.com/apache/airflow/blob/main/docs/apache-airflow-providers/howto/create-update-providers.rst>`_ under the section "Having sensors return XOM values".
Review Comment:
Donβt do this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mingshi-wang commented on a diff in pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
mingshi-wang commented on code in PR #22562:
URL: https://github.com/apache/airflow/pull/22562#discussion_r945515731
##########
airflow/decorators/__init__.pyi:
##########
@@ -239,5 +241,11 @@ class TaskDecoratorCollection:
:param cap_add: Include container capabilities
"""
# [END decorator_signature]
+ @overload
+ def sensor(self, python_callable: Optional[Callable] = None, **kwargs) -> TaskDecorator:
Review Comment:
Will fix!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1296509397
@mingshi-wang - I'd love to merge it - can you please address the last comments of @uranusjr ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham closed pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
jedcunningham closed pull request #22562: Adding sensor decorator
URL: https://github.com/apache/airflow/pull/22562
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mingshi-wang commented on a diff in pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
mingshi-wang commented on code in PR #22562:
URL: https://github.com/apache/airflow/pull/22562#discussion_r945505924
##########
airflow/decorators/sensor.py:
##########
@@ -0,0 +1,80 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from inspect import signature
+from typing import Any, Callable, Collection, Dict, Mapping, Optional, Sequence
+
+from airflow.decorators.base import TaskDecorator, get_unique_task_id, task_decorator_factory
+from airflow.models.taskinstance import Context
+from airflow.sensors.base import BaseSensorOperator, PokeReturnValue
+
+
+class DecoratedSensorOperator(BaseSensorOperator):
Review Comment:
Hi @uranusjr Thanks for the comments! The PR was blocked by CI tests failures which I don't think is related to the PR. Do you have any suggestions on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mingshi-wang commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
mingshi-wang commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1290840929
> I dismissed Jed's review - can you please rebase it @mingshi-wang (there is a conflict) - sorry for long delay - I've been on holidays :).
I rebased the PR. Could you help merge it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mingshi-wang commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
mingshi-wang commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1251743696
>
> Rebased it to see if the Helm cheart failures were carried from another error.
Hey @potiuk The Helm local executor still fails. Could you help me understand what could be the cause of the failure? I cannot find any useful message. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mingshi-wang commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
mingshi-wang commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1277933803
Hi @uranusjr @potiuk I think I need some help merging this PR. @jedcunningham is not available to approve this PR. Would you provide some help wrapping up this PR? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #22562:
URL: https://github.com/apache/airflow/pull/22562#discussion_r945490122
##########
airflow/decorators/__init__.pyi:
##########
@@ -239,5 +241,11 @@ class TaskDecoratorCollection:
:param cap_add: Include container capabilities
"""
# [END decorator_signature]
+ @overload
+ def sensor(self, python_callable: Optional[Callable] = None, **kwargs) -> TaskDecorator:
Review Comment:
This is meaningless. Please see above for examples on how to write a useful type annotation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #22562:
URL: https://github.com/apache/airflow/pull/22562#discussion_r945513695
##########
docs/apache-airflow/tutorial_taskflow_api.rst:
##########
@@ -266,6 +266,20 @@ Python version to run your function.
These two options should allow for far greater flexibility for users who wish to keep their workflows more simple
and Pythonic.
+Using the TaskFlow API with Sensor operators
+--------------------------------------------
+You can apply the ``@task.sensor`` decorator to convert a regular Python function to an instance of the
+BaseSensorOperator class. The Python function implements the poke logic and returns an instance of
+the ``PokeReturnValue`` class as the ``poke()`` method in the BaseSensorOperator does. The ``PokeReturnValue`` is
+a new feature in Airflow 2.3 that allows a sensor operator to push an XCom value as described in
+`Community Providers <https://github.com/apache/airflow/blob/main/docs/apache-airflow-providers/howto/create-update-providers.rst>`_ under the section "Having sensors return XOM values".
Review Comment:
Why are you linking to GitHub?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mingshi-wang commented on a diff in pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
mingshi-wang commented on code in PR #22562:
URL: https://github.com/apache/airflow/pull/22562#discussion_r854479496
##########
airflow/example_dags/example_sensor_decorator.py:
##########
@@ -0,0 +1,49 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Example DAG demonstrating the usage of the sensor decorator."""
+
+# [START tutorial]
+import pendulum
+
+from airflow.decorators import dag, task
+from airflow.sensors.base import PokeReturnValue
+
+
+@dag(
+ schedule_interval=None,
+ start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
+ catchup=False,
+ tags=['example'],
+)
+def example_sensor_decorator():
+
+ # Using a sensor operator to wait for the upstream data to be ready.
+ @task.sensor(poke_interval=60, timeout=3600, mode="reschedule")
+ def wait_for_upstream() -> PokeReturnValue:
+ return PokeReturnValue(is_done=True, xcom_value="xcom_value")
+
+ @task
+ def dummy_operator() -> None:
+ pass
+
+ wait_for_upstream() >> dummy_operator()
+
+
+tutorial_etl_dag = example_sensor_decorator()
+# [START tutorial]
Review Comment:
Fixed as suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1108908207
I think this one is a bit risky to add for 2.3.0 so let's iterate and merge after we branch off.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1252250814
I think our Public runners are not big enough to handle "full tests" with k8s :( . So I removed "full tests needed" label and closed/reopened it to run a smaller set of tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mingshi-wang commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
mingshi-wang commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1252511493
HI @jedcunningham I have addressed your comments. Can you take a look again? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mingshi-wang commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
mingshi-wang commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1253862047
Hi @potiuk will you be ok to merge this PR? I see all tests passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1146647197
@mingshi-wang - docs build is failing (that's something that you should fix first) and rebasing the change would also be useful while you do it.
Also we are waiting for @jedcunningham 's re-review - just re-requested it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] AndreGodinho7 commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
AndreGodinho7 commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1245737631
Any updates? Eagerly waiting for this to be merged and released!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #22562:
URL: https://github.com/apache/airflow/pull/22562#discussion_r945489624
##########
airflow/decorators/sensor.py:
##########
@@ -0,0 +1,80 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from inspect import signature
+from typing import Any, Callable, Collection, Dict, Mapping, Optional, Sequence
+
+from airflow.decorators.base import TaskDecorator, get_unique_task_id, task_decorator_factory
+from airflow.models.taskinstance import Context
+from airflow.sensors.base import BaseSensorOperator, PokeReturnValue
+
+
+class DecoratedSensorOperator(BaseSensorOperator):
Review Comment:
Why does this not use `PythonSensor`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1251977929
I think this was accidental failure - simply a problem with lack of space on the machine running it I rebased it and let's see if it happens again. It might also be that we need to optimise some log handling (it looks like logs were filling the disk space. It also could be that (sometimes it happens) that the cahnge causes some recursive behaviour that simpy fills up all available disk while running (by producing continuous stream of logs) - in this case that will be a bug to be fixed by you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mingshi-wang commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
mingshi-wang commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1252509727
> I think our Public runners are not big enough to handle "full tests" with k8s :( . So I removed "full tests needed" label and closed/reopened it to run a smaller set of tests
Thanks for your help!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a diff in pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on code in PR #22562:
URL: https://github.com/apache/airflow/pull/22562#discussion_r850047503
##########
airflow/example_dags/example_sensor_decorator.py:
##########
@@ -0,0 +1,49 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Example DAG demonstrating the usage of the sensor decorator."""
+
+# [START tutorial]
+import pendulum
+
+from airflow.decorators import dag, task
+from airflow.sensors.base import PokeReturnValue
+
+
+@dag(
+ schedule_interval=None,
+ start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
+ catchup=False,
+ tags=['example'],
+)
+def example_sensor_decorator():
+
+ # Using a sensor operator to wait for the upstream data to be ready.
+ @task.sensor(poke_interval=60, timeout=3600, mode="reschedule")
+ def wait_for_upstream() -> PokeReturnValue:
+ return PokeReturnValue(is_done=True, xcom_value="xcom_value")
+
+ @task
+ def dummy_operator() -> None:
+ pass
+
+ wait_for_upstream() >> dummy_operator()
+
+
+tutorial_etl_dag = example_sensor_decorator()
+# [START tutorial]
Review Comment:
```suggestion
# [END tutorial]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mingshi-wang commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
mingshi-wang commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1265735034
> We are waiting for @jedcunningham to confirm that he is OK with it - can't merge it without it.
Hi @potiuk, Jed seems unavailable. Is it possible to get another reviewer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1288131990
I dismissed Jed's review - can you please rebase it @mingshi-wang - sorry for long delay - I've been on holidays :).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mingshi-wang commented on a diff in pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
mingshi-wang commented on code in PR #22562:
URL: https://github.com/apache/airflow/pull/22562#discussion_r945509287
##########
docs/apache-airflow/tutorial_taskflow_api.rst:
##########
@@ -266,6 +266,20 @@ Python version to run your function.
These two options should allow for far greater flexibility for users who wish to keep their workflows more simple
and Pythonic.
+Using the TaskFlow API with Sensor operators
+--------------------------------------------
+You can apply the ``@task.sensor`` decorator to convert a regular Python function to an instance of the
+BaseSensorOperator class. The Python function implements the poke logic and returns an instance of
+the ``PokeReturnValue`` class as the ``poke()`` method in the BaseSensorOperator does. The ``PokeReturnValue`` is
+a new feature in Airflow 2.3 that allows a sensor operator to push an XCom value as described in
+`Community Providers <https://github.com/apache/airflow/blob/main/docs/apache-airflow-providers/howto/create-update-providers.rst>`_ under the section "Having sensors return XOM values".
Review Comment:
Can you be specific?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mingshi-wang commented on a diff in pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
mingshi-wang commented on code in PR #22562:
URL: https://github.com/apache/airflow/pull/22562#discussion_r945513441
##########
airflow/decorators/__init__.pyi:
##########
@@ -239,5 +241,11 @@ class TaskDecoratorCollection:
:param cap_add: Include container capabilities
"""
# [END decorator_signature]
+ @overload
+ def sensor(self, python_callable: Optional[Callable] = None, **kwargs) -> TaskDecorator:
Review Comment:
Can you be more specific?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #22562:
URL: https://github.com/apache/airflow/pull/22562#discussion_r945518068
##########
airflow/decorators/__init__.pyi:
##########
@@ -245,5 +247,11 @@ class TaskDecoratorCollection:
:param cap_add: Include container capabilities
"""
# [END decorator_signature]
+ @overload
+ def sensor(self, *, python_callable: Optional[Callable] = None, **kwargs) -> TaskDecorator:
+ """
+ Wraps a Python function into a sensor operator.
+ :param python_callable: decorated function that implements the poke() logics of a sensor operator.
+ """
Review Comment:
By not using `PythonSensor`, this implementation looses the ability to pass context variables (such as `ti`) to the callable. The ability of passing in `templates_dict` is also lost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mingshi-wang commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
mingshi-wang commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1299272567
> @mingshi-wang - I'd love to merge it - can you please address the last comments of @uranusjr ?
@potiuk @uranusjr I addressed the comments. Could you please take a look again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mingshi-wang commented on a diff in pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
mingshi-wang commented on code in PR #22562:
URL: https://github.com/apache/airflow/pull/22562#discussion_r945511736
##########
airflow/decorators/sensor.py:
##########
@@ -0,0 +1,80 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from inspect import signature
+from typing import Any, Callable, Collection, Dict, Mapping, Optional, Sequence
+
+from airflow.decorators.base import TaskDecorator, get_unique_task_id, task_decorator_factory
+from airflow.models.taskinstance import Context
+from airflow.sensors.base import BaseSensorOperator, PokeReturnValue
+
+
+class DecoratedSensorOperator(BaseSensorOperator):
Review Comment:
I don't see much benefit of having another layer of indirection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #22562:
URL: https://github.com/apache/airflow/pull/22562#discussion_r945514260
##########
airflow/decorators/__init__.pyi:
##########
@@ -239,5 +241,11 @@ class TaskDecoratorCollection:
:param cap_add: Include container capabilities
"""
# [END decorator_signature]
+ @overload
+ def sensor(self, python_callable: Optional[Callable] = None, **kwargs) -> TaskDecorator:
Review Comment:
https://airflow.apache.org/docs/apache-airflow/stable/howto/create-custom-decorator.html#optional-adding-ide-auto-completion-support
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
potiuk merged PR #22562:
URL: https://github.com/apache/airflow/pull/22562
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mingshi-wang commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
mingshi-wang commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1296690039
> @mingshi-wang - I'd love to merge it - can you please address the last comments of @uranusjr ?
Sure will do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1251010471
Rebased it to see if the Helm cheart failures were carried from another error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mingshi-wang commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
mingshi-wang commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1272590760
Hi @jedcunningham - do you have a chance to take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1098576374
No objections here π
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mingshi-wang commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
mingshi-wang commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1146642804
@potiuk Is it OK to merge this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] abhilash1in commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
abhilash1in commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1146168953
Eagerly waiting for this to be merged and released!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1094021719
> Is it still in time for 2.3?
I'd say yes - but leave it up to @jedcunningham @ephraimbuddy to decide.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1291322902
Just one more check @uranusjr @jedcunningham ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mingshi-wang commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
mingshi-wang commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1303896558
> @mingshi-wang - I'd love to merge it - can you please address the last comments of @uranusjr ?
Hi @potiuk are you ok to merge the PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mingshi-wang commented on a diff in pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
mingshi-wang commented on code in PR #22562:
URL: https://github.com/apache/airflow/pull/22562#discussion_r945514696
##########
docs/apache-airflow/tutorial_taskflow_api.rst:
##########
@@ -266,6 +266,20 @@ Python version to run your function.
These two options should allow for far greater flexibility for users who wish to keep their workflows more simple
and Pythonic.
+Using the TaskFlow API with Sensor operators
+--------------------------------------------
+You can apply the ``@task.sensor`` decorator to convert a regular Python function to an instance of the
+BaseSensorOperator class. The Python function implements the poke logic and returns an instance of
+the ``PokeReturnValue`` class as the ``poke()`` method in the BaseSensorOperator does. The ``PokeReturnValue`` is
+a new feature in Airflow 2.3 that allows a sensor operator to push an XCom value as described in
+`Community Providers <https://github.com/apache/airflow/blob/main/docs/apache-airflow-providers/howto/create-update-providers.rst>`_ under the section "Having sensors return XOM values".
Review Comment:
I see. This is probably why breeze build-docs fails in the CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mingshi-wang commented on a diff in pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
mingshi-wang commented on code in PR #22562:
URL: https://github.com/apache/airflow/pull/22562#discussion_r945505924
##########
airflow/decorators/sensor.py:
##########
@@ -0,0 +1,80 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from inspect import signature
+from typing import Any, Callable, Collection, Dict, Mapping, Optional, Sequence
+
+from airflow.decorators.base import TaskDecorator, get_unique_task_id, task_decorator_factory
+from airflow.models.taskinstance import Context
+from airflow.sensors.base import BaseSensorOperator, PokeReturnValue
+
+
+class DecoratedSensorOperator(BaseSensorOperator):
Review Comment:
Hi @uranusjr Thanks for the comments! The PR was blocked by CI tests failures which I don't think are related to the PR. Do you have any suggestions on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #22562: Adding sensor decorator
Posted by GitBox <gi...@apache.org>.
ashb commented on PR #22562:
URL: https://github.com/apache/airflow/pull/22562#issuecomment-1098753709
Someone will need to check how this interacts with task mapping first please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a diff in pull request #22562: Adding sensor decorator
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #22562:
URL: https://github.com/apache/airflow/pull/22562#discussion_r1136908313
##########
airflow/decorators/sensor.py:
##########
@@ -0,0 +1,74 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from typing import Callable, Sequence
+
+from airflow.decorators.base import TaskDecorator, get_unique_task_id, task_decorator_factory
+from airflow.models.taskinstance import Context
+from airflow.sensors.base import PokeReturnValue
+from airflow.sensors.python import PythonSensor
+
+
+class DecoratedSensorOperator(PythonSensor):
Review Comment:
Oh, this doesn't inherit from DecoratedOperator -- was there any reason for it not?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a diff in pull request #22562: Adding sensor decorator
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #22562:
URL: https://github.com/apache/airflow/pull/22562#discussion_r1136908633
##########
airflow/decorators/sensor.py:
##########
@@ -0,0 +1,74 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from typing import Callable, Sequence
+
+from airflow.decorators.base import TaskDecorator, get_unique_task_id, task_decorator_factory
+from airflow.models.taskinstance import Context
+from airflow.sensors.base import PokeReturnValue
+from airflow.sensors.python import PythonSensor
+
+
+class DecoratedSensorOperator(PythonSensor):
Review Comment:
(Not a problem I guess, since it's all tested etc.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org