You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/12/29 04:39:33 UTC
[GitHub] [airflow] dstandish opened a new pull request #20547: Allow sensors to return XCom values with PokeReturnValue
dstandish opened a new pull request #20547:
URL: https://github.com/apache/airflow/pull/20547
Historically sensors do not support returning XCom unless you override the `execute` method.
With this change we can optionally return an XCom value by returning one along with `True`
in new class PokeReturnValue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20547: Allow sensors to return XCom values with PokeReturnValue
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20547:
URL: https://github.com/apache/airflow/pull/20547#issuecomment-1002694727
It's still not really backwards compatible (in the sense a having a New Sensor returnig PokeReturnValue run on airlfow pre 2.3).
If you return PokeReturnValue() in Airlfow 2.2. it will not work. The BaseSensorOperator from Airflow 2.2 will run this:
```
while not self.poke():
````
And it will always finish after first check no matter what "is_done" is. In this case self.poke() returns PokeReturnValue() class which is Truthy).
But It could be made backwards compatible - and I like this much more than my "return Truthy" approach.
Should be enough to do this:
```
class PokeReturnValue:
"""
Sensors can optionally return this an instance of this class in the poke method.
If an xcom value is supplied when sensor is done, then the XCom value will be
push through the operator return value.
"""
def __init__(self, is_done, xcom_value=None):
self.xcom_value = xcom_value
self.is_done = is_done
def __bool__(self):
return self.is_done
```
I think this is the BEST approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mingshi-wang commented on pull request #20547: Allow sensors to return XCom values with PokeReturnValue
Posted by GitBox <gi...@apache.org>.
mingshi-wang commented on pull request #20547:
URL: https://github.com/apache/airflow/pull/20547#issuecomment-1002760879
> It's still not really backwards compatible (in the sense a having a New Sensor returnig PokeReturnValue run on airlfow pre 2.3).
>
> If you return PokeReturnValue() in Airlfow 2.2. it will not work. The BaseSensorOperator from Airflow 2.2 will run this:
>
> ```
> while not self.poke():
> ```
>
> And it will always finish after first check no matter what "is_done" is. In this case self.poke() returns PokeReturnValue() class which is Truthy no matter what is set in `is_done`).
>
> But It could be made backwards compatible - and I like this much more than my "return Truthy" approach.
>
> Should be enough to do this:
>
> ```
> class PokeReturnValue:
> """
> Sensors can optionally return this an instance of this class in the poke method.
> If an xcom value is supplied when sensor is done, then the XCom value will be
> push through the operator return value.
> """
>
> def __init__(self, is_done, xcom_value=None):
> self.xcom_value = xcom_value
> self.is_done = is_done
>
>
> def __bool__(self):
> return self.is_done
> ```
>
> ~I think this is the BEST approach.~
>
> UPDATE: Nope. It won't work, because in Airflow 2.2 there wil be no PokeReturnValue class.
> It's still not really backwards compatible (in the sense a having a New Sensor returnig PokeReturnValue run on airlfow pre 2.3).
>
> If you return PokeReturnValue() in Airlfow 2.2. it will not work. The BaseSensorOperator from Airflow 2.2 will run this:
>
> ```
> while not self.poke():
> ```
>
> And it will always finish after first check no matter what "is_done" is. In this case self.poke() returns PokeReturnValue() class which is Truthy no matter what is set in `is_done`).
>
> But It could be made backwards compatible - and I like this much more than my "return Truthy" approach.
>
> Should be enough to do this:
>
> ```
> class PokeReturnValue:
> """
> Sensors can optionally return this an instance of this class in the poke method.
> If an xcom value is supplied when sensor is done, then the XCom value will be
> push through the operator return value.
> """
>
> def __init__(self, is_done, xcom_value=None):
> self.xcom_value = xcom_value
> self.is_done = is_done
>
>
> def __bool__(self):
> return self.is_done
> ```
>
> ~I think this is the BEST approach.~
>
> UPDATE: Nope. It won't work, because in Airflow 2.2 there wil be no PokeReturnValue class.
A not-so-ideal way is to declare a dynamic return type:
def poke(self, context: Context) -> Union[bool, 'PokeReturnValue']:
...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20547: Allow sensors to return XCom values with PokeReturnValue
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20547:
URL: https://github.com/apache/airflow/pull/20547#issuecomment-1002549896
Duplicate of #20546 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20547: Allow sensors to return XCom values with PokeReturnValue
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20547:
URL: https://github.com/apache/airflow/pull/20547#issuecomment-1002690316
Ah sorry. let me take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish closed pull request #20547: Allow sensors to return XCom values with PokeReturnValue
Posted by GitBox <gi...@apache.org>.
dstandish closed pull request #20547:
URL: https://github.com/apache/airflow/pull/20547
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mingshi-wang commented on pull request #20547: Allow sensors to return XCom values with PokeReturnValue
Posted by GitBox <gi...@apache.org>.
mingshi-wang commented on pull request #20547:
URL: https://github.com/apache/airflow/pull/20547#issuecomment-1002770040
> > A not-so-ideal way is to declare a dynamic return type:
> > def poke(self, context: Context) -> Union[bool, 'PokeReturnValue']:
>
> That's not enough. The problem is that any sensor that uses PokeReturnValue would have to be Airlfow 2.3+ only (PokeReturnValue will fail to be imported for such sensor)
You are right - what I can think of is to introduce a new type of sensor, e.x RichSensor and make the poke() method return a PokeReturnValue type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on pull request #20547: Allow sensors to return XCom values with PokeReturnValue
Posted by GitBox <gi...@apache.org>.
dstandish commented on pull request #20547:
URL: https://github.com/apache/airflow/pull/20547#issuecomment-1055902474
this pr taken over by @mingshi-wang in https://github.com/apache/airflow/pull/20656
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #20547: Allow sensors to return XCom values with PokeReturnValue
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #20547:
URL: https://github.com/apache/airflow/pull/20547#issuecomment-1002694727
It's still not really backwards compatible (in the sense a having a New Sensor returnig PokeReturnValue run on airlfow pre 2.3).
If you return PokeReturnValue() in Airlfow 2.2. it will not work. The BaseSensorOperator from Airflow 2.2 will run this:
```
while not self.poke():
````
And it will always finish after first check no matter what "is_done" is. In this case self.poke() returns PokeReturnValue() class which is Truthy no matter what is set in `is_done`).
But It could be made backwards compatible - and I like this much more than my "return Truthy" approach.
Should be enough to do this:
```
class PokeReturnValue:
"""
Sensors can optionally return this an instance of this class in the poke method.
If an xcom value is supplied when sensor is done, then the XCom value will be
push through the operator return value.
"""
def __init__(self, is_done, xcom_value=None):
self.xcom_value = xcom_value
self.is_done = is_done
def __bool__(self):
return self.is_done
```
~I think this is the BEST approach.~
UPDATE: Nope. It won't work, because in Airflow 2.2 there wil be no PokeReturnValue class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20547: Allow sensors to return XCom values with PokeReturnValue
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20547:
URL: https://github.com/apache/airflow/pull/20547#issuecomment-1002550086
Closing as duplicate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on pull request #20547: Allow sensors to return XCom values with PokeReturnValue
Posted by GitBox <gi...@apache.org>.
dstandish commented on pull request #20547:
URL: https://github.com/apache/airflow/pull/20547#issuecomment-1002682708
it's an alternative approach @potiuk that, in contrast with https://github.com/apache/airflow/pull/20546, is fully backward compatible...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #20547: Allow sensors to return XCom values with PokeReturnValue
Posted by GitBox <gi...@apache.org>.
potiuk closed pull request #20547:
URL: https://github.com/apache/airflow/pull/20547
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #20547: Allow sensors to return XCom values with PokeReturnValue
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #20547:
URL: https://github.com/apache/airflow/pull/20547#issuecomment-1002694727
It's still not really backwards compatible (in the sense a having a New Sensor returnig PokeReturnValue run on airlfow pre 2.3).
If you return PokeReturnValue() in Airlfow 2.2. it will not work. The BaseSensorOperator from Airflow 2.2 will run this:
```
while not self.poke():
````
And it will always finish after first check no matter what "is_done" is. In this case self.poke() returns PokeReturnValue() class which is Truthy no matter what is set in `is_done`).
But It could be made backwards compatible - and I like this much more than my "return Truthy" approach.
Should be enough to do this:
```
class PokeReturnValue:
"""
Sensors can optionally return this an instance of this class in the poke method.
If an xcom value is supplied when sensor is done, then the XCom value will be
push through the operator return value.
"""
def __init__(self, is_done, xcom_value=None):
self.xcom_value = xcom_value
self.is_done = is_done
def __bool__(self):
return self.is_done
```
I think this is the BEST approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20547: Allow sensors to return XCom values with PokeReturnValue
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20547:
URL: https://github.com/apache/airflow/pull/20547#issuecomment-1002762554
> A not-so-ideal way is to declare a dynamic return type:
> def poke(self, context: Context) -> Union[bool, 'PokeReturnValue']:
That's not enough. The problem is that any sensor that uses PokeReturnValue would have to be Airlfow 2.3+ only (PokeReturnValue will fail to be imported for such sensor)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org