You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "kolfild26 (via GitHub)" <gi...@apache.org> on 2023/08/13 21:39:30 UTC
[GitHub] [airflow] kolfild26 opened a new issue, #33373: TimeDeltaSensor suggestion
kolfild26 opened a new issue, #33373:
URL: https://github.com/apache/airflow/issues/33373
### Description
Currently, in TimeDeltaSensor its `delta` interval starts from [its dagrun creation](https://github.com/apache/airflow/blob/main/airflow/sensors/time_delta.py#L44) but not from when the task's actually been running.
### Use case/motivation
Imagine we've got some tasks before the TimeDeltaSensor task.
In order to use TimeDeltaSensor we need to know their expected execution time and add this time to the sensor's delta parameter. Ok, it's not good for using but can be acceptable. But what if their expected execution time may vary for some reason from time to time? How to calculate the delta to be passed in sensor?
So, is there any reason it's done this way? Wouldn't it be better if `delta` was attached to its task instance, which seems to be more intuitive. smth like this:
```
L44 target_dttm = context['task_instance'].start_date
```
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #33373: TimeDeltaSensor suggestion
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33373:
URL: https://github.com/apache/airflow/issues/33373#issuecomment-1676784684
I think - if you **really** have problem with calculating your time in JINJA - better approach would be to add an optional `delta` parameter in TimeSensor. TimeDeltaSensor is specifically to handle case where you want to pause something until some time passed since DagRun started. This is the purpose of this operator. It's purpose is not to wait till specific time.
TimeSensor on the other hand IS designed to wait until some specific time. And so it happen that in order to use it currently you would need to set that time to be "now() + timedelta" if you want to wait for specific delta since the Sensor has been created. You can do it with JINJA even now if you wish, but in order to be slightly easier, maybe it makes sense to allow the target_time in TimeSync to be not only `time` but also `timedelta' and in case it is timedelta, calculate target_time as now+ timedelta.
I think I would welcome such PR given it is nicely documented and explains the options.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kolfild26 commented on issue #33373: TimeDeltaSensor suggestion
Posted by "kolfild26 (via GitHub)" <gi...@apache.org>.
kolfild26 commented on issue #33373:
URL: https://github.com/apache/airflow/issues/33373#issuecomment-1677351121
@potiuk
> TimeSensor ... You can do it with JINJA even now if you wish
yes, that's what i actually do.
Originally I didn't think of the time sensor change, but it sounds interesting, Will consider to implement this via PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kolfild26 commented on issue #33373: TimeDeltaSensor suggestion
Posted by "kolfild26 (via GitHub)" <gi...@apache.org>.
kolfild26 commented on issue #33373:
URL: https://github.com/apache/airflow/issues/33373#issuecomment-1676471247
Oh, after publishing I found one related thread https://github.com/apache/airflow/issues/8876
Looks like this we'll not be accepted. Sad, some guys also wrote they want this feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #33373: TimeDeltaSensor suggestion
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33373:
URL: https://github.com/apache/airflow/issues/33373#issuecomment-1676793943
Assigned you @kolfild26
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] TimeDeltaSensor suggestion [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33373:
URL: https://github.com/apache/airflow/issues/33373#issuecomment-3986495628
@kolfild26 We are unassigning you from this issue as part of our updated [assignment policy](https://github.com/apache/airflow/blob/main/contributing-docs/04_how_to_contribute.rst#contribute-code-changes).
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org