You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "kolfild26 (via GitHub)" <gi...@apache.org> on 2023/08/23 21:50:27 UTC
[GitHub] [airflow] kolfild26 opened a new issue, #33678: DateTimeSensor does not respect poke_interval when executing from PythonOperator
kolfild26 opened a new issue, #33678:
URL: https://github.com/apache/airflow/issues/33678
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
### **v.2.6.3**
Executing the DateTimeSensor with `reschedule` mode from inside the PythonOperator leads to strange behaviour.
### What you think should happen instead
_No response_
### How to reproduce
```
def _wait(**context):
target_time = datetime.now(timezone.utc) + timedelta(hours=5)
wait = DateTimeSensor(
task_id="wait",
target_time=target_time,
poke_interval=10,
timeout=10000,
mode="reschedule",
soft_fail=False,
)
wait.execute(context=context)
wait_trigger = PythonOperator(
task_id="wait_trigger",
python_callable=_wait,
)
wait_trigger
```
What's happening ⬇️
1) Despite poke_interval is set as 10 seconds, it actually runs every second. See below.
2) Moreover, every time (1 second) it updates its target_time
```
[2023-08-23, 23:35:35 ] {{date_time.py:74}} INFO - Checking if the time (2023-08-24T01:35:35.753491+00:00) has come
[2023-08-23, 23:35:36 ] {{date_time.py:74}} INFO - Checking if the time (2023-08-24T01:35:36.823583+00:00) has come
[2023-08-23, 23:35:37 ] {{date_time.py:74}} INFO - Checking if the time (2023-08-24T01:35:37.906547+00:00) has come
[2023-08-23, 23:35:39 ] {{date_time.py:74}} INFO - Checking if the time (2023-08-24T01:35:38.986302+00:00) has come
[2023-08-23, 23:35:39 ] {{date_time.py:74}} INFO - Checking if the time (2023-08-24T01:35:39.760245+00:00) has come
[2023-08-23, 23:35:40 ] {{date_time.py:74}} INFO - Checking if the time (2023-08-24T01:35:40.823194+00:00) has come
[2023-08-23, 23:35:41 ] {{date_time.py:74}} INFO - Checking if the time (2023-08-24T01:35:41.911287+00:00) has come
[2023-08-23, 23:35:43 ] {{date_time.py:74}} INFO - Checking if the time (2023-08-24T01:35:42.994946+00:00) has come
[2023-08-23, 23:35:43 ] {{date_time.py:74}} INFO - Checking if the time (2023-08-24T01:35:43.558043+00:00) has come
[2023-08-23, 23:35:44 ] {{date_time.py:74}} INFO - Checking if the time (2023-08-24T01:35:44.625291+00:00) has come
```
As for the tagret time, it's because i'm using now() in the example above. Wiil be no problem if it's fixed. But this is a symptom that every 1 second the scheduler re-run `_wait` function again.
I thought things should be going like:
- It executes the PythonOperator once
- There is a callabe function inside the PythonOperator which is also supposed to be running ONCE
- It creates and executes the DateTimeSensor task
- Then the DateTimeSensor works based on those parameters which was passed into it in the callable.
But in real life it executes the callable again each time it reschedules the task which is executing in the callable itself.
And even assuming it's an expected behaviour (is it?), why `poke_interval` is not taking into account is still unclear.
p.s All this happens when the `reschedule` mode is set
### Operating System
rhel
### Versions of Apache Airflow Providers
_No response_
### Deployment
Virtualenv installation
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis closed issue #33678: DateTimeSensor does not respect poke_interval when executing from PythonOperator
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis closed issue #33678: DateTimeSensor does not respect poke_interval when executing from PythonOperator
URL: https://github.com/apache/airflow/issues/33678
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on issue #33678: DateTimeSensor does not respect poke_interval when executing from PythonOperator
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #33678:
URL: https://github.com/apache/airflow/issues/33678#issuecomment-1690749417
You should not manually run `execute` method of operators, this method intend to use by Airflow itself.
Between initialise operator and execute it a lot of internal things happen which required for proper work Airflow itself: scheduling, execution, monitoring, and etc, but if you run `execute` in your code, you just skip this steps and behaviour might be unpredictable.
For example, sensors in rescheduling mode just execute once and if result False send signal to scheduler for reschedule it.
I would recommend write your own operator/sensor, you could create it from scratch or overwrite one of existed.
Some useful links for reference:
- [Creating a custom Operator](https://airflow.apache.org/docs/apache-airflow/stable/howto/custom-operator.html)
- [Public Interface of Airflow](https://airflow.apache.org/docs/apache-airflow/stable/public-airflow-interface.html)
- [How-to Guides](https://airflow.apache.org/docs/apache-airflow/stable/howto/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org