You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/05/28 08:00:22 UTC
[GitHub] [airflow] anon-john opened a new issue #9046: TimeDeltaSensor and schedule_interval="@once" leads to crash
anon-john opened a new issue #9046:
URL: https://github.com/apache/airflow/issues/9046
**Apache Airflow version**: 1.10.10
**Environment**:
- **OS** (e.g. from /etc/os-release): centos7, Ubuntu 18.04.4 LTS
- **Install tools**: pip inside virtualenv
Using TimeDeltaSensor inside a DAG with `schedule_interval="@once"` leads to the following error:
```
[2020-05-28 09:41:59,409] {taskinstance.py:669} INFO - Dependencies all met for <TaskInstance: test_once.wait_10s 2020-05-26T00:00:00+00:00 [queued]>
[2020-05-28 09:41:59,417] {taskinstance.py:669} INFO - Dependencies all met for <TaskInstance: test_once.wait_10s 2020-05-26T00:00:00+00:00 [queued]>
[2020-05-28 09:41:59,418] {taskinstance.py:879} INFO -
--------------------------------------------------------------------------------
[2020-05-28 09:41:59,418] {taskinstance.py:880} INFO - Starting attempt 1 of 2
[2020-05-28 09:41:59,418] {taskinstance.py:881} INFO -
--------------------------------------------------------------------------------
[2020-05-28 09:41:59,425] {taskinstance.py:900} INFO - Executing <Task(TimeDeltaSensor): wait_10s> on 2020-05-26T00:00:00+00:00
[2020-05-28 09:41:59,427] {standard_task_runner.py:53} INFO - Started process 26581 to run task
[2020-05-28 09:41:59,496] {logging_mixin.py:112} INFO - Running %s on host %s <TaskInstance: test_once.wait_10s 2020-05-26T00:00:00+00:00 [running]> muttley
[2020-05-28 09:41:59,505] {taskinstance.py:1145} ERROR - unsupported operand type(s) for +=: 'NoneType' and 'datetime.timedelta'
Traceback (most recent call last):
File "/home/tfruboes/tmp/2020.05.dag_once/venv/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 983, in _run_raw_task
result = task_copy.execute(context=context)
File "/home/tfruboes/tmp/2020.05.dag_once/venv/lib/python3.6/site-packages/airflow/sensors/base_sensor_operator.py", line 107, in execute
while not self.poke(context):
File "/home/tfruboes/tmp/2020.05.dag_once/venv/lib/python3.6/site-packages/airflow/sensors/time_delta_sensor.py", line 44, in poke
target_dttm += self.delta
TypeError: unsupported operand type(s) for +=: 'NoneType' and 'datetime.timedelta'
[2020-05-28 09:41:59,506] {taskinstance.py:1168} INFO - Marking task as UP_FOR_RETRY
[2020-05-28 09:42:09,404] {logging_mixin.py:112} INFO - [2020-05-28 09:42:09,403] {local_task_job.py:103} INFO - Task exited with return code 1
```
(with `@daily` it runs ok). I would guess, that TimeDeltaSensor should run fine for any schedule interval, especially that combining it with `@daily` is useful for debugging/development purposes.
DAG definition, leading to the above crash, is the following:
```
from airflow import DAG
from airflow.operators.sensors import TimeDeltaSensor
from airflow.utils.dates import days_ago
from datetime import datetime, timedelta
default_args = {
"owner": "airflow",
"depends_on_past": False,
"start_date": days_ago(2),
"email": ["airflow@airflow.com"],
"email_on_failure": False,
"email_on_retry": False,
"retries": 1,
"retry_delay": timedelta(minutes=5),
"schedule_interval": None,
}
dag = DAG("test_once", default_args=default_args, is_paused_upon_creation=False, schedule_interval = "@once")
delay = TimeDeltaSensor(
task_id='wait_10s',
delta=timedelta(seconds=10),
dag=dag)
```
Moreover (not sure if it's a bug) - scheduler reports a success, despite the problem visible in the task log:
```
INFO - Executor reports execution of test_once.wait_10s execution_date=2020-05-26 00:00:00+00:00 exited with status success for try_number 1
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sr-regify commented on issue #9046: TimeDeltaSensor and schedule_interval="@once" leads to crash
Posted by GitBox <gi...@apache.org>.
sr-regify commented on issue #9046:
URL: https://github.com/apache/airflow/issues/9046#issuecomment-672975239
➜ airflow version
1.10.10
The above version reports the same issue. I found a pattern that, if there is any task the DAG unable to allocate the resources and continue to stay in running state then I changed the task status to **up for retry**
I got the error as below
TypeError: unsupported operand type(s) for +: 'NoneType' and 'datetime.timedelta'
Process DagFileProcessor0-Process:
Traceback (most recent call last):
File "/Users/ABCl/opt/anaconda3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/Users/ABCl/opt/anaconda3/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/Users/ABCl/opt/anaconda3/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 158, in _run_file_processor
pickle_dags)
File "/Users/ABC/opt/anaconda3/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper
return func(*args, **kwargs)
File "/Users/ABC/opt/anaconda3/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1611, in process_file
self._process_dags(dagbag, dags, ti_keys_to_schedule)
File "/Users/ABC/opt/anaconda3/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1295, in _process_dags
self._process_task_instances(dag, tis_out)
File "/Users/ABC/opt/anaconda3/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper
return func(*args, **kwargs)
File "/Users/ABC/opt/anaconda3/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 801, in _process_task_instances
session=session):
File "/Users/ABC/opt/anaconda3/lib/python3.7/site-packages/airflow/utils/db.py", line 70, in wrapper
return func(*args, **kwargs)
File "/Users/ABCl/opt/anaconda3/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 658, in are_dependencies_met
session=session):
File "/Users/ABC/opt/anaconda3/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 682, in get_failed_dep_statuses
dep_context):
File "/Users/ABC/opt/anaconda3/lib/python3.7/site-packages/airflow/ti_deps/deps/base_ti_dep.py", line 106, in get_dep_statuses
for dep_status in self._get_dep_statuses(ti, session, dep_context):
File "/Users/Neel/opt/anaconda3/lib/python3.7/site-packages/airflow/ti_deps/deps/not_in_retry_period_dep.py", line 47, in _get_dep_statuses
next_task_retry_date = ti.next_retry_datetime()
File "/Users/ABC/opt/anaconda3/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 729, in next_retry_datetime
return self.end_date + delay
TypeError: unsupported operand type(s) for +: 'NoneType' and 'datetime.timedelta'
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #9046: TimeDeltaSensor and schedule_interval="@once" leads to crash
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #9046:
URL: https://github.com/apache/airflow/issues/9046#issuecomment-635182466
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] anon-john commented on issue #9046: TimeDeltaSensor and schedule_interval="@once" leads to crash
Posted by GitBox <gi...@apache.org>.
anon-john commented on issue #9046:
URL: https://github.com/apache/airflow/issues/9046#issuecomment-638038365
I see your point. Maybe it would be useful to give an explicit error message stating "TimeDeltaSensor and `@once` make no sense" (instead of kinda cryptic `TypeError: unsupported operand type(s) for +=: 'NoneType' and 'datetime.timedelta'`)?
BTW what about the executor reporting success status in the scenario above? Is this expected/correct?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #9046: TimeDeltaSensor and schedule_interval="@once" leads to crash
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #9046:
URL: https://github.com/apache/airflow/issues/9046#issuecomment-830801831
As explained it doesn't make sense to use `@once` with TimeDeltaSensor. Note that the sensor calculates `execution_date + schedule_interval` when you set `schedule_interval="@once"` there is no interval (so it's None) which is exactly what the error message says:
`TypeError: unsupported operand type(s) for +=: 'NoneType' and 'datetime.timedelta'`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal closed issue #9046: TimeDeltaSensor and schedule_interval="@once" leads to crash
Posted by GitBox <gi...@apache.org>.
eladkal closed issue #9046:
URL: https://github.com/apache/airflow/issues/9046
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] klsnreddy commented on issue #9046: TimeDeltaSensor and schedule_interval="@once" leads to crash
Posted by GitBox <gi...@apache.org>.
klsnreddy commented on issue #9046:
URL: https://github.com/apache/airflow/issues/9046#issuecomment-637734773
This is kind of expected behavior, as TimeDeltaSensor adds the delta time to the schedule end time. So in case of manual or once executions there is not specified end time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org