You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/02/06 15:18:19 UTC
[GitHub] [airflow] iercan opened a new issue #14116: Potential bug related with max_retry_delay
iercan opened a new issue #14116:
URL: https://github.com/apache/airflow/issues/14116
**Apache Airflow version**: 2.0.0
**Environment**:
I'm using apache/airflow:2.0.0 docker image and configured celery executor with postgresql 9.6 and redis 5.0.5
**What happened**:
I upgraded airflow from 1.10.14 to 2.0.0. First a few hours there was no error but then scheduler started to fail by giving below error.
I figured it out which dag case error and exception disappear when I remove `max_retry_delay=timedelta(minutes=60)` line from dag configuration. I believe this could be a bug about scheduler.
```
scheduler_1 | [2021-02-06 14:38:29,200] {scheduler_job.py:1293} ERROR - Exception when executing SchedulerJob._run_scheduler_loop
scheduler_1 | Traceback (most recent call last):
scheduler_1 | File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1275, in _execute
scheduler_1 | self._run_scheduler_loop()
scheduler_1 | File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1377, in _run_scheduler_loop
scheduler_1 | num_queued_tis = self._do_scheduling(session)
scheduler_1 | File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1515, in _do_scheduling
scheduler_1 | self._schedule_dag_run(dag_run, active_runs_by_dag_id.get(dag_run.dag_id, set()), session)
scheduler_1 | File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1677, in _schedule_dag_run
scheduler_1 | schedulable_tis, callback_to_run = dag_run.update_state(session=session, execute_callbacks=False)
scheduler_1 | File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/session.py", line 62, in wrapper
scheduler_1 | return func(*args, **kwargs)
scheduler_1 | File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dagrun.py", line 405, in update_state
scheduler_1 | info = self.task_instance_scheduling_decisions(session)
scheduler_1 | File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/session.py", line 62, in wrapper
scheduler_1 | return func(*args, **kwargs)
scheduler_1 | File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dagrun.py", line 499, in task_instance_scheduling_decisions
scheduler_1 | schedulable_tis, changed_tis = self._get_ready_tis(scheduleable_tasks, finished_tasks, session)
scheduler_1 | File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dagrun.py", line 525, in _get_ready_tis
scheduler_1 | if st.are_dependencies_met(
scheduler_1 | File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/session.py", line 62, in wrapper
scheduler_1 | return func(*args, **kwargs)
scheduler_1 | File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 813, in are_dependencies_met
scheduler_1 | for dep_status in self.get_failed_dep_statuses(dep_context=dep_context, session=session):
scheduler_1 | File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 834, in get_failed_dep_statuses
scheduler_1 | for dep_status in dep.get_dep_statuses(self, session, dep_context):
scheduler_1 | File "/home/airflow/.local/lib/python3.8/site-packages/airflow/ti_deps/deps/base_ti_dep.py", line 101, in get_dep_statuses
scheduler_1 | yield from self._get_dep_statuses(ti, session, dep_context)
scheduler_1 | File "/home/airflow/.local/lib/python3.8/site-packages/airflow/ti_deps/deps/not_in_retry_period_dep.py", line 47, in _get_dep_statuses
scheduler_1 | next_task_retry_date = ti.next_retry_datetime()
scheduler_1 | File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 880, in next_retry_datetime
scheduler_1 | delay = min(self.task.max_retry_delay, delay)
scheduler_1 | TypeError: '<' not supported between instances of 'datetime.timedelta' and 'float'
```
**What you expected to happen**:
Scheduler should work as expected.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #14116: Potential bug related with max_retry_delay
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #14116:
URL: https://github.com/apache/airflow/issues/14116#issuecomment-774492627
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] iercan commented on issue #14116: Potential bug related with max_retry_delay
Posted by GitBox <gi...@apache.org>.
iercan commented on issue #14116:
URL: https://github.com/apache/airflow/issues/14116#issuecomment-774496181
Sorry just saw this issue also mentioned here #13086
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #14116: Potential bug related with max_retry_delay
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #14116:
URL: https://github.com/apache/airflow/issues/14116#issuecomment-775411388
Closing in favour of that issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb closed issue #14116: Potential bug related with max_retry_delay
Posted by GitBox <gi...@apache.org>.
ashb closed issue #14116:
URL: https://github.com/apache/airflow/issues/14116
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org