You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/24 03:30:19 UTC
[GitHub] [airflow] manuzhang opened a new issue #19801: Airflow scheduler crashed with TypeError: '>=' not supported between instances of 'datetime.datetime' and 'NoneType'
manuzhang opened a new issue #19801:
URL: https://github.com/apache/airflow/issues/19801
### Apache Airflow version
2.1.4
### Operating System
Ubuntu 20.04.3 LTS
### Versions of Apache Airflow Providers
_No response_
### Deployment
Other Docker-based deployment
### Deployment details
_No response_
### What happened
Airflow scheduler crashed with following exception
```
[2021-11-23 19:51:17,491] {process_utils.py:100} INFO - Sending Signals.SIGTERM to GPID 19
[2021-11-23 19:51:18,085] {process_utils.py:66} INFO - Process psutil.Process(pid=28, status='terminated', started='19:51:17') (28) terminated with exit code None
[2021-11-23 19:51:18,123] {process_utils.py:207} INFO - Waiting up to 5 seconds for processes to exit...
[2021-11-23 19:51:18,125] {process_utils.py:66} INFO - Process psutil.Process(pid=29, status='terminated', started='19:51:17') (29) terminated with exit code None
[2021-11-23 19:51:18,129] {process_utils.py:66} INFO - Process psutil.Process(pid=19, status='terminated', exitcode=0, started='19:51:13') (19) terminated with exit code 0
[2021-11-23 19:51:18,130] {scheduler_job.py:722} INFO - Exited execute loop
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.8/dist-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/usr/local/lib/python3.8/dist-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/airflow/utils/cli.py", line 92, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/airflow/cli/commands/scheduler_command.py", line 70, in scheduler
job.run()
File "/usr/local/lib/python3.8/dist-packages/airflow/jobs/base_job.py", line 245, in run
self._execute()
File "/usr/local/lib/python3.8/dist-packages/airflow/jobs/scheduler_job.py", line 695, in _execute
self._run_scheduler_loop()
File "/usr/local/lib/python3.8/dist-packages/airflow/jobs/scheduler_job.py", line 788, in _run_scheduler_loop
num_queued_tis = self._do_scheduling(session)
File "/usr/local/lib/python3.8/dist-packages/airflow/jobs/scheduler_job.py", line 901, in _do_scheduling
callback_to_run = self._schedule_dag_run(dag_run, session)
File "/usr/local/lib/python3.8/dist-packages/airflow/jobs/scheduler_job.py", line 1143, in _schedule_dag_run
schedulable_tis, callback_to_run = dag_run.update_state(session=session, execute_callbacks=False)
File "/usr/local/lib/python3.8/dist-packages/airflow/utils/session.py", line 67, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/airflow/models/dagrun.py", line 438, in update_state
info = self.task_instance_scheduling_decisions(session)
File "/usr/local/lib/python3.8/dist-packages/airflow/utils/session.py", line 67, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/airflow/models/dagrun.py", line 539, in task_instance_scheduling_decisions
schedulable_tis, changed_tis = self._get_ready_tis(scheduleable_tasks, finished_tasks, session)
File "/usr/local/lib/python3.8/dist-packages/airflow/models/dagrun.py", line 565, in _get_ready_tis
if st.are_dependencies_met(
File "/usr/local/lib/python3.8/dist-packages/airflow/utils/session.py", line 67, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/airflow/models/taskinstance.py", line 890, in are_dependencies_met
for dep_status in self.get_failed_dep_statuses(dep_context=dep_context, session=session):
File "/usr/local/lib/python3.8/dist-packages/airflow/models/taskinstance.py", line 911, in get_failed_dep_statuses
for dep_status in dep.get_dep_statuses(self, session, dep_context):
File "/usr/local/lib/python3.8/dist-packages/airflow/ti_deps/deps/base_ti_dep.py", line 101, in get_dep_statuses
yield from self._get_dep_statuses(ti, session, dep_context)
File "/usr/local/lib/python3.8/dist-packages/airflow/ti_deps/deps/ready_to_reschedule.py", line 66, in _get_dep_statuses
if now >= next_reschedule_date:
TypeError: '>=' not supported between instances of 'datetime.datetime' and 'NoneType'
```
### What you expected to happen
_No response_
### How to reproduce
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] manuzhang edited a comment on issue #19801: Airflow scheduler crashed with TypeError: '>=' not supported between instances of 'datetime.datetime' and 'NoneType'
Posted by GitBox <gi...@apache.org>.
manuzhang edited a comment on issue #19801:
URL: https://github.com/apache/airflow/issues/19801#issuecomment-977576861
As per https://dev.mysql.com/doc/refman/5.7/en/datetime.html,
> The TIMESTAMP data type is used for values that contain both date and time parts. TIMESTAMP has a range of '1970-01-01 00:00:01' UTC to '2038-01-19 03:14:07' UTC.
reschedule_date was set to `0000-00-00 00:00:00.000000` since it's exceeded the upper limit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] manuzhang commented on issue #19801: Airflow scheduler crashed with TypeError: '>=' not supported between instances of 'datetime.datetime' and 'NoneType'
Posted by GitBox <gi...@apache.org>.
manuzhang commented on issue #19801:
URL: https://github.com/apache/airflow/issues/19801#issuecomment-977733180
@eladkal
I generated the dag from code but the interval was mistakenly calculated in milliseconds rather than seconds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19801: Airflow scheduler crashed with TypeError: '>=' not supported between instances of 'datetime.datetime' and 'NoneType'
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19801:
URL: https://github.com/apache/airflow/issues/19801#issuecomment-984020278
> @uranusjr That's the second time this week you've done a task I was looking at, and did it better than me. +1 smile
Great minds think alike !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] manuzhang commented on issue #19801: Airflow scheduler crashed with TypeError: '>=' not supported between instances of 'datetime.datetime' and 'NoneType'
Posted by GitBox <gi...@apache.org>.
manuzhang commented on issue #19801:
URL: https://github.com/apache/airflow/issues/19801#issuecomment-977576861
As per https://dev.mysql.com/doc/refman/5.7/en/datetime.html,
> the range for TIMESTAMP values is '1970-01-01 00:00:01.000000' to '2038-01-19 03:14:07.999999'
reschedule_date was set to
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #19801: Airflow scheduler crashed with TypeError: '>=' not supported between instances of 'datetime.datetime' and 'NoneType'
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #19801:
URL: https://github.com/apache/airflow/issues/19801#issuecomment-983957409
@ferruzzi I proposed a change to BaseSensorOperator in #19821. Could you take a look and see if I covered anything correctly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #19801: Airflow scheduler crashed with TypeError: '>=' not supported between instances of 'datetime.datetime' and 'NoneType'
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #19801:
URL: https://github.com/apache/airflow/issues/19801#issuecomment-977814292
So the value was unintended? In that case I guess we could just invent an arbitrary cap to `poke_interval` and just make the DAG error out on parse. It does not need to be that precise either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] manuzhang closed issue #19801: Airflow scheduler crashed with TypeError: '>=' not supported between instances of 'datetime.datetime' and 'NoneType'
Posted by GitBox <gi...@apache.org>.
manuzhang closed issue #19801:
URL: https://github.com/apache/airflow/issues/19801
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] manuzhang edited a comment on issue #19801: Airflow scheduler crashed with TypeError: '>=' not supported between instances of 'datetime.datetime' and 'NoneType'
Posted by GitBox <gi...@apache.org>.
manuzhang edited a comment on issue #19801:
URL: https://github.com/apache/airflow/issues/19801#issuecomment-977576861
As per https://dev.mysql.com/doc/refman/5.7/en/datetime.html,
> The TIMESTAMP data type is used for values that contain both date and time parts. TIMESTAMP has a range of '1970-01-01 00:00:01' UTC to '2038-01-19 03:14:07' UTC.
> Invalid DATE, DATETIME, or TIMESTAMP values are converted to the โzeroโ value of the appropriate type ('0000-00-00' or '0000-00-00 00:00:00'), if the SQL mode permits this conversion
reschedule_date was set to `0000-00-00 00:00:00.000000` since it's exceeded the upper limit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal edited a comment on issue #19801: Airflow scheduler crashed with TypeError: '>=' not supported between instances of 'datetime.datetime' and 'NoneType'
Posted by GitBox <gi...@apache.org>.
eladkal edited a comment on issue #19801:
URL: https://github.com/apache/airflow/issues/19801#issuecomment-977670466
It's a big problem if the scheduler crash due to problem with a specific DAG. This means that a single DAG causes problems for all the other DAGs.
I'm also interested in why would you want to set interval for more than 9999 days? While crashed scheduler is a big problem the use case of getting there seems theoretical to me so I wouldn't say that this is require an urgent fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ferruzzi commented on issue #19801: Airflow scheduler crashed with TypeError: '>=' not supported between instances of 'datetime.datetime' and 'NoneType'
Posted by GitBox <gi...@apache.org>.
ferruzzi commented on issue #19801:
URL: https://github.com/apache/airflow/issues/19801#issuecomment-983941612
> So the value was unintended? In that case I guess we could just invent an arbitrary cap to `poke_interval` and just make the DAG error out on parse. It does not need to be that precise either.
That would be an easy check in the base sensor, I can get that one done. Should we implement a cap to dag schedule-interval as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #19801: Airflow scheduler crashed with TypeError: '>=' not supported between instances of 'datetime.datetime' and 'NoneType'
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #19801:
URL: https://github.com/apache/airflow/issues/19801#issuecomment-979173999
Moving this to 2.3 since itโs not a 2.2 regression and can arguably be categorised as a new feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19801: Airflow scheduler crashed with TypeError: '>=' not supported between instances of 'datetime.datetime' and 'NoneType'
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19801:
URL: https://github.com/apache/airflow/issues/19801#issuecomment-978062987
> Ugh, MySQLs date "handling" makes me sad.
Very much so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr closed issue #19801: Airflow scheduler crashed with TypeError: '>=' not supported between instances of 'datetime.datetime' and 'NoneType'
Posted by GitBox <gi...@apache.org>.
uranusjr closed issue #19801:
URL: https://github.com/apache/airflow/issues/19801
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #19801: Airflow scheduler crashed with TypeError: '>=' not supported between instances of 'datetime.datetime' and 'NoneType'
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #19801:
URL: https://github.com/apache/airflow/issues/19801#issuecomment-977670466
It's a big problem if the scheduler crash due to problem with a specific DAG. This means that a single DAG causes problems for all the other DAGs.
I'm also interested in why would you want to set interval for more than 9999 days? While crashed scheduler is a big problem the use case of getting there seems a theoretical to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #19801: Airflow scheduler crashed with TypeError: '>=' not supported between instances of 'datetime.datetime' and 'NoneType'
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #19801:
URL: https://github.com/apache/airflow/issues/19801#issuecomment-977471381
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #19801: Airflow scheduler crashed with TypeError: '>=' not supported between instances of 'datetime.datetime' and 'NoneType'
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #19801:
URL: https://github.com/apache/airflow/issues/19801#issuecomment-977910934
Ugh, MySQLs date "handling" makes me sad.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ferruzzi commented on issue #19801: Airflow scheduler crashed with TypeError: '>=' not supported between instances of 'datetime.datetime' and 'NoneType'
Posted by GitBox <gi...@apache.org>.
ferruzzi commented on issue #19801:
URL: https://github.com/apache/airflow/issues/19801#issuecomment-983978552
@uranusjr That's the second time this week you've done a task I was looking at, and did it better than me. ๐ ๐
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org