You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/08/17 17:56:31 UTC
[GitHub] [airflow] l0n3r4n83r opened a new issue, #25772: Scheduler Will not Start if there is a task instance with a missing queued_dttm
l0n3r4n83r opened a new issue, #25772:
URL: https://github.com/apache/airflow/issues/25772
### Apache Airflow version
Other Airflow 2 version
### What happened
Airflow 2.2.5
Airflow scheduler will fail to start with the following lof when a task_instance has a blank queued_dttm
Setting the start_dt as queued_dttm resolved the issue. I am not sure when a task would have an empty queued_dttm
```
2022-08-17T18:57:27.340+02:00 | [2022-08-17 16:57:27,339] {{process_utils.py:124}} INFO - Sending Signals.SIGTERM to group 9. PIDs of all processes in the group: [18, 27, 20, 21, 22, 23, 24, 9]
-- | --
| 2022-08-17T18:57:27.340+02:00 | [2022-08-17 16:57:27,340] {{process_utils.py:75}} INFO - Sending the signal Signals.SIGTERM to group 9
| 2022-08-17T18:57:27.475+02:00 | [2022-08-17 16:57:27,474] {{process_utils.py:70}} INFO - Process psutil.Process(pid=18, status='terminated', started='16:57:26') (18) terminated with exit code None
| 2022-08-17T18:57:27.475+02:00 | [2022-08-17 16:57:27,475] {{process_utils.py:70}} INFO - Process psutil.Process(pid=20, status='terminated', started='16:57:26') (20) terminated with exit code None
| 2022-08-17T18:57:27.475+02:00 | [2022-08-17 16:57:27,475] {{process_utils.py:70}} INFO - Process psutil.Process(pid=22, status='terminated', exitcode=<Negsignal.SIGTERM: -15>, started='16:57:26') (22) terminated with exit code Negsignal.SIGTERM
| 2022-08-17T18:57:27.476+02:00 | [2022-08-17 16:57:27,475] {{process_utils.py:70}} INFO - Process psutil.Process(pid=21, status='terminated', exitcode=<Negsignal.SIGTERM: -15>, started='16:57:26') (21) terminated with exit code Negsignal.SIGTERM
| 2022-08-17T18:57:27.608+02:00 | [2022-08-17 16:57:27,608] {{process_utils.py:70}} INFO - Process psutil.Process(pid=23, status='terminated', exitcode=<Negsignal.SIGTERM: -15>, started='16:57:26') (23) terminated with exit code Negsignal.SIGTERM
| 2022-08-17T18:57:27.609+02:00 | [2022-08-17 16:57:27,608] {{process_utils.py:70}} INFO - Process psutil.Process(pid=24, status='terminated', exitcode=<Negsignal.SIGTERM: -15>, started='16:57:26') (24) terminated with exit code Negsignal.SIGTERM
| 2022-08-17T18:57:27.781+02:00 | [2022-08-17 16:57:27,781] {{process_utils.py:70}} INFO - Process psutil.Process(pid=27, status='terminated', started='16:57:27') (27) terminated with exit code None
| 2022-08-17T18:57:27.781+02:00 | [2022-08-17 16:57:27,781] {{process_utils.py:237}} INFO - Waiting up to 5 seconds for processes to exit...
| 2022-08-17T18:57:27.788+02:00 | [2022-08-17 16:57:27,788] {{process_utils.py:70}} INFO - Process psutil.Process(pid=9, status='terminated', exitcode=0, started='16:57:25') (9) terminated with exit code 0
| 2022-08-17T18:57:27.788+02:00 | [2022-08-17 16:57:27,788] {{scheduler_job.py:753}} INFO - Exited execute loop
| 2022-08-17T18:57:27.798+02:00 | Traceback (most recent call last):
| 2022-08-17T18:57:27.798+02:00 | File "/usr/local/bin/airflow", line 8, in <module>
| 2022-08-17T18:57:27.798+02:00 | sys.exit(main())
| 2022-08-17T18:57:27.798+02:00 | File "/usr/local/lib/python3.7/site-packages/airflow/__main__.py", line 48, in main
| 2022-08-17T18:57:27.798+02:00 | args.func(args)
| 2022-08-17T18:57:27.798+02:00 | File "/usr/local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
| 2022-08-17T18:57:27.799+02:00 | return func(*args, **kwargs)
| 2022-08-17T18:57:27.799+02:00 | File "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 92, in wrapper
| 2022-08-17T18:57:27.799+02:00 | return f(*args, **kwargs)
| 2022-08-17T18:57:27.799+02:00 | File "/usr/local/lib/python3.7/site-packages/airflow/cli/commands/scheduler_command.py", line 75, in scheduler
| 2022-08-17T18:57:27.799+02:00 | _run_scheduler_job(args=args)
| 2022-08-17T18:57:27.799+02:00 | File "/usr/local/lib/python3.7/site-packages/airflow/cli/commands/scheduler_command.py", line 46, in _run_scheduler_job
| 2022-08-17T18:57:27.799+02:00 | job.run()
| 2022-08-17T18:57:27.799+02:00 | File "/usr/local/lib/python3.7/site-packages/airflow/jobs/base_job.py", line 246, in run
| 2022-08-17T18:57:27.799+02:00 | self._execute()
| 2022-08-17T18:57:27.799+02:00 | File "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 726, in _execute
| 2022-08-17T18:57:27.799+02:00 | self._run_scheduler_loop()
| 2022-08-17T18:57:27.799+02:00 | File "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 779, in _run_scheduler_loop
| 2022-08-17T18:57:27.799+02:00 | self.adopt_or_reset_orphaned_tasks()
| 2022-08-17T18:57:27.799+02:00 | File "/usr/local/lib/python3.7/site-packages/airflow/utils/session.py", line 70, in wrapper
| 2022-08-17T18:57:27.799+02:00 | return func(*args, session=session, **kwargs)
| 2022-08-17T18:57:27.799+02:00 | File "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1215, in adopt_or_reset_orphaned_tasks
| 2022-08-17T18:57:27.799+02:00 | for attempt in run_with_db_retries(logger=self.log):
| 2022-08-17T18:57:27.799+02:00 | File "/usr/local/lib/python3.7/site-packages/tenacity/__init__.py", line 382, in __iter__
| 2022-08-17T18:57:27.799+02:00 | do = self.iter(retry_state=retry_state)
| 2022-08-17T18:57:27.799+02:00 | File "/usr/local/lib/python3.7/site-packages/tenacity/__init__.py", line 349, in iter
| 2022-08-17T18:57:27.799+02:00 | return fut.result()
| 2022-08-17T18:57:27.799+02:00 | File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 428, in result
| 2022-08-17T18:57:27.799+02:00 | return self.__get_result()
| 2022-08-17T18:57:27.799+02:00 | File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
| 2022-08-17T18:57:27.799+02:00 | raise self._exception
| 2022-08-17T18:57:27.799+02:00 | File "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1260, in adopt_or_reset_orphaned_tasks
| 2022-08-17T18:57:27.799+02:00 | to_reset = self.executor.try_adopt_task_instances(tis_to_reset_or_adopt)
| 2022-08-17T18:57:27.799+02:00 | File "/usr/local/lib/python3.7/site-packages/airflow/executors/celery_executor.py", line 485, in try_adopt_task_instances
| 2022-08-17T18:57:27.799+02:00 | self.adopted_task_timeouts[ti.key] = ti.queued_dttm + self.task_adoption_timeout
| 2022-08-17T18:57:27.799+02:00CopyTypeError: unsupported operand type(s) for +: 'NoneType' and 'datetime.timedelta' | TypeError: unsupported operand type(s) for +: 'NoneType' and 'datetime.timedelta'
```
### What you think should happen instead
The scheduler should be able to start or a empty queued_dttm should not be allowed
### How to reproduce
Have a task instance with null queued_dttm
### Operating System
Ubuntu Docker
### Versions of Apache Airflow Providers
_No response_
### Deployment
Other Docker-based deployment
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr closed issue #25772: Scheduler Will not Start if there is a task instance with a missing queued_dttm
Posted by GitBox <gi...@apache.org>.
uranusjr closed issue #25772: Scheduler Will not Start if there is a task instance with a missing queued_dttm
URL: https://github.com/apache/airflow/issues/25772
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] l0n3r4n83r commented on issue #25772: Scheduler Will not Start if there is a task instance with a missing queued_dttm
Posted by GitBox <gi...@apache.org>.
l0n3r4n83r commented on issue #25772:
URL: https://github.com/apache/airflow/issues/25772#issuecomment-1218683615
> Did you recently upgrade by any chance? (Just thinking about how it ended up in this situation, if we have a bug in code, or if the migration needs fixing.)
It has happened twice quite a few days after the upgrade to 2.2.5 version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #25772: Scheduler Will not Start if there is a task instance with a missing queued_dttm
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #25772:
URL: https://github.com/apache/airflow/issues/25772#issuecomment-1218446485
Did you recently upgrade by any chance? (Just thinking about how it ended up in this situation, if we have a bug in code, or if the migration needs fixing.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #25772: Scheduler Will not Start if there is a task instance with a missing queued_dttm
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #25772:
URL: https://github.com/apache/airflow/issues/25772#issuecomment-1219016182
Duplicate of #22160, fixed in 2.3.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org